Presenting an efficient model based on the subcombinations extracted from the feature to recognize human physical activities

Doctoral thesis in the field of computer engineering (artificial intelligence)

Abstract

Understanding and extracting information from images and videos is the common thread of most problems related to machine vision. Finding the main and useful parts of a movie and modeling the actions between these parts is one of the main goals of movie analysis. In the last decade, human activity detection using video images has been raised as a challenging topic in machine vision. Among the applications of this subject, we can mention the issues of surveillance and security, medicine and human-computer interaction. It is difficult and complex to extract the main components and summarize an activity due to the great diversity in the way an activity is performed. If we consider the beginning of video analysis to process the brightness of image pixels in different frames and the final goal to detect human activity, there is a great distance between the level of analysis and the final goal, and an urgent need to extract meaningful and higher-level features is felt. In fact, the main challenge is to fill the deep gap between low-level descriptors to express the type of activity and summarize it. In recent decades, researchers have not been very successful in providing effective summarization methods using vision and machine learning techniques, even at the level of images. In this regard, separation methods [1] have been proposed, which model the differential boundary of different classes. Despite their success, these models require a lot of labeled data and are limited to a specific context. In addition, the risk of overfitting [2] also threatens them. On the other hand, generative models [3] solved this problem by adding additional constraints to the model using a large amount of available unlabeled data. As an example, we can point to the unsupervised feature learning methods that reduce the distance between the low-level descriptors and the final model by adding some basic knowledge about the overall data structure. In this thesis, by presenting five different frameworks, the problem of human activity recognition is solved with the approach of summarizing and extracting higher-level features. The main steps of doing the work can be divided into three main parts, 1- feature extraction, 2- their quantization and 3- classification. In this research, the features of shape and movement related to two-dimensional images of video frames have been extracted. In the second part, which is almost the main part of this research, in order to reduce the quantization error and raise the level of the features (using the basic knowledge hidden in the data) and also for easier classification in the later stages, instead of the common methods such as K-means, thin coding methods and some improved versions of it, which are considered as unsupervised feature learning methods, have been used. In such methods, the goal is to find higher-level basic functions and describe the video using a linear combination of them. Also, we have used the very useful method of thin group encoding to extract useful information of the time sequence. Then, in order to avoid overfitting the model, spatial and temporal integration of coefficients is suggested. Finally, by using two different algorithms of the general methods of generative and separator classification, we have finished the activity detection.

Among the highlights of this thesis, we can mention the combination of several features with different modalities, extracting the meaningful components of an activity and modeling their relationship by considering the temporal structure of the data, reducing the quantization error, and also significantly reducing the spatial and temporal complexity. The presented methods have been evaluated on several activity recognition databases consisting of artificial and real data with different challenges, and good results have been obtained.

Key words: human activity recognition, basic knowledge, data structure, multi-category system, sparse coder, group sparse coder, unsupervised feature learning.

Chapter 1

Introduction

Introduction

Introduction

Understanding and analyzing images is the common thread of most machine vision problems. In this regard and with the advancement of different machine vision techniques, the analysis of different scenes has risen above the image level and analyzes the film (a sequence of frames) taking into account the time relationships between them. This provides a better and more accurate understanding of the intended scene. Today, human activity detection is one of the most important and interesting research topics in the field of machine vision. The purpose of this diagnosis is to analyze the activities of humans in an unknown video.In general, the analysis of human movements can be divided into three categories: 1- detection of human activity [1], 2- tracking human movements [2] and 3- analysis of movements of different parts of the human body [3]. Each of these analyzes can be performed on two or three-dimensional frames. In many practical problems, after finding people in images and following them, we seek to categorize their activities. Activity detection is a process of tagging human activities that can be done using various sensors such as vision and sound. In this research, we only use field of vision observations that can be taken from one or more cameras.  The label of a specific activity is a name that almost average people, upon hearing it, imagine the same activity and can do it the same way. In other words, the activity tag is the best descriptor of an example of an activity performed by different people in different conditions.

Looking deeper into the problem of activity recognition, it can be considered similar to some fields of artificial intelligence, such as natural language processing, text processing, and voice recognition, from different perspectives. It is useful to use different perspectives to analyze this issue. For example, we use the concepts of natural language and human speech to define the activity more precisely and recognize it. Humans use sentences in their daily conversations. Each simple sentence consists of subject, object and verb. There is almost the same structure to express the visual concepts in a movie. From this point of view, the subject or performer of the activity is usually humans. The object can usually be other people or objects or the environment on which the subject performs its activity. Finally, the verb indicates the type of activity or interaction between the subject and objects. From the point of view of sound processing, as components such as phonemes, letters and words make a sentence in this area, the sequence and order of movements together form a meaningful activity. Considering the existing similarities, it seems that we can achieve a more efficient solution to our problem by examining different methods in the mentioned areas.  

There are different types of human activities. We divide the activities into 4 different levels according to their complexity[1]:

1.[4]: The basic movements of body parts are atomic and are used to describe meaningful human movements. Such as opening the hand from the elbow or folding it, fisting the hand, etc.

Human activity [5]: We put the simple activities that can include several movements from the first category in the time dimension into the second category. In other words, the combination of human atomic movements constitutes an activity. Such as walking, shaking hands, etc.

Interaction of human activities[6]: In this category, two or more people or people and objects are connected. Such as two people fighting with each other or someone stealing someone's bag, which is an example of the interaction of two people with the same object.

Group activities [7]: an operation that is carried out by a group of people with each other or with objects. Like marching a group of soldiers, meeting a group, etc.

For example, a game of tennis is an interaction of human activity. This interaction includes several activities such as serving, returning the ball or time-out, etc. Each of these activities includes basic movements. For example, serving includes throwing the ball upwards, moving the racket back, moving the racket and hitting the ball. It should be noted that the choice of initial movements is an important and influential issue in the continuation of the diagnosis process. For example, arm movement may not be a sufficient movement for part of the activity of playing tennis, while it may be a sufficient movement for the activity of drinking. Therefore, the extraction of the basic movements of an activity is somewhat dependent on the type of activity, and a precise definition is not completely possible. Applications The ability to recognize complex human activities has various applications. including automatic monitoring systems in public places such as airports and highways that require the detection of abnormal and suspicious movements and activities against normal and normal activities [1]. For example, in airports, detecting some activities such as a person leaving a bag or throwing a person's handbag in the trash can be considered as suspicious movements.