Intelligent improvement of question selection based on examiner's knowledge level in computer adaptive test

Master's Thesis of Information Technology Engineering Major in Information Systems Management

Abstract

Computer adaptive testing is a growing assessment method in many educational as well as non-educational institutions around the world.  The distinguishing feature of the computer adaptive test is the execution of the test according to the test taker's ability based on the answers to the previous questions.  Hence, it is possible to administer a shorter test and at the same time obtain a more accurate estimate of the test taker's ability.  Various methods have been proposed to create a computer adaptive test based on question-answer theory.  The aim of this thesis is to improve the question selection method in the computer adaptive test based on the question-answer theory according to the examiner's knowledge level.  In this thesis, in addition to the parameters of the questions that have been proposed in the question-answer theory, a structure for the questions has been proposed, based on which the question selection method can be improved, in addition, the test length will be shorter.  It is also possible to identify the subject in which the examiner is weak and suggest a suitable approach for the examiner based on that.  That is, if the examinees were weak in a subject, the teacher can make a decision according to the examinee's level. 

In choosing the question, another issue that is discussed is the calculation of the examiner's ability level, because based on his level, the appropriate question is chosen for him.  There are different methods to calculate and estimate the examinee's level, in this thesis, neural networks are used to determine the level of knowledge.  Three neural network models are presented here, the first model is comprehensive pruning neural network, which is an accurate network but with high training time, the other models are multilayer perceptron neural network and radial basis function neural network model. 

Key words: computer assessment, computer-based tests, computer adaptive test, question-answer theory, Bayesian network, neural network. 

1- Chapter 1: Introduction

Examination is one of the most common ways to test knowledge.   The main purpose of the test is to determine the examiner's knowledge level of one or more subjects in which a field of knowledge has been selected.  Today, various methods such as presenting material in class, writing articles, projects, etc. are used to evaluate knowledge.  However, the most common means of measuring knowledge is the oral test.   Since the computer has been the most widely used teaching tool in recent decades, and since its use has become common at all levels of education, computer-based testing is very popular. 

Of the various test methods that are available today, "Computer Adaptive Test" provides the maximum balance of accuracy and efficiency.   Over the past few decades, computer-based compliance testing has been widely used in the fields of training, certification, and licensing.   "Computer adaptive test" selects questions based on the testee's answer to the previous questions, in the form of progressions, to increase the accuracy of the test.  From the examinee's point of view, it seems that the difficulty of the question adapts to his ability level.  For example, if the subject performs well in answering a question with medium difficulty, he will be presented with a question that is more difficult, or if he performs well in answering a question with medium difficulty, he will be presented with an easier question.  In computer adaptive tests, compared to fixed multiple-choice tests where a fixed set of questions are presented to the examinee, fewer questions are required to achieve the same exact results.  Of course, there is no restriction that multiple-choice questions must be used in the methodology of "computer adaptive test", but since most of the questions are multiple-choice, these types of questions are also used in most computer adaptive tests. 

Computerized adaptive testing avoids running irrelevant questions.  Such as questions that are too easy or too difficult for the subject and inappropriate behaviors such as guessing, inattention, and patterns that stimulate answering.  These questions are largely eliminated.  When the questions are presented in a way that does not match the ability level of the examinee and the challenge level of the question (difficulty level) is high, the examinee will be anxious, and if it is low, he will be disinterested..  Only if the person will experience the appropriate test experience if the difficulty of the question matches his ability level, adaptive tests have this feature. 

1-1 research objectives and explanation of the problem

In this thesis, the method of choosing the next question for the examiner has been improved considering that it is closest to his level of knowledge.  To select the question, three functions are defined for it:

Question selection based on the subject of the lesson

Question selection based on the question answer theory

Question selection based on the history of the answered questions

In the first function, questions related to the desired topic are selected.  Then, in the function, by using the question-answer theory and improving the methods of calculating the probability of the examinee answering the question, the questions that are most likely to be answered by the examinee are selected.  After the questions are selected in the second function, based on the history of the questions, a question is selected from among them and asked to the examiner.   Improvement of the question selection process in the first and second function is desired in this thesis.  In fact, questions will be selected that are the best for the examiner in terms of measuring his knowledge. 

In this dissertation, the structure of the computer adaptive test will be described, which is made by combining the methods used by others and a change in their implementation. The parameters proposed in the question-answer theory are not enough to select the question and other features are needed in this function. For this purpose, the question structure is created using the subject, topic and concept, and the questions with this category are selected from the question bank. did 

Another important issue that should be considered in choosing the questions of the computer adaptive test is the estimation of the examiner's knowledge level.  There are different methods for this estimation, such as Fisher's information, Newton-Rafster method, Bayesian networks and neural network.  In this thesis, this estimation is done using three neural network models, which are comprehensive pruning neural network, multilayer perceptron neural network, and radial basis function neural network.  In this thesis, these three neural network models will be implemented and compared using the obtained results. 

This thesis is organized as follows.  In the second chapter, the computer adaptive test will be discussed, and in the third chapter, topics related to the computer adaptive test will be discussed, including question-answer theory, Bayesian networks and its application in the computer adaptive test, and finally neural networks.  The proposed method will be described in the fourth chapter and the implementation of the proposed method will be discussed in the fifth chapter.  In the sixth chapter, comparisons, conclusions and future work will be discussed. 

Chapter 2: Computer Adaptive Test

2-1 Introduction

"Computer Adaptive Test" by Lord (1971), Owen (1975), and Weiss (1976), among others, to measure the ability level of examinees more accurately than conventional tests and by making an individual test for each The examiner was suggested [1]. 

"Computer adaptive test" is a type of test developed to increase the efficiency of evaluating the examiner's knowledge.  The main goal of computer adaptive testing is to optimize the test taker's learning process [2].  which obtains the estimate by selecting the questions for the examinee based on his answers (so it is often called the appropriate test[1]) during the previous test period.  The difficulty level of the next question is chosen so that it is neither too difficult nor too easy for the examinee.   More precisely, the question is chosen so that the examinee answers the question correctly with a probability of 50%.  Of course, the first question cannot be determined in this way because at this point nothing is known about the capabilities of the exam (the question with moderate difficulty is chosen), but the choice from the second question onwards can be chosen for each examiner with better adaptability.  With each answer to the question, the computer can better assess the examiner's knowledge. 

2-2 types of adaptive assessment

2-2-1 Wright and Douglas theory[3]

Wright and Douglas (1975) proposed adaptive assessment in which the questions were scaled based on the difficulty level of the Rush logic model.