Detection of web spam using data mining techniques

Master Thesis (M.sc)

Abstract:

Nowadays, spam [1] is one of the main problems of search engines, because it makes the quality of search results unfavorable. In recent years, there have been many advances in detecting fake pages, but new spamming techniques have also emerged in response. It is necessary to improve anti-spam techniques to overcome these attacks.

A common problem we encounter in this field is that many documents are ranked high by the search engine when they do not deserve it. Considering the ever-increasing expansion of the web and the emergence of new spam techniques by spammers, the purpose of this thesis is to investigate data mining methods to better identify spam pages from non-spam.

Algorithms and data mining software are among the tools used in this research. The UK2007 standard data set and Waka software have been used to provide optimal models, and an attempt is made to provide models that reduce the features used to identify spam pages from non-spam and provide optimal efficiency. It is a process for extracting useful patterns from databases [1]. Data mining can extract the useful patterns desired by its users from different types of databases. Most researchers consider data mining as synonymous with knowledge discovery in databases. Knowledge discovery includes the following steps that are performed sequentially:

Data refinement: removes clutter and inconsistent data.

Data integration: combines data sources when necessary.

Data transformation: transforms data into a suitable form for data mining.

Data mining: is a necessary process in which intelligent methods of patterns they extract appropriate data.

Pattern evaluation: evaluates the extracted patterns.

Knowledge display: In this stage, different knowledge display techniques are used to show the discovered and explored knowledge to the user.

Increasing the ability of different techniques and tools in creating and collecting data and the importance of databases due to their availability and strength in various industries and research, as well as the wide global network that is known as It is an important source of information, we are faced with a huge amount of data and databases.

Although search engines have developed many techniques to identify web spam, but web spammers have developed new tactics to influence the results of search engine ranking algorithms, in order to achieve higher ranks.

Data mining as an important and new tool widely used in identifying spam pages from non-spam.

Statement of the problem:

Search engines have become a place to search for information on the web. Considering the phenomenon of spam, the search results are not always favorable.

For more than two decades, the research on the recovery of hostile information has many interested in academia and industry. Spams have cast a shadow on every information system, e-mail, web and blogs and social networks. This concept was proposed for the first time in 1996 and was soon presented as a challenge for search engines. Recently, all major search engine companies have determined the recovery of hostile information as a high priority due to the numerous and negative effects caused by the appearance of spam [3, 2]. First, spam makes the quality of search results unfavorable and reduces the efficiency that legitimate sites can have in the absence of spam.

Secondly, it causes a user's lack of confidence in the search engine and ultimately leads to the replacement of the search engine, which will not cost the user.

The purpose is to determine the different characteristics of web pages in order to rank the results of the search engine, and based on this, the classification is done in order to identify spam sites from valid sites. related to adults and malware and attacks are raised.For example, the ranking of 100 million pages based on page ranking algorithms showed that 11 out of 20 results were pronographic sites that reached this result by manipulating content and links [5, 4]. In the past, this caused a significant amount of computing and storage resources from search engine companies to be wasted. In 2005, the damage caused by spam was estimated at 50 billion dollars. In 2009, it was estimated at 130 billion dollars [6]. Among the new challenges are the rapid growth of the web and its heterogeneity and the simplification of content creation tools (for example, wiki websites, blogging platforms, etc.) and the reduction of website maintenance costs (such as domain registration, web hosting, etc.), which has caused the evolution of spam and the emergence of new strains of web spam that cannot be detected by previous successful methods. They have high ranking results. For 85% of the questions, only the first page of the result was noticed and only three links were clicked [7]. Therefore, the effort to be included in the first page of the search engine result will have a clear economic incentive due to the increase in website traffic. In order to achieve this goal, website owners try to manipulate search engine ranking results. According to the conducted studies, the amount of spam varies from 6 to 22 percent, and this indicates the scope and scope of the problem [9, 8]. Structure of the thesis: According to the subject of the thesis, at the beginning, in the second chapter, the structure of the web and the concepts of spam and types of spam and some of the most important machine learning methods have been investigated. In the third chapter, the available data sets are introduced and the techniques of dealing with link spam, content spam, and link-content spam have been examined. In the fourth chapter, the introduction and selected data set are discussed and the results related to the optimal data mining models are stated. In the fifth chapter, as the final chapter, the final result of the work has been summarized and the issues that can be considered and investigated as the subject of the master's thesis in the future are stated.

Chapter Two:

Web and web spam

In view of the topic of the thesis, it seems necessary to examine the structure of the web and types of spam as well as the most important machine learning algorithms. Therefore, in this chapter, the concepts of the web, then the types of spam, and finally the machine learning algorithms have been examined.

2-1- The World Wide Web:

The World Wide Web can be considered as a database managed by humanity for the storage and sharing of various documents, however, the web differs from the usual databases in terms of size, very fast dynamics, and heterogeneity. First, the web is very large, its size cannot be determined and measured accurately. The number of pages is unlimited and the content depends on the information entered by the user. Jolly and Signorini [2] have reported that the web contained 11 billion pages in January 2005 [10]. In 2008, Google claimed that their system processed one trillion URLs on the web] 11.[

Another challenge is that web content changes rapidly. Chu and Garc?a-Molina [3] have evaluated the speed of change by downloading 720,000 pages in a four-month period in 1999 [12]. They concluded that the page contents were modified for 23% of the collection, and within 50 days, 50% of the collection was edited or removed [13].

Web documents are heterogeneous for different reasons and different perspectives. In addition to text, websites can contain images, videos, and audio files in various formats. The size of these documents can vary from one byte to thousands of megabytes. Among the most common HTML files, you can find different versions and pages with incorrect syntax that do not follow W3C standards but are still viewable by web browsers[4]. Web content is often unstructured, consists of different languages ??and styles, and varies widely in quality. Although HTML pages contain some metadata[5], they are not generally reliable. The main goal of the semantic web is to put the data in a machine-friendly structured form [6], which makes cooperation between humans and machines possible [14].