All of us have used a search engine, in example Google, in every single day for searching everything, even on simple things. But have you ever imagined, how that search engine can retrieve all of our documents based on what we want to search (query)?

我们所有人每天都使用搜索引擎(例如Google)来搜索所有内容，即使是简单的东西。 但是您是否曾想过，该搜索引擎如何根据我们要搜索(查询)的内容来检索我们所有的文档？

In this article, I will show you on how to build a simple search engine from scratch using Python and its supporting library. After you read the article, I hope you can understand how to build your own search engine based on what you need. Without further, let’s go!

在本文中，我将向您展示如何使用Python及其支持库从头构建一个简单的搜索引擎。 阅读本文后，希望您能了解如何根据需要构建自己的搜索引擎。 没有进一步，我们走吧！

Side note: I’ve also created a notebook of the code, so if you want to follow along with me you can click on this link here. Also, the documents that I will use is in Indonesian. But don’t worry, you can use any documents regardless of the language.

旁注：我还创建了代码笔记本，因此，如果您想跟我一起学习，可以单击此处的链接。 另外，我将使用的文档是印尼文。 但请放心，无论使用哪种语言，都可以使用任何文档。

大纲 (Outline)

Before we get our hands dirty, let me give you the steps on how to implement this, and on each section, I will explain on how to build it. They are,

在开始动手之前，让我为您提供如何实现此步骤的步骤，并在每个部分中说明如何构建它。 他们是，

Preparing the documents

准备文件 Create a Term-Document Matrix with TF-IDF weighting

创建具有TF-IDF权重的术语文档矩阵Calculate the similarities between query and documents using Cosine Similarity

使用余弦相似度计算查询和文档之间的相似度Retrieve the articles that have the highest similarity on it.

检索相似度最高的文章。

流程 (The Process)

检索文件(Retrieve the documents)

The first thing that we have to do is to retrieve the documents from the Internet. In this case, we can use web scraping to extract documents from a website. I will scrape documents from kompas.com on sport category, especially on the popular articles. Because of the documents are using HTML format, we initialize a BeautifulSoup object to parse the HTML file, so we can extract each element that we want much easier.

我们要做的第一件事是从Internet检索文档。 在这种情况下，我们可以使用网络抓取从网站中提取文档。 我将从kompas.com上抓取有关体育类别的文档，尤其是有关热门文章的文档。 由于文档使用的是HTML格式，我们初始化了BeautifulSoup对象以解析HTML文件，因此我们可以轻松提取每个想要的元素。

Based on the figures below, I’ve shown the screenshot of the website with an inspect element to it. On Figure 1, I’ve shown the tags that we want to retrieve, which is the href attribute of the highlighted <a> tag with class “most__link”. On the Figure 2, We will retrieve text on <p> tags from <div> tag with class “read__content”.

根据下图，我显示了带有检查元素的网站屏幕截图。 在图1上，我展示了我们要检索的标签，它是突出显示的<a>标签的href属性，其类别为“ most__link”。 在图2中，我们将从<div>标记的类“ read__content”中的<p>标记上检索文本。

Figure 1, Figure 2 图1，图2

Here is the code that I used for extracting the documents and its explanations on each line,

这是我用于提取文档的代码及其每一行的说明，

import requestsfrom bs4 import BeautifulSoup# Make a request to the websiter = requests.get('https://bola.kompas.com/')# Create an object to parse the HTML formatsoup = BeautifulSoup(r.content, 'html.parser')# Retrieve all popular news links (Fig. 1)link = []for i in soup.find('div', {'class':'most__wrap'}).find_all('a'): i['href'] = i['href'] + '?page=all' link.append(i['href'])# For each link, we retrieve paragraphs from it, combine each paragraph as one string, and save it to documents (Fig. 2)documents = []for i in link: # Make a request to the link r = requests.get(i) # Initialize BeautifulSoup object to parse the content soup = BeautifulSoup(r.content, 'html.parser') # Retrieve all paragraphs and combine it as one sen = [] for i in soup.find('div', {'class':'read__content'}).find_all('p'): sen.append(i.text) # Add the combined paragraphs to documents documents.append(' '.join(sen))

清洁文件 (Clean the documents)

Right after we extract the documents, we have to clean it, so our retrieval process becomes much easier. For each document, we have to remove all unnecessary words, numbers and punctuations, lowercase the word, and remove the doubled space. Here is the code for it,

提取文档后，我们必须对其进行清理，因此我们的检索过程变得更加容易。 对于每个文档，我们必须删除所有不必要的单词，数字和标点符号，将单词小写，并删除加倍的空格。 这是它的代码，

import redocuments_clean = []for d in documents: # Remove Unicode document_test = re.sub(r'[^\x00-\x7F]+', ' ', d) # Remove Mentions document_test = re.sub(r'@\w+', '', document_test) # Lowercase the document document_test = document_test.lower() # Remove punctuations document_test = re.sub(r'[%s]' % re.escape(string.punctuation), ' ', document_test) # Lowercase the numbers document_test = re.sub(r'[0-9]', '', document_test) # Remove the doubled space document_test = re.sub(r'\s{2,}', ' ', document_test) documents_clean.append(document_test)

使用TF-IDF权重创建术语文档矩阵 (Create Term-Document Matrix with TF-IDF weighting)

After each document is clean, it’s time to create a matrix. Thankfully, scikit-learn library has prepared for us the code of it, so we don’t have to implement it from scratch. The code looks like this,

清理完每个文档后，该创建矩阵了。 幸运的是，scikit-learn库已经为我们准备了它的代码，因此我们不必从头开始实现它。 代码看起来像这样，

from sklearn.feature_extraction.text import TfidfVectorizer# Instantiate a TfidfVectorizer objectvectorizer = TfidfVectorizer()# It fits the data and transform it as a vectorX = vectorizer.fit_transform(docs)# Convert the X as transposed matrixX = X.T.toarray()# Create a DataFrame and set the vocabulary as the indexdf = pd.DataFrame(X, index=vectorizer.get_feature_names())

The result (matrix) will become a representation of the documents. By using that, we can find the similarity between different documents based on the matrix. The matrix looks like this,

结果(矩阵)将成为文档的表示形式。 通过使用它，我们可以基于矩阵找到不同文档之间的相似性。 矩阵看起来像这样，

Term-Document Matrix 期限文档矩阵

The matrix above is called as Term-Document Matrix. It consists of rows that represent by each token (term) from all documents, and the columns consist of the identifier of the document. Inside of the cell is the number of frequency of each word that is weighted by some number.

上面的矩阵称为术语文档矩阵。 它由代表所有文档中每个标记(术语)的行组成，而列则由文档的标识符组成。 单元内部是每个单词的频率(按某个数字加权)的数量。

We will use the column vector, which is a vector that represents each document to calculate the similarity with a given query. We can call this vector as embeddings.

我们将使用列向量，该向量是代表每个文档的向量，用于计算与给定查询的相似度。 我们可以将此向量称为嵌入。

For calculating the cell value, the code uses the TF-IDF method to do this. TF-IDF (Term Frequency — Inverse Document Frequency) is a frequency of a word that is weighted by IDF. Let me explain each one of them,

为了计算单元格值，代码使用TF-IDF方法执行此操作。 TF-IDF(术语频率-逆文档频率)是由IDF加权的单词频率。 让我解释一下每个

Term Frequency (TF) is a frequency of term (t) on document (d). The formula looks like this,

术语频率(TF)是文档(d)上术语(t)的频率。 公式看起来像这样

Beside of that, we can use a log with bases of 10 to calculate the TF, so the number becomes smaller, and the computation process becomes faster. Also, make sure to add one on it because we don’t want log 0 exist.

除此之外，我们可以使用以10为底的对数来计算TF，因此数量变得更小，并且计算过程变得更快。 另外，请确保在其上添加一个，因为我们不希望存在日志0。

Then, there is the Inverse Document Frequency (IDF). This formula will be used for calculating the rarity of the word in all documents. It will be used as weights for the TF. If a word is frequent, then the IDF will be smaller. In opposite, if the word is less frequent, then the IDF will be larger. The formula looks like this,

然后，存在反向文档频率(IDF) 。 此公式将用于计算所有文档中单词的稀有度。 它将用作TF的权重。 如果单词很频繁，则IDF会更小。 相反，如果单词较少出现，则IDF将更大。 公式看起来像这样

Recall the TF-IDF, we can see how does it affect the value on each cell. It will remove all the words that are frequently shown in documents but at the same time not important, such as and, or, even, actually, etc. Based on that, we use this as the value on each cell on our matrix.

回想一下TF-IDF，我们可以看到它如何影响每个单元格上的值。 它将删除文档中经常显示但同时并不重要的所有单词，例如and和乃至实际上，等等。基于此，我们将其用作矩阵上每个单元格的值。

使用余弦相似度计算相似度。 (Calculate the similarity using cosine similarity.)

After we create the matrix, we can prepare our query to find articles based on the highest similarity between the document and the query. To calculate the similarity, we can use the cosine similarity formula to do this. It looks like this,

创建矩阵后，我们可以准备查询以基于文档和查询之间的最高相似度来查找文章。 要计算相似度，我们可以使用余弦相似度公式执行此操作。 看起来像这样

The formula calculates the dot product divided by the multiplication of the length on each vector. The value ranges from [1, 0], but in general, the cosine value ranges from [-1, 1]. Because there are no negative values on it, we can ignore the negative value because it never happens.

该公式计算出点积除以每个向量上长度的乘积。 该值的范围是[1，0]，但通常，余弦值的范围是[-1，1]。 因为没有负值，所以我们可以忽略负值，因为它永远不会发生。

Now, we will implement the code to find similarities on documents based on a query. The first thing that we have to do is to transform the query as a vector on the matrix that we have. Then, we calculate the similarities between them. And finally, we retrieve all documents that have values above 0 in similarity. The code looks like this,

现在，我们将实现代码以基于查询查找文档的相似性。 我们要做的第一件事是将查询转换为我们拥有的矩阵上的向量。 然后，我们计算它们之间的相似度。 最后，我们检索相似度大于0的所有文档。 代码看起来像这样，

def get_similar_articles(q, df): print("query:", q) print("Berikut artikel dengan nilai cosine similarity tertinggi: ") # Convert the query become a vector q = [q] q_vec = vectorizer.transform(q).toarray().reshape(df.shape[0],) sim = {} # Calculate the similarity for i in range(10): sim[i] = np.dot(df.loc[:, i].values, q_vec) / np.linalg.norm(df.loc[:, i]) * np.linalg.norm(q_vec) # Sort the values sim_sorted = sorted(sim.items(), key=lambda x: x[1], reverse=True) # Print the articles and their similarity values for k, v in sim_sorted: if v != 0.0: print("Nilai Similaritas:", v) print(docs[k]) print()# Add The Queryq1 = 'barcelona'# Call the functionget_similar_articles(q1, df)

Suppose that we want to find articles that talk about Barcelona. If we run the code based on that, we will get the result like this,

假设我们想查找有关巴塞罗那的文章。 如果我们以此为基础运行代码，我们将得到如下结果：

query: barcelonaBerikut artikel dengan nilai cosine similarity tertinggi:Nilai Similaritas: 0.4641990113096689 kompas com perombakan skuad yang dilakukan pelatih anyar barcelona ronald koeman memakan korban baru terkini ronald koeman dikabarkan akan mendepak bintang muda barcelona yang baru berusia tahun riqui puig menurut media spanyol rac koeman sudah meminta riqui puig mencari tim baru karena tidak masuk dalam rencananya di barcelona rumor itu semakin kuat karena puig....Nilai Similaritas: 0.4254860197361395kompas com pertandingan trofeo joan gamper mempertemukan barcelona dengan salah satu tim promosi liga spanyol elche laga barcelona vs elche usai digelar di camp nou pada minggu dini hari wib trofeo joan gamper merupakan laga tahunan yang diadakan oleh barca kali ini sudah memasuki edisi ke blaugrana julukan tuan rumah menang dengan skor gol kemenangan barcelona....

最后的想法 (Final Thoughts)

That is how we can create a simple search engine using Python and its dependencies. It still very basic, but I hope you can learn something from here and can implement your own search engine based on what you need. Thank you.

这就是我们可以使用Python及其依赖关系创建简单的搜索引擎的方式。 它仍然很基础，但是我希望您可以从这里学到一些东西，并可以根据需要实现自己的搜索引擎。 谢谢。

翻译自: https://towardsdatascience.com/create-a-simple-search-engine-using-python-412587619ff5

相关资源：四史答题软件安装包exe