python如何验证数据库

动机(Motivation)

Your script works with the training data, but when you use that script for a new but supposedly similar data, you run into an error. What is going on? That might be because the structure of your data is not like what you expected.

您的脚本可以与训练数据一起使用，但是当您将该脚本用于新的但据说相似的数据时，会遇到错误。 到底是怎么回事？ 那可能是因为您的数据结构与您期望的不一样。

But it might be difficult for you to take a look at every row of your new data to find out where the problem could be. It can also be time-consuming to manually analyze your data every time the new data is used.

但是，您可能很难查看新数据的每一行以找出问题所在。 每次使用新数据时，手动分析数据也可能很耗时。

It is even worse if your code does not throw any error but the data changes. As the result, the performance of your model might get worse because the data is different from what you expected.

如果您的代码没有引发任何错误，但是数据发生了变化，那就更糟了。 结果，由于数据与预期不同，因此模型的性能可能会变差。

If we can write a test for functions with tools such as Pytest, is there a way to can write a test for data as well?

如果我们可以使用诸如Pytest之类的工具编写针对功能的测试，是否还有办法针对数据编写测试？

We can do that with schema. This article will show you how to use schema in a variety of scenarios.

我们可以使用模式来做到这一点。 本文将向您展示如何在各种情况下使用架构。

什么是架构？ (What is schema?)

Schema is a library for validating Python data structures.

Schema是用于验证Python数据结构的库。

Install schema with

使用安装架构

pip install schema

We will use faker to create data that is a list of dictionary. Faker is a Python library that enables us to create fake data with ease. I wrote about how to use faker here.

我们将使用fakerr创建作为字典列表的数据。 Faker是一个Python库，使我们能够轻松创建虚假数据。 我在这里写了关于如何使用造假者的文章。

Imagine this is data that presents the information about your friends.

想象一下，这是显示您朋友信息的数据。

!pip install faker from faker import Faker import numpy as np Faker.seed(0) fake = Faker() def create_data(x: int): '''Create fake data'''' # dictionary data = [] for i in range(0, x): data_i = {} data_i['name']= fake.name() data_i['city']= fake.city() data_i['closeness (1-5)'] = np.random.randint(1,5) data_i['extrovert'] = fake.pybool() data_i['favorite_temperature'] = fake.pyfloat(left_digits=2, right_digits=2) data.append(data_i) return data data = create_data(3) [{'name': 'Norma Fisher', 'city': 'South Richard', 'closeness (1-5)': 4, 'extrovert': True, 'favorite_temperature': -45.74}, {'name': 'Colleen Taylor', 'city': 'North Laurenshire', 'closeness (1-5)': 4, 'extrovert': False, 'favorite_temperature': 93.9}, {'name': 'Melinda Kennedy', 'city': 'South Cherylside', 'closeness (1-5)': 1, 'extrovert': True, 'favorite_temperature': 66.33}]

架构入门 (Get Started with Schema)

验证数据类型(Validate Data Types)

We can use schema to validate data types such as below.

我们可以使用模式来验证如下数据类型。

from schema import Schema schema = Schema([{'name': str, 'city': str, 'closeness (1-5)': int, 'extrovert': bool, 'favorite_temperature': float}]) schema.validate(data) [{'name': 'Norma Fisher', 'city': 'South Richard', 'closeness (1-5)': 4, 'extrovert': True, 'favorite_temperature': -45.74}, {'name': 'Colleen Taylor', 'city': 'North Laurenshire', 'closeness (1-5)': 4, 'extrovert': False, 'favorite_temperature': 93.9}, {'name': 'Melinda Kennedy', 'city': 'South Cherylside', 'closeness (1-5)': 1, 'extrovert': True, 'favorite_temperature': 66.33}]We want to make sure that ‘name’, ‘city’ columns are string type, ‘closeness (1–5)’ are

Since schema returns the output without throwing any error, we know that our data is valid.

由于架构返回输出而不会引发任何错误，因此我们知道我们的数据有效。

Let’s see what happens if the data types are not like what we expect

让我们看看如果数据类型不符合我们的预期会发生什么

schema = Schema([{'name': int, 'city': str, 'closeness (1-5)': int, 'extrovert': bool, 'favorite_temperature': float}]) schema.validate(data) SchemaError: Or({'name': <class 'int'>, 'city': <class 'str'>, 'closeness (1-5)': <class 'int'>, 'extrovert': <class 'bool'>, 'favorite_temperature': <class 'float'>}) did not validate {'name': 'Norma Fisher', 'city': 'South Richard', 'closeness (1-5)': 3, 'extrovert': True, 'favorite_temperature': -45.74}Key 'name' error:'Norma Fisher' should be instance of 'int'

From the error, we know exactly which column and value of the data are different from what we expect. Thus, we can go back to the data to fix or delete that value.

从错误中，我们确切地知道数据的哪一列和哪些值与我们期望的不同。 因此，我们可以返回到数据来修复或删除该值。

If all you care about is whether the data is valid or not, use

如果您只关心数据是否有效，请使用

schema.is_valid(data)

This will return True if the data is as expected or False otherwise.

如果数据符合预期，则返回True否则返回False 。

在忽略其余列的同时验证某些列的数据类型 (Validate datatype of some columns while ignoring the rest)

But what if we don’t care about the data types of all of the columns but just care about the value of some columns? We can specify that with str: object

但是，如果我们不在乎所有列的数据类型，而仅在乎某些列的值，该怎么办？ 我们可以用str: object来指定

schema = Schema([{'name': str, 'city': str, 'favorite_temperature': float, str: object }]) schema.is_valid(data) Output: True

As you can see, we try to validate the data types of ‘name’, ‘city’, and ‘favorite_temperature’, while ignoring the data types of the rest of the features in our data.

如您所见，我们尝试验证“名称”，“城市”和“收藏夹温度”的数据类型，同时忽略数据中其余功能的数据类型。

The data is valid because the data types of the 3 features specified are correct.

该数据有效，因为指定的3个要素的数据类型正确。

通过功能验证 (Validate with Function)

What if we want to determine whether the data within a column satisfies a specific condition that is not relevant to data types such as the range of the values in a column?

如果我们要确定列中的数据是否满足与数据类型(例如，列中的值的范围)无关的特定条件，该怎么办？

Schema allows you to use a function to specify the condition for your data.

架构允许您使用函数来指定数据条件。

If we want to check whether the values in the ‘closeness’ column is between 1 and 5, we can use lambda like below

如果我们要检查“接近度”列中的值是否在1到5之间，可以使用如下所示的lambda

schema = Schema([{'name': str, 'city': str, 'favorite_temperature': float, 'closeness (1-5)': lambda n : 1 <= n <= 5, str: object }]) schema.is_valid(data) Output: True

As you can see, we specify n,the value in each row of the column ‘closeness’, to between 1 and 5 with lambda n: 1 <= n <=5. Neat!

如您所见，我们将“封闭性”列的每一行中的值n,指定为1到5之间的值， lambda n: 1 <= n <=5. 整齐！

验证几种模式 (Validate Several Schemas)

And

和

What if you want to make sure your ‘closeness’ column to be between 1 and 5 and the data type to be an integer?

如果您想确保“ closeness”列在1到5之间并且数据类型为整数，该怎么办？

That is when And comes in handy

那是当And派上用场的时候

schema = Schema([{'name': str, 'city': str, 'favorite_temperature': float, 'closeness (1-5)': And(lambda n : 1 <= n <= 5, float), str: object }]) schema.is_valid(data) Output: False

While all the values are within 1 and 5, the data type is not a float. Because one of the conditions is not satisfied, the data is not valid

虽然所有值都在1到5之间，但数据类型不是浮点型。 由于不满足条件之一，因此数据无效

Or

要么

If we want the data of column to be valid if either of the conditions is satisfied, we can use Or

如果我们希望满足以下任一条件，则列的数据有效，则可以使用Or

For example, if we want the city’s name to contain either 1 or 2 words, we can use

例如，如果我们希望城市名称包含1个或2个单词，则可以使用

schema = Schema([{'name': str, 'city': Or(lambda n: len(n.split())==2, lambda n: len(n.split()) ==1), 'favorite_temperature': float, 'closeness (1-5)': int, str: object }]) schema.is_valid(data)

Combination of And and Or

与和或的组合

What if we want the data type of ‘city’ to be a string but the length can be either 1 or 2? Luckily, this could be handled easily by combining And and Or

如果我们希望'city'的数据类型为字符串但长度可以为1或2怎么办？ 幸运的是，将And和Or组合在一起可以轻松解决

schema = Schema([{'name': str, 'city': And(str, Or(lambda n: len(n.split())==2, lambda n: len(n.split()) ==1)), 'favorite_temperature': float, 'closeness (1-5)': int, str: object }]) schema.is_valid(data) Output: True

可选的 (Optional)

What if we don’t have the detailed information about some of your friends?

如果我们没有有关您的某些朋友的详细信息怎么办？

data.append({'name': fake.name(), 'city': fake.city(), 'closeness (1-5)' : np.random.randint(1,5)}) data [{'name': 'Norma Fisher', 'city': 'South Richard', 'closeness (1-5)': 4, 'detailed_info': {'favorite_color': 'Pink', 'phone number': '7593824219489'}}, {'name': 'Emily Blair', 'city': 'Suttonview', 'closeness (1-5)': 4, 'detailed_info': {'favorite_color': 'Chartreuse', 'phone number': '9387784080160'}}, {'name': 'Samantha Cook', 'city': 'Janeton', 'closeness (1-5)': 3}]

Since the ‘detailed_info’ of Samantha Cook is not available with all of your friends, we want to make this column optional. Schema allows us to set that condition with Optional

由于萨曼莎·库克(Samantha Cook)的“ detailed_info”不适用于您的所有朋友，因此我们希望将此列设为可选。 模式允许我们使用Optional设置该条件

schema = Schema([{'name': str, 'city':str, 'closeness (1-5)': int, Optional('detailed_info'): {'favorite_color': str, 'phone number': str} }]) schema.is_valid(data) Output: True

禁止的 (Forbidden)

Sometimes, we might also want to make sure a certain kind of data is not in our data, such as private information. We can specify which column is forbidden with Forbidden

有时，我们可能还想确保某种类型的数据不在我们的数据中，例如私人信息。 我们可以指定哪些列，严禁和Forbidden

from schema import Forbidden schema = Schema([{'name': str, 'city':str, 'closeness (1-5)': int, Forbidden('detailed_info'): dict }]) schema.validate(data) Forbidden key encountered: 'detailed_info' in {'name': 'Norma Fisher', 'city': 'South Richard', 'closeness (1-5)': 4, 'detailed_info': {'favorite_color': 'Pink', 'phone number': '7593824219489'}}

Now we are aware of the existence of the forbidden column every time schema throws an error!

现在我们知道每次模式抛出错误时，禁止列的存在！

嵌套字典 (Nested dictionary)

So far, schema has enabled us to perform many sophisticated validations in several lines of code. But in the real-life, we might deal with a more sophisticated data structure than the example above.

到目前为止，模式已使我们能够在几行代码中执行许多复杂的验证。 但是在现实生活中，我们可能会处理比上面的示例更复杂的数据结构。

Can we use it for data with a more complicated structure? Such as a dictionary within a dictionary? Yes we can

我们可以将其用于结构更复杂的数据吗？ 如字典内的字典？ 我们可以

We create another data that contains a nested dictionary

我们创建另一个包含嵌套字典的数据

fake = Faker() Faker.seed(0) def create_data(x): # dictionary data = [] for i in range(0, x): data_i = {} data_i['name']= fake.name() data_i['city']= fake.city() data_i['closeness (1-5)'] = np.random.randint(1,5) data_i['detailed_info'] = {'favorite_color': fake.color_name(), 'phone number': fake.msisdn()} data.append(data_i) return data data = create_data(2) >>> data[{'name': 'Norma Fisher', 'city': 'South Richard', 'closeness (1-5)': 4, 'detailed_info': {'favorite_color': 'Pink', 'phone number': '7593824219489'}}, {'name': 'Emily Blair', 'city': 'Suttonview', 'closeness (1-5)': 4, 'detailed_info': {'favorite_color': 'Chartreuse', 'phone number': '9387784080160'}}]

Now we validate with a nested dictionary

现在我们使用嵌套字典进行验证

schema = Schema([{'name': str, 'city':str, 'closeness (1-5)': int, 'detailed_info': {'favorite_color': str, 'phone number': str} }]) schema.is_valid(data)

The syntax is straight forward! We just need to write another dictionary within the dictionary and specify the data type for each key.

语法很简单！ 我们只需要在字典中编写另一个字典并为每个键指定数据类型。

转换数据类型 (Convert Data Type)

Not only can schema be used to validate data but also can be used to convert the data type if it happens not to be like what we expected!

模式不但可以用来验证数据，而且还可以用来转换数据类型(如果它与我们期望的不一样)！

For example, we can convert string ‘123’ to integer 123 with Use(int)

例如，我们可以Use(int)将字符串“ 123”转换为整数123

>>> Schema(Use(int)).validate('123')123

结论 (Conclusion)

Congratulations! You have just learned how to validate and convert your data structures with schema. If you want your code to be reproducible, it is not only necessary to test for the code but also necessary to test your data. If you are looking for a way to validate your data, give this tool a try. It is not only helpful but also easy to use.

恭喜你！ 您刚刚学习了如何使用模式验证和转换数据结构。 如果您希望代码可重现，则不仅需要测试代码，而且还必须测试数据。 如果您正在寻找一种验证数据的方法，请尝试使用此工具。 它不仅有用，而且易于使用。

Source code with more examples of schema could be found here.

带有更多模式示例的源代码可以在这里找到。

Check out the doc of schema if you want to find more ways to validate your data.

如果您想找到更多验证数据的方法，请查看架构文档。

I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.

我喜欢写有关基本数据科学概念的文章，并喜欢使用不同的算法和数据科学工具。 您可以在LinkedIn和Twitter上与我联系。

Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these:

如果您想查看我编写的所有文章的代码，请给此回购加注星号。 在“ Medium”上关注我，以随时了解最新的数据科学文章，例如：

翻译自: https://towardsdatascience.com/introduction-to-schema-a-python-libary-to-validate-your-data-c6d99e06d56a

python如何验证数据库

相关资源：Python-voluptuous一个Python数据验证库