数据点的异常,单个指标值,或者多个指标值形成的向量点。
每个数据点都包含两类属性:
Contextual Attributes: 空间数据的经纬度、时间序列数据的时间戳、每个数据点附带的某些属性
Behavior Attributes: 数据点的实际value值
无法从单个数据点去判断是否异常,而是一段数据序列的集合型异常。
normal & anomalous
监督型异常检测:需要打了便签的训练集,根据训练的模型来检测。这部分涉及到异常的插入。
半-监督型异常检测:针对normal建立模型。
无监督型异常检测:
结合很多实际的应用场景,简要介绍了各种场景的异常检测方法。
整个第三部分都是在讲这个。
多分类:多个normal类,不在其中的为异常
一分类:只有一个normal类型,不在其中的为异常
使用距离、相似度作为衡量指标
分成两类:
距离相对密度*The anomaly score of a data instance is defifined as its distance to its k th nearest neighbor in a given data set
Local Outlier Factor (LOF)
Connectivity based Outlier Factor* (COF)
N的平方
优缺点…
同样分成了三类,适用于三种场景。
assumption1 : Normal data instances belong to a cluster in the data, while anomalies either do not belong to any cluster.
assumption2:Normal data instances lie close to their closest cluster centroid, while anomalies are far away from their closest cluster centroid.
assumption3:Normal data instances belong to large and dense clusters, while anomalies either belong to small or sparse clusters.
The key difffference between the two techniques, however, is that clustering based techniques evaluate each instance with respect to the cluster it belongs to, while nearest neighbor based techniques analyze each instance with respect to its local neighborhood.
基本定律:
The underlying principle of any statistical anomaly detection technique is: “An anomaly is an observation which is suspected of being partially or wholly irrelevant because it is not generated by the stochastic model assumed”。
统计异常检测的假设:
Normal data instances occur in high probability regions of a stochastic model, while anomalies occur in the low probability regions of the stochastic model.
Parametric techniques assume the knowledge of underlying distribution and estimate the parameters from the given data.
Non-parametric techniques do not generally assume knowledge of underlying distribution.
Information theoretic techniques analyze the information content of a data set using difffferent information theoretic measures such as Kolomogorov Complexity, entropy, relative entropy, etc.
Assumption:
Anomalies in data induce irregularities in the information content of the data set.
Spectral techniques try to find an approximation of the data using a combination of attributes that capture the bulk of variability in the data.
Assumption:
Data can be embedded into a lower dimensional subspace in which normal instances and anomalies appear signifificantly difffferent.
The anomaly detection techniques discussed in the previous sections primarily focus on detecting point anomalies. In this section, we will discuss anomaly detection techniques that handle contextual anomalies.