《Anomaly Detection-A survey》

    科技2023-10-24  99

    《Anomaly Detection-A survey》

    Anomaly Detection

    检测技术

    异常类型

    Point Anomaly

    数据点的异常,单个指标值,或者多个指标值形成的向量点。

    Contextual Anomaly

    每个数据点都包含两类属性:

    Contextual Attributes: 空间数据的经纬度、时间序列数据的时间戳、每个数据点附带的某些属性

    Behavior Attributes: 数据点的实际value值

    Collective Anomaly

    无法从单个数据点去判断是否异常,而是一段数据序列的集合型异常。

    标签与检测技术

    normal & anomalous

    detection techniques

    监督型异常检测:需要打了便签的训练集,根据训练的模型来检测。这部分涉及到异常的插入。

    半-监督型异常检测:针对normal建立模型。

    无监督型异常检测:

    应用场景

    结合很多实际的应用场景,简要介绍了各种场景的异常检测方法。

    整个第三部分都是在讲这个。

    基于分类的异常检测

    多分类:多个normal类,不在其中的为异常

    一分类:只有一个normal类型,不在其中的为异常

    多种方法

    基于神经网络基于贝叶斯网络基于SVM支持向量机基于规则

    优缺点

    基于近邻的异常检测

    使用距离、相似度作为衡量指标

    分成两类:

    距离相对密度

    前K个近邻

    *The anomaly score of a data instance is defifined as its distance to its k th nearest neighbor in a given data set

    相对密度

    Local Outlier Factor (LOF)

    Connectivity based Outlier Factor* (COF)

    计算复杂度与优缺点

    复杂度

    N的平方

    优缺点

    基于聚类的异常检测

    同样分成了三类,适用于三种场景。

    assumption1 : Normal data instances belong to a cluster in the data, while anomalies either do not belong to any cluster.

    assumption2:Normal data instances lie close to their closest cluster centroid, while anomalies are far away from their closest cluster centroid.

    assumption3:Normal data instances belong to large and dense clusters, while anomalies either belong to small or sparse clusters.

    聚类与近邻的区别

    The key difffference between the two techniques, however, is that clustering based techniques evaluate each instance with respect to the cluster it belongs to, while nearest neighbor based techniques analyze each instance with respect to its local neighborhood.

    计算复杂度与优缺点

    基于统计的异常检测

    基本定律:

    The underlying principle of any statistical anomaly detection technique is: “An anomaly is an observation which is suspected of being partially or wholly irrelevant because it is not generated by the stochastic model assumed”。

    统计异常检测的假设:

    Normal data instances occur in high probability regions of a stochastic model, while anomalies occur in the low probability regions of the stochastic model.

    Parametric Techniques

    Parametric techniques assume the knowledge of underlying distribution and estimate the parameters from the given data.

    高斯模型
    回归模型
    参数分布的混合

    Non-parametric Techniques

    Non-parametric techniques do not generally assume knowledge of underlying distribution.

    直方图
    kernel function

    计算复杂度与优缺点

    信息论异常检测

    Information theoretic techniques analyze the information content of a data set using difffferent information theoretic measures such as Kolomogorov Complexity, entropy, relative entropy, etc.

    Assumption:

    Anomalies in data induce irregularities in the information content of the data set.

    计算复杂度与优缺点

    频谱异常检测

    Spectral techniques try to find an approximation of the data using a combination of attributes that capture the bulk of variability in the data.

    Assumption:

    Data can be embedded into a lower dimensional subspace in which normal instances and anomalies appear signifificantly difffferent.

    Handling Contextual Anomaly

    The anomaly detection techniques discussed in the previous sections primarily focus on detecting point anomalies. In this section, we will discuss anomaly detection techniques that handle contextual anomalies.

    Reduction to Point Anomaly Detection Problem

    Utilizing the Structure in Data

    Handling Collective Anomaly

    Handling Sequential Anomaly

    Handling Spatial Anomaly

    异常检测技术对的相对优缺点

    结论

    Processed: 0.009, SQL: 8