Anomaly detection: Difference between revisions
No edit summary |
No edit summary |
||
| Line 1: | Line 1: | ||
{{see also|Machine learning terms}} | {{see also|Machine learning terms}} | ||
==Introduction== | ==Introduction== | ||
In [[machine learning]], [[anomaly detection]] is the process of recognizing [[examples]] in a [[dataset]] that deviate from normal behavior. These abnormal outcomes are known as anomalies, [[outlier]]s, or exceptions. As an example, if the mean for a certain [[feature]] is 50 with a standard deviation of 5, then anomaly detection should flag a value of 300 as suspicious. | |||
==Types of Anomalies== | ==Types of Anomalies== | ||
Anomalies can be divided into three primary categories: point anomalies, contextual anomalies and collective anomalies. | Anomalies can be divided into three primary categories: [[point anomalies]], [[contextual anomalies]] and [[collective anomalies]]. | ||
===Point Anomalies=== | ===Point Anomalies=== | ||
Point anomalies, also referred to as global anomalies, refer to individual data points that differ significantly from the majority. Examples of point anomalies include credit card fraud, sensor | [[Point anomalies]], also referred to as [[global anomalies]], refer to individual data points ([[examples]]) that differ significantly from the majority. Examples of point anomalies include [[credit card fraud]], [[sensor glitch]]es and [[network intrusion]]s. They can be detected using [[statistical methods]] like the [[z-score]], [[interquartile range]] or [[Mahalanobis distance]]; or [[machine learning]] techniques like [[isolation forest]], [[one-class SVM]] or [[autoencoder]]. | ||
===Contextual Anomalies=== | ===Contextual Anomalies=== | ||
Contextual anomalies, also referred to as conditional anomalies, refer to data points that are anomalous only within certain contexts or subpopulations of the data. For instance, a high heart rate may be considered normal during physical exercise but abnormal when sleeping. To detect contextual anomalies, context information must be integrated into the equation; this can be done through rule-based | [[Contextual anomalies]], also referred to as [[conditional anomalies]], refer to data points that are anomalous only within certain contexts or subpopulations of the data. For instance, a high heart rate may be considered normal during physical exercise but abnormal when sleeping. To detect contextual anomalies, context information must be integrated into the equation; this can be done through [[rule-based system]]s, [[Bayesian network]]s or [[decision tree]]s. | ||
===Collective Anomalies=== | ===Collective Anomalies=== | ||
Collective anomalies, also referred to as group anomalies, refer to a collection of data points that exhibit unusual behavior when taken together but not individually. Examples include sudden spikes in web traffic or power outages in an area. To detect collective anomalies requires the detection of patterns or dependencies between data points and identification of subpopulations that show anomalous | [[Collective anomalies]], also referred to as [[group anomalies]], refer to a collection of data points that exhibit unusual behavior when taken together but not individually. Examples include sudden spikes in web traffic or power outages in an area. To detect collective anomalies requires the detection of patterns or dependencies between data points and the identification of subpopulations that show anomalous behavior. [[Clustering]], [[principal component analysis]] or [[local outlier factor]] can all be utilized for detection. | ||
==Challenges of Anomaly Detection== | ==Challenges of Anomaly Detection== | ||
| Line 19: | Line 19: | ||
===Data Imbalance=== | ===Data Imbalance=== | ||
One of the major obstacles lies in data imbalance, where anomalies make up a small fraction of all instances compared to normal data points. This makes it difficult for machine learning models to learn characteristics | One of the major obstacles lies in [[data imbalance]], where anomalies make up a small fraction of all instances compared to normal data points. This makes it difficult for machine learning models to learn the characteristics of anomalies and distinguish them from regular instances. | ||
===Labeling=== | ===Labeling=== | ||
Another challenge lies in labeling, where labeled anomalies may be scarce or unavailable and the definition of what constitutes an anomaly may be uncertain or context dependent. To address this, unsupervised or semi-supervised techniques that do not require labeled data may be utilized instead, along with expert knowledge and feedback to refine the definition of anomalies. | Another challenge lies in [[labeling]], where labeled anomalies may be scarce or unavailable and the definition of what constitutes an anomaly may be uncertain or context dependent. To address this, [[unsupervised]] or [[semi-supervised]] techniques that do not require labeled data may be utilized instead, along with expert knowledge and feedback to refine the definition of anomalies. | ||
===High Dimensionality=== | ===High Dimensionality=== | ||
Anomaly detection often faces the problem of high dimensionality, where data may contain many features or variables that make it challenging to detect anomalies and visualize them. To address this challenge, feature selection, dimensionality reduction techniques or visualization strategies can be employed in order to simplify the data and focus on the most pertinent ones. | Anomaly detection often faces the problem of [[high dimensionality]], where data may contain many features or variables that make it challenging to detect anomalies and visualize them. To address this challenge, feature selection, dimensionality reduction techniques or visualization strategies can be employed in order to simplify the data and focus on the most pertinent ones. | ||
===Concept Drift=== | ===Concept Drift=== | ||
Another difficulty is concept drift, in which the distribution of data alters over time and makes a model outdated or ineffective at detecting new anomalies. To combat this problem, adaptive or online learning techniques such as reinforcement learning should be utilized that update models in real-time or adapt to changes in data distribution. | Another difficulty is [[concept drift]], in which the distribution of data alters over time and makes a model outdated or ineffective at detecting new anomalies. To combat this problem, [[adaptive learning|adaptive]] or [[online learning]] techniques such as [[reinforcement learning]] should be utilized that update models in real-time or adapt to changes in data distribution. | ||
==Applications== | ==Applications== | ||
Anomaly detection is used in many fields to detect and prevent potentially hazardous events. Some of its applications include: | Anomaly detection is used in many fields to detect and prevent potentially hazardous events. Some of its applications include: | ||
#[[Fraud detection]]: In finance, anomaly detection is employed to spot fraudulent transactions. For instance, a credit card company could utilize anomaly detection to spot purchases that deviate from typical spending patterns of an individual customer. | |||
#[[Network intrusion detection]]: Anomaly detection can be employed to detect network intrusions by monitoring network traffic for any deviations from normal behavior that might indicate an attack. This method monitors network activity to detect anomalies that might indicate an attack has taken place. | |||
#[[Fault detection]] in industrial systems: Industrial systems utilize anomaly detection to identify faults in equipment. For instance, a manufacturing plant can utilize anomaly detection to recognize when a machine deviates from its usual operating parameters. | |||
==Techniques== | ==Techniques== | ||
There are various techniques used for anomaly detection, such as statistical | There are various techniques used for anomaly detection, such as [[statistical method]]s, [[machine learning]] [[algorithm]]s and [[data mining]] methods. Some of the most popular ones include: | ||
#Statistical Methods: Statistical methods assume the data follows a particular distribution, such as a [[Gaussian distribution]]. They calculate the mean and standard deviation of the data points to identify those that deviate significantly from it. | |||
#Machine Learning Algorithms: Machine learning algorithms such as [[clustering]], [[classification]] and [[deep learning]] can be employed for anomaly detections [[Clustering]] algorithms identify clusters of similar data points; any that do not belong to one are considered anomalies. [[Classification]] algorithms label data as normal or abnormal and train their algorithm to recognize patterns associated with each label. [[Deep learning]] techniques like [[autoencoder]]s also work to detect anomalies by reconstructing input data and comparing its reconstruction error with a threshold value. | |||
#Data Mining Techniques: Data mining techniques such as [[association rule mining]] and [[deviation detection]] can be employed for anomaly detection. Association rule mining helps identify relationships between variables in the data, while deviation detection identifies data points that deviate significantly from expected values. | |||
==Explain Like I'm 5 (ELI5)== | ==Explain Like I'm 5 (ELI5)== | ||
| Line 72: | Line 56: | ||
[[Category:Terms]] [[Category:Machine learning terms | [[Category:Terms]] [[Category:Machine learning terms]] | ||