Figure 1 - uploaded by Zhengzhang Chen
Content may be subject to copyright.
Source publication
Given a large number of low-quality heterogeneous categorical alerts collected from an anomaly detection system, how to characterize the complex relationships between different alerts and deliver trustworthy rankings to end users? While existing techniques focus on either mining alert patterns or filtering out false positive alerts, it can be more...
Contexts in source publication
Context 1
... interaction/events (e.g., a process accesses a sensitive file) and generate isolated alerts, while a system's ab- normal behaviors are typically high-level activities composed of multiple low-level events/steps [12]. For example, in an enterprise security system, a well-known network attack called Advanced Persistent Threat (APT), as shown in Fig. 1, includes a sequence of computer hacking processes. Usually, the first attempt is to gain a foothold in the environment. Then, it uses the compromised systems as access to the target network, which is followed by de- ploying additional tools to fulfill the attack. Thus, it can be difficult to identify truly relevant processes/alerts to ...
Context 2
... instance, an ATP attack shown in Figure 1 can be deemed to be an alert pattern, consisting of a number of steps/alerts in it. Each alert indicates one step in the attack. ...
Context 3
... interaction/events (e.g., a process accesses a sensitive file) and generate isolated alerts, while a system's ab- normal behaviors are typically high-level activities composed of multiple low-level events/steps [12]. For example, in an enterprise security system, a well-known network attack called Advanced Persistent Threat (APT), as shown in Fig. 1, includes a sequence of computer hacking processes. Usually, the first attempt is to gain a foothold in the environment. Then, it uses the compromised systems as access to the target network, which is followed by de- ploying additional tools to fulfill the attack. Thus, it can be difficult to identify truly relevant processes/alerts to ...
Similar publications
Actors of the Cooperative Intelligent Transport Systems (C-ITS) generate various amounts of data. Useful information on various issues such as anomalies, failures, road profiles, etc., could be revealed from the analysis of these data. The analysis, could be managed by operators and vehicles, and its output could be very helpful for future decision...
Advanced Metering Infrastructure (AMI) is a component of electrical networks that combines the energy and telecommunication infrastructure to collect, measure and analyze consumer energy consumptions. One of the main elements of AMI is a smart meter that used to manage electricity generation and distribution to end-user. The rapid implementation of...
Citations
... However, the system entities linked to the KPI may not always be the root causes. This is because the malfunctioning effects will spread to neighboring entities starting from the root causes [11,13,33]. We present a random walk-based method for capturing such patterns and more precisely locating root causes. ...
The task of root cause analysis (RCA) is to identify the root causes of system faults/failures by analyzing system monitoring data. Efficient RCA can greatly accelerate system failure recovery and mitigate system damages or financial losses. However, previous research has mostly focused on developing offline RCA algorithms, which often require manually initiating the RCA process, a significant amount of time and data to train a robust model, and then being retrained from scratch for a new system fault.
In this paper, we propose CORAL, a novel online RCA framework that can automatically trigger the RCA process and incrementally update the RCA model. CORAL consists of Trigger Point Detection, Incremental Disentangled Causal Graph Learning, and Network Propagation-based Root Cause Localization. The Trigger Point Detection component aims to detect system state transitions automatically and in near-real-time. To achieve this, we develop an online trigger point detection approach based on multivariate singular spectrum analysis and cumulative sum statistics. To efficiently update the RCA model, we propose an incremental disentangled causal graph learning approach to decouple the state-invariant and state-dependent information. After that, CORAL applies a random walk with restarts to the updated causal graph to accurately identify root causes. The online RCA process terminates when the causal graph and the generated root cause list converge. Extensive experiments on three real-world datasets demonstrate the effectiveness and superiority of the proposed framework.
... Firstly, existing works [2], [3] mainly focus on binary classification which are coarse-grained. Coarse-grained alerts cannot pragmatically reduce the number of alerts examined by security personnel. ...
... Especially, when dealing with a huge number of alerts, existing approaches cannot effectively reduce the workload of security personnel. For example, Lin et al. [2] classify alerts into alerts triggered by attacks and by nonattacks, respectively. They believe that alerts triggered by attacks are all important high-threats. ...
... High-threat correlation graphs are discovered by calculating the similarity among correlation graphs. Lin et al. [2] jointly model alert temporal dependencies and textual dependencies to discover actual attacks in alerts. Temporal dependence exploits Bayes' rule and prefix tree to extract attack patterns. ...
... Physical attack events and attack sequences are correlated using topological time correlation, frequent attack sequence patterns are extracted, and hidden patterns are discovered from alarm logs. Ying Lin et al. [165] propose a collaborative alert ranking framework (CAR) that uses both time correlation and alert content correlation. CAR builds a hierarchical Bayesian model to capture short-term and longterm dependencies in alert sequences. ...
Security event correlation approaches are necessary to detect and predict incremental threats such as multi-step or targeted attacks (advanced persistent threats) and other causal sequences of abnormal events. The use of security event correlation techniques also makes it possible to reduce the volume of the original data stream by grouping the events and eliminating their redundancy. The variety of event correlation methods, in turn, requires choosing the most appropriate way to handle security events, depending on the purpose and available resources. This paper presents a systematization of security event correlation methods into several categories, such as publication year, applied correlation methods, knowledge extraction methods, used data sources, architectural solutions, and quality evaluation of correlation methods. The research method is a systematic literature review, which includes the formulation of research questions, the choice of keywords and criteria for inclusion and exclusion. The review corpus is formed by using search queries in Google Scholar, IEEE Xplore, ACM Digital Library, ScienceDirect, and selection criteria. The final review corpus includes 127 publications from the existing literature for 2010–2021 and reflects the current state of research in the security event correlation field. The results of the analysis include the main directions of research in the field of event correlation and methods used for correlation both single events and their sequences in attack scenarios. The review also describes the datasets and metrics used to evaluate security event correlation approaches. In conclusion, the existing problems and possible ways to overcome them are identified. The main contribution of the review is the most complete classification and comparison of existing approaches to the security event correlation, considered not only from the point of view of the algorithm, but also the possibility of unknown attack detection, architectural solutions and the use of event initial data.
... The attacks were performed by professional hackers hired by the company. We choose six typical attacks [5,31] in the following: (1) Diversifying Attack Vectors. This intrusion scenario is a sixstep attack chain [5], as shown in Figure 4. First, hackers create malicious php files, download malware binary (trojan.exe), ...
Detecting anomalies in dynamic graphs is a vital task, with numerous practical applications in areas such as security, finance, and social media. Existing network embedding based methods have mostly focused on learning good node representations, whereas largely ignoring the subgraph structural changes related to the target nodes in a given time window. In this paper, we propose StrGNN, an end-to-end structural temporal Graph Neural Network model for detecting anomalous edges in dynamic graphs. In particular, we first extract the h-hop enclosing subgraph centered on the target edge and propose a node labeling function to identify the role of each node in the subgraph. Then, we leverage the graph convolution operation and Sortpooling layer to extract the fixed-size feature from each snapshot/timestamp. Based on the extracted features, we utilize the Gated Recurrent Units to capture the temporal information for anomaly detection. We fully implement StrGNN and deploy it into a real enterprise security system, and it greatly helps detect advanced threats and optimize the incident response. Extensive experiments on six benchmark datasets also demonstrate the effectiveness of StrGNN.
... Although the recent years have witnessed significant progress of anomaly detection techniques [4], [5], [6], [7], [8], [9], [10], the rise of big data has introduced new challenges for the design of efficient and accurate anomaly detection approaches. First, a real complex system typically deals with a large volume of system event data (normally thousands of events per second). ...
Anomaly detection has been widely applied in modern data-driven security applications to detect abnormal events/entities that deviate from the majority. However, less work has been done in terms of detecting suspicious event sequences/paths, which are better discriminators than single events/entities for distinguishing normal and abnormal behaviors in complex systems such as cyber-physical systems. A key and challenging step in this endeavor is how to discover those abnormal event sequences from millions of system event records in an efficient and accurate way. To address this issue, we propose NINA, a network diffusion based algorithm for identifying anomalous event sequences. Experimental results on both static and streaming data show that NINA is efficient (processes about 2 million records per minute) and accurate.
... In this section, we first present the preliminaries, then define the machine learning problem that we are concerned with Deep Program Reidentification. System Entity There are three main types of system entities in an operating system [6,16]: processes, files, and Internet sockets (INETSockets). And each entity is associated with a set of categorical attributes. ...
... Similar to [6,16], we evaluate the performance of different methods using a variety of measures, including accuracy (ACC), F-1 score, AUC score, precision, and recall. ...
... Existing anomaly detection approaches in large-scale enterprise network systems have been separately considering different data representations. In particular, host-based anomaly detection methods [6,10,16] locally extract patterns from process-level events as the discriminators of abnormal intrusion. In contrast, network-based anomaly detection methods [18] focus on disclosing abnormal subgraph structures from network-level events, most of which are inspired by graph properties. ...
Program or process is an integral part of almost every IT/OT system. Can we trust the identity/ID (e.g., executable name) of the program? To avoid detection, malware may disguise itself using the ID of a legitimate program, and a system tool (e.g., PowerShell) used by the attackers may have the fake ID of another common software, which is less sensitive. However, existing intrusion detection techniques often overlook this critical program reidentification problem (i.e., checking the program's identity). In this paper, we propose an attentional heterogeneous graph neural network model (DeepHGNN) to verify the program's identity based on its system behaviors. The key idea is to leverage the representation learning of the heterogeneous program behavior graph to guide the reidentification process. We formulate the program reidentification as a graph classification problem and develop an effective attentional heterogeneous graph embedding algorithm to solve it. Extensive experiments using real-world enterprise monitoring data and real attacks-demonstrate the effectiveness of DeepHGNN across multiple popular metrics and the robustness to the normal dynamic changes like program version upgrades.
... In this section, we first present the preliminaries, then define the machine learning problem that we are concerned with Deep Program Reidentification. System Entity There are three main types of system entities in an operating system [6,16]: processes, files, and Internet sockets (INETSockets). And each entity is associated with a set of categorical attributes. ...
... Similar to [6,16], we evaluate the performance of different methods using a variety of measures, including accuracy (ACC), F-1 score, AUC score, precision, and recall. ...
... Existing anomaly detection approaches in large-scale enterprise network systems have been separately considering different data representations. In particular, host-based anomaly detection methods [6,10,16] locally extract patterns from process-level events as the discriminators of abnormal intrusion. In contrast, network-based anomaly detection methods [18] focus on disclosing abnormal subgraph structures from network-level events, most of which are inspired by graph properties. ...
Program or process is an integral part of almost every IT/OT system. Can we trust the identity/ID (e.g., executable name) of the program? To avoid detection, malware may disguise itself using the ID of a legitimate program, and a system tool (e.g., PowerShell) used by the attackers may have the fake ID of another common software, which is less sensitive. However, existing intrusion detection techniques often overlook this critical program reidentification problem (i.e., checking the program's identity). In this paper, we propose an attentional heterogeneous graph neural network model (DeepHGNN) to verify the program's identity based on its system behaviors. The key idea is to leverage the representation learning of the heterogeneous program behavior graph to guide the reidentification process. We formulate the program reidentification as a graph classification problem and develop an effective attentional heterogeneous graph embedding algorithm to solve it. Extensive experiments --- using real-world enterprise monitoring data and real attacks --- demonstrate the effectiveness of DeepHGNN across multiple popular metrics and the robustness to the normal dynamic changes like program version upgrades.
... In this section, we first present the preliminaries, then define the machine learning problem that we are concerned with Deep Program Reidentification. System Entity There are three main types of system entities in an operating system [6,16]: processes, files, and Internet sockets (INETSockets). And each entity is associated with a set of categorical attributes. ...
... Similar to [6,16], we evaluate the performance of different methods using a variety of measures, including accuracy (ACC), F-1 score, AUC score, precision, and recall. ...
... Existing anomaly detection approaches in large-scale enterprise network systems have been separately considering different data representations. In particular, host-based anomaly detection methods [6,10,16] locally extract patterns from process-level events as the discriminators of abnormal intrusion. In contrast, network-based anomaly detection methods [18] focus on disclosing abnormal subgraph structures from network-level events, most of which are inspired by graph properties. ...
Program or process is an integral part of almost every IT/OT system. Can we trust the identity/ID (e.g., executable name) of the program? To avoid detection, malware may disguise itself using the ID of a legitimate program, and a system tool (e.g., PowerShell) used by the attackers may have the fake ID of another common software, which is less sensitive. However, existing intrusion detection techniques often overlook this critical program reidentification problem (i.e., checking the program's identity). In this paper, we propose an attentional multi-channel graph neural network model (DeepRe-ID) to verify the program's identity based on its system behaviors. The key idea is to leverage the representation learning of the program behavior graph to guide the reidentification process. We formulate the program reidentification as a graph classification problem and develop an effective multi-channel attentional graph embedding algorithm to solve it. Extensive experiments --- using real-world enterprise monitoring data and real attacks --- demonstrate the effectiveness of DeepRe-ID across multiple popular metrics and the robustness to the normal dynamic changes like program version upgrades.
Keeping high service levels of a fast-growing number of servers is crucial and challenging for IT operations teams. Online monitoring systems trigger many occurrences that experts find hard to keep up with. In addition, most of the triggered warnings do not correspond to real, critical problems, making it difficult for technicians to know which to focus on and address in a timely manner. Outlier and concept drift detection techniques can be applied to multiple streams of readings related to server monitoring metrics, but they also generate many False Positives. Ranking algorithms can already prioritize relevant results in information retrieval and recommender systems. However, these approaches are supervised, making them inapplicable in event detection on data streams. We propose a framework that combines event aggregations and uses a customized clustering algorithm to score and rank alarms in the context of IT operations. To the best of our knowledge, this is the first unsupervised, online, high-dimensional approach to rank IT ops events and contributes to advancing knowledge about associated key concepts and challenges of this problem.
