Back to the tech blog overview

The Many Security Usages of Anomaly Detection

The most common use of network infrastructure is to facilitate legitimate communication between two parties across the network. Unfortunately, the same network infrastructure provides an opportunity for malicious communications.

One of the most popular security approaches to detect and later block malicious communication is anomaly detection. In the world of DNS based traffic, any type of unusual network activity can be seen as an anomaly. The only problem is that not all anomalies are illegitimate, and in a world where customers expect to a low false/positive rate that means… more work and laser-focused anomaly detection algorithms.

The Nominum Security Research team is exposed to an enormous volume of real-time DNS traffic, gathered from all over the globe. Receiving this large magnitude of data every second provides us a great opportunity to observe and detect many types of anomalies (…and a big enough of a training set to enhance our machine learning algorithms).

In this post we’d like to share some of the most common methods the anomalies we detect ‘translate’ into actual security enhancement tools, and present some real-life examples for attacks we thwarted.

DNS-Based DDoS Anomalies
In the incoming DNS traffic, we often observe large sudden changes. For instance, on 26th of December 2016, we witnessed a high rate queries for the following domain names:

Diagram 1 shows the distribution of queries have seen in our DNS traffic:

amplified attacks chartDiagram 1: a PRSD attack that was detected at the very early stage of forming. The volume of query counts shows the magnitude of the attack.

This is an example of an anomaly known as a DNS-based DDoS attack; these DDoS attacks usually have shown up as very large spikes.

Let us define a spike as an anomaly in a stream of incoming DNS queries. Here we define and explain three types of network attacks that can be identified and prevented by spike detection method.

The first type of spike happens when the network is observing a very large number of subdomains for a given domain name. We designed and implemented a mechanism that is sensitive to the changes in the rate of subdomains and/or a number of times a specific domain name is queried (query count). In order to calculate this rate change, we first calculate the expected values for the number of subdomains and the number of queries for each domain name. The expected value is calculated by counting the number of distinct subdomains and queries in pre-defined time intervals (for instance the last N days).

We now feed these metrics (current subdomain counts, current query counts, subdomain counts expected value, and query counts expected value, resolution status) to our scoring function. The scoring function compares the subdomain counts and their expected value in addition to query counts and their expected values. If the gap between expected values and current counts are larger than certain thresholds, our detection system takes necessary actions to protect the network traffic from a possible attack. This mechanism proved to be effective in detecting DDoS attacks at very early stage. We also, construct a very large subdomain database over time from every FQDN we see (there are hundreds of millions of parent domains in this database) and use it together with our fine-grained policy to protect the true legitimate subdomains from DDoS attack.

Amplification Attack Anomalies
At the end of September 2017, the domain name was frequently seen in a number spikes. This was a contradictory example to a PRSD attack because all of these queries are being resolved, which makes it suspicious if this name is really being under the attack. A close look, reveals that the answer size that is returned by this domain name is very large. In addition, we observe that although the spike is caused by large query counts the number of subdomain counts is relatively small. These signals provide enough evidence that we are dealing with an Amplification attack rather than a PRSD.

Therefore, the key features that help us detect the Amplification attack are: when response size is relatively large, the query types are mostly ANY or TXT and queries often come from a minuscule number of clients (single client IP). In this case, our Anomaly Detection system will classify the domain name as possible Amplification attack. We then, apply additional processing to categorize the attack as a real attack or dual-use domain name, if the name is legitimate.

The figure below shows the spikes that are crucial in detecting the amplification attacks. As you can see, this spike analysis on query types in the whole DNS traffic:

amplified attacks on query types in DNS

The query count spikes are only one side of the coin in Amplification attacks. It is important to pay attention to a secondary spike observed from a number of query type. These two signals lead to identifying Amplification attacks in the traffic. Our anomaly detection system identifies these types of attack in the mater of seconds of their occurrence and applies necessary policies to protect our customers’ networks.

Tunneling Anomalies
Another example of spike detection anomaly is the DNS Tunneling (add an external link for reference). We employ a similar process that we used for DDoS and Amplification detection but due to the difference in nature of tunneling, we apply different heuristics and features.

On March 2017, our Anomaly detection system identified the domain name as a tunnel.

Typically, tunnels encrypt different requests in the subdomain of a domain name. Therefore, our focus is on the subdomain count spike preceded by the same magnitude of spikes for query count. Our main assumption is that we are detecting newly activated tunnels. We propose that the expected values for subdomain and query count for a new tunnel should be around zero. We also take into account the facts that tunnels should always resolve and they contain some contents in the DNS answer field. Applying these heuristics can help us identify tunneling activities. In addition, we applied reverse engineering to learn the possible patterns used in the contents of the answer fields. These patterns provide strong evidence for tunneling detection. We describe the concept of new domain names in depth in the next part.

New-domains Anomalies
The great visibility to the DNS level data allows the Nominum Security Research team to observe and identify new domains that are generated in real time. Therefore, the suspicious new domains can be considered as another family of anomalies. The nature of new domain anomalies allows the attackers to generate the queries in a more distributed (subtle) manner in which the spike detection methods will not be effective. These techniques hide malicious activities under the radar of the security researchers.

Let us assume that we have all the malicious domains ever existed on the Internet. A domain is considered malicious if it is associated with at least one malicious activity in the past. With this assumption we do not need decide whether an existing domain is malicious or not. We only focus on identifying maliciousness of the newly generated domains that appear in the DNS traffic. Our definition of new domain is a domain that has not been seen prior to the start time of our detection algorithm. Therefore, we have no a prior assumption about creation date or update date of the domain name. This assumption not only helps simplify our method, but also, identifies the domains that have been registered in the past but used for malicious activity in the future.

To this end we constructed a large lookup table to keep track of all domain names have been seen in the DNS traffic. We then look at every new domain along with some additional features detect malicious activities. For example, an additional feature is when a new domain is being queried from many different ISP’s in variety of locations all over the globe. We can identify such domains as suspicious with high degree of confidence.

Our recent browser hijackers related research found a growing tendency towards using dictionary-based DGA’s (Domain Generation Algorithms) when generating the redirected domains. By using pattern detection tools, focusing on the new domain names, we are able to identify and block these dictionary-based DGA domains as soon as they surface.

In Nominum, we established an anomaly detection system in order to supervise the network traffic and detect or estimate the suspicious/malicious activities and apply corresponding policies about them. As the anomaly concept is vast and dynamic, the architecture of AD is designed in a way to be compatible for new threat detections, or getting updated for new rules. In this way, we provide the defense against a big range of attacks, from DDoS to browser hijackers.

Back to the tech blog overview