Back in December of 2011 I wrote a post on the ThreatSim blog called “Fighting The Advanced Attacker: 9 Security Controls You Should Add To Your Network Right Now“. One of the controls we recommended that folks implement was to log all DNS queries and the client that requested it:
Log DNS queries and the client that requested it: It’s been said that DNS is the linchpin of the Internet. It’s arguably the most basic and under appreciated human-to-technology interface. It’s no different for malware. When you suspect that a device has been compromised on your network, it’s important to be able to see what the suspected device has been up to. The DNS logs of a compromised machine will quickly allow responders to identify other machines that may also be infected.
Telling people to log DNS requests is super easy. Logging DNS queries is pretty easy too. Understanding how to ingest, analyze, process the data is where you need to bring in some tools. We love Splunk (as do many of our customers). Splunk is the perfect tool for the job of figuring out what your clients are up to and who they are talking to. You can also set up real-time alerts if Splunk sees a DNS lookup for a known bad domain name.
Let’s implement our recommendation in real life. Here is what we did:
1. To get your Windows domain controller to log DNS lookups, follow the directions here:
http://technet.microsoft.com/en-us/library/cc759581(v=ws.10).aspx
Here is how our DNS server debugging configuration looks:
Once you see data in c:\windows\system32\dns\dns.log you know that it’s working. The output looks like this:
As you can see, it’s ugly. The requesting client is 192.168.10.18. The name requested is imap.gmail.com. Don’t worry, Splunk will fix this later.
2. A good practice when sending new data types to Splunk is to put them in their own index. This way if you screw up or are not happy with the data you can blow away just that index and not screw your primary index up. I chose win_dns as my index that will hold DNS logs.
3. Next, we need to get the Splunk universal forwarder on the domain controller so that we can forward the dns.log to the Splunk server. I followed the directions here:
Note: I had to remove the Windows “Read Only” flag on some of the Splunk .conf files to edit them.
3. Restart the forwarder on the DNS server. You may need to wait for 8k worth of DNS activity to happen before it writes to the dns.log.
4. Now query the win_dns index to see if you are getting data in dns.log. I ran the following query:
Which shows that we’re getting data:
5. (4)imap(5)gmail(3)com(0) is ugly. Let’s use some regex to parse that out. Try this search:
A bit nicer to look at. I use the “rex mode=sed” command in the query window to perfect my regex then add it to props.conf on the Splunk server. Let’s make that parsing permanent. On your Splunk server edit:
and add the following to the end of the file:
SEDCMD-win_dns = s/\(\d+\)/./g
This will ensure that data from dns.log is cleaned up as it goes into your index. It won’t be perfect but that’s fine for now.
6. Now we need to create a field extraction so that we can query against the requested DNS domain name in its own field. If you don’t know how to do this check out this page. This page will walk you through how to extract the field that you are looking for. However I found that the interface could never get the regex just right so I wrote my own. A regex to select the DNS name is:
This should select the data in the domain field. If it looks right, save the field extraction as “domain”.
Re-run a search within your win_dns index and you should now see the extracted field “domain” in the results. Note the new “domain=imap.gmail.com” field:
7. We can now query Splunk for all DNS names that given host has asked for and the results look nice. That’s very useful for forensics and troubleshooting. However we want to know if any hosts on our vast enterprise network are up to no good and talking to people on the internet that they shouldn’t. Introducing the Splunk App Getwatchlist.
The description reads:
Getwatchlist is a custom search command for Splunk which will return a CSV formatted list from a URL. This is useful for creating lookup tables and keeping them up to date from external or internal sources. These watchlists can contain virtually anything such as domain names, IP addresses, host names, email addresses, filenames, etc. and can then be used in searches against your events. These watchlists can be in any delimited format (tab, comma, space, etc.).
Install this app in Splunk and restart your server.
7. Once Getwatchlist is installed, we need to grab a list of known bad domains. Run the following query in Splunk:
This will save the malwaredomains.csv to /opt/splunk/etc/apps/search/lookups/malwaredomains.csv (Assuming you are on a Linux box)
8. We need to setup a lookup so that we can query it on an on going basis. Go into the Manager >> Lookups >> Lookup table files and ensure that malwaredomains.csv is present.
9. Go back to Lookups and then select “Lookup definitions”. Then click “New” and name it “malwaredomains”, Type: File-based, Lookup file: malwaredomains.csv. Click “Save” and you are done creating the lookup.
10. Now run the following query:
And I see that a client has been talking to a bad domain!?!?:
But not really. I requested a lookup from a few of the malware domains by running a “host catww.com 192.168.100.100″ from my Mac, 192.168.100.100 being the IP address of my domain controller. This ensured that I had the domain in my dns.log. I also watched the dns.log file to ensure that the data had been written to it, remember the dns.log doesn’t get updated right away if you are on a quiet network.
Obviously, using Getwatchlist is a signature-based approach looking for known “bad” domains. It requires that you periodically update your malwaredomains.csv file with the latest entries. Higher-end attackers may use well-known domains as their C&C (e.g. Twitter) that will not show up in this list. However getting your DNS logs into Splunk in a searchable format is a huge step in developing your organization’s ability to investigate incidents.
Here is a helpful blog entry with more info on using Splunk and Getwatchlist to watch for malware, phishing victims, etc.
Hat-tip to @automine for creating Getwatchlist!
If you are in a role where you are using Splunk to search for signs of malware in DNS logs, you may be interested in our ThreatSim service.
-Trevor
@packetwerks
Follow us: @stratumsecurity
