Zeek

Zeek (formerly Bro) is a free and open-source software network analysis framework. It gives insights on DNS queries, HTTP and TLS information and details on transmitted files. I find Zeek one of the best network monitoring tools available to provide detailed visibility on network traffic.

Zeek has a built-in intelligence framework. This framework allows you to add information received via MISP directly into the network visibility capabilities of Zeek. This includes

• Visits to URLs or domains that are included in MISP events;
• Network requests to IPs that exist as attributes in MISP events;
• Download of files that match with hashes described in MISP events.

Workflow

My workflow consists of these steps:

1. Export MISP information in a format readable by Zeek. The MISP server is not accessible by the host running Zeek;
2. Copy the exported information to a transfer host;
3. The host running Zeek picks up the information from this transfer host;
4. Matches are captured in a log file and this log file is pushed to a transfer host, where it is picked up by a visualisation host.

Export MISP to Zeek

The first step involves exporting the MISP information in a format that is readable by Zeek. I created a Python script that queries MISP for events changed in the last 30 days, excludes events with specific tags and stores the attributes in a correct Zeek format. See the GitHub repository cudeso/misp_to_zeek.

First create an API key. In MISP you can do this via Global Actions, My Profile and then click on Auth keys. Add a new key via Add authentication key and give it a meaningful name. Because the script does not need to manipulate data, it’s best to set it to read-only. The script runs locally so access is limited to localhost (127.0.0.1).

Next, on the MISP server clone the GitHub repository, create the keys.py file and edit this file. This keys.py file holds the API key and the URL that points to your MISP server.

git clone https://github.com/cudeso/misp_to_zeek
cd misp_to_zeek
cp keys.py.default keys.py

misp_url = "https://misp"
misp_key = "secret_api_key"
misp_verifycert = False

The Python script only requires the Python requests library. You can tune the execution of the script by changing the variable misp_config.

misp_config = { "last": "30d", "to_ids": 1 , "enforceWarninglist": 1, "tags": [ "!tlp:red" ]}

In this example

• last:30d : The information changed in the last 30 days is fetched;
• to_ids:1 : Only those attributes that have the to_ids (‘actionable’) flag set are taken into account;
• enforceWarninglist:1 : Any matches with warninglists are excluded;
• tags: “!tlp:red” : All TLP:RED information is excluded. You can add additional tags to include or exclude information.

Add a cronjob to execute the script together with the secure copy.

00 5,9,13,17  * * * cd /home/misp/misp_to_zeek/ ; python3 /home/misp/misp_to_zeek/update_intel.py ; scp -i /home/misp/misp_to_zeek/sshkey.private /home/misp/misp_to_zeek/misp-*.intel upload_user@transfer:/transfer/misp_to_zeek/

The Python script generates files that start with misp- and end with .intel. These ‘intel’
files contain the MISP attributes in a format that Zeek understands.

Transfer host

The configuration of the transfer host is straightforward. This host allows secure file transfer with public/private key access from a limited set of devices and users. The transfer host does not initiate new connections.

Install Zeek

Finally, on the host that is running Zeek I also clone the GitHub repository, but this time I do this in the location where Zeek is installed.

cd /opt/zeek/share/zeek/site/
git clone https://github.com/cudeso/misp_to_zeek
cd misp_to_zeek

The files __load__.zeek and main contain the configuration that is needed for the Zeek intelligence framework to include the MISP information. In __load__.zeek we instruct Zeek to load all settings from main, which is the list of “Intel” files.

##! Load Intel Framework
@load policy/integration/collective-intel
@load policy/frameworks/intel/seen
@load policy/frameworks/intel/do_notice
redef Intel::read_files += {
	"/opt/zeek/share/zeek/site/misp_to_zeek/misp-domain.intel",
    "/opt/zeek/share/zeek/site/misp_to_zeek/misp-hostname.intel",
    "/opt/zeek/share/zeek/site/misp_to_zeek/misp-ip-dst.intel",
    "/opt/zeek/share/zeek/site/misp_to_zeek/misp-ip-src.intel",
    "/opt/zeek/share/zeek/site/misp_to_zeek/misp-ja3-fingerprint-md5.intel",
    "/opt/zeek/share/zeek/site/misp_to_zeek/misp-md5.intel",
    "/opt/zeek/share/zeek/site/misp_to_zeek/misp-sha1.intel",
    "/opt/zeek/share/zeek/site/misp_to_zeek/misp-sha256.intel",
    "/opt/zeek/share/zeek/site/misp_to_zeek/misp-sha512.intel",
    "/opt/zeek/share/zeek/site/misp_to_zeek/misp-url.intel"
};

Then change the Zeek configuration in /opt/zeek/share/zeek/site/local.zeek to include the intelligence files and set the logging configuration to JSON format. The logs are send to an Elastic host and JSON makes parsing the logs much more easier.

# Output to JSON format
@load policy/tuning/json-logs.zeek

# Include MISP
@load misp_to_zeek

As a last step add a cron job that fetches the intel files from the transfer host and deploys the new settings (intel files) to Zeek.

/opt/zeek/bin/zeekctl deploy

Zeek logs matches to /opt/zeek/logs/current/intel.log and because the log format is set to JSON this is easy to read/parse.

{
	"ts": 1673518745.662658,
	"uid": "CZ2lksFrAiip3gF7a",
	"id.orig_h": "192.168.0.1",
	"id.orig_p": 38203,
	"id.resp_h": "192.168.0.2",
	"id.resp_p": 53,
	"seen.indicator": "__redacted__.com",
	"seen.indicator_type": "Intel::DOMAIN",
	"seen.where": "DNS::IN_REQUEST",
	"seen.node": "zeek",
	"matched": ["Intel::DOMAIN"],
	"sources": ["OSINT - IOCs (event ID 93042)"]
}

Visualisation

The visualisation of the Zeek information and alerts is done with Elastic and the Filebeat module for Zeek. Unfortunately (at least in the v7) the module does not include a visualisation for the “intel” files. I added minor visualisation extensions by filtering on the intel data set and then creating visualisations for matched intel, matched intel types, source MISP events and number of occurrences.

event.dataset :"zeek.intel"

The alerting is then done via Elastalert.

Conclusion

Zeek is a simple a great tool to monitor your network traffic. It’s complementary to Suricata, which is more aimed to signature-based detection. In this case, the monitoring host cannot run both Suricata and Zeek. Adding the “intel” detection to Zeek gives sufficient alerting if there’s activity matching a known threat event.

Note that there is also Dovehawk which allows you to send sightings back to MISP. Although MISP sightings are very useful, this approach with Dovehawk was not possible in my setup.