Using open source intelligence feeds, OSINT, with MISP

Posted on in internet, open source, osint, security
   24 Replies

MISP

I love MISP, Malware Information Sharing Platform & Threat Sharing. I did three earlier posts on how to use and setup MISP. part 1, part 2 and part 3.

Open Source Information by MISP, OSINT

One of the nice new features by MISP is including feeds from different open source intelligence feed providers.

How does it work? Basically the feeds are provided as a JSON feed, you can browse them within MISP, import them individually or subscribe to the feed to get automatic updates.

Using OSINT feeds within MISP

First I assume that you upgraded MISP to the latest version. This is straightforward by pulling the latest version from Github.

git pull

Once this is done, log in to your MISP instance and go to Sync actions, List Feeds. You will then get a list of the available open source feeds.


MISP Feeds

Then you will need to enable or edit a feed. This will bring you to a screen where you can set the default distribution level and most importantly, the default tag.

For my setup I use the TLP:White tag. Remember that these tags are defined in the taxonomies that you have defined in MISP (see Event Actions -> List Taxonomies). I use TLP:White because OSINT information is by default TLP:White. Not sure about the TLP code? Use the post How to use the traffic light protocol – TLP.

Do not forget to tick the checkbox Enabled because otherwise the feed will not be enabled.

MISP-Feed Edit

Once you have edited the feed you will return to the overview of available feeds.

You can browse the feed content. Do this for the feed that you enabled.


MISP feed browse

This will result in a an overview of the open source events from that feed. You can now select an invidividual event to be included in your MISP instance.

MISP Feed import

Basically that is all there is for including an open source feed to your MISP data.

  • Enable a feed
  • Set the distribution-level and tag it
  • Sit back and enjoy

I want to contribute open source intelligence feeds

Using the OSINT feed is great but maybe you also want to contribute your OSINT feed to the community? You can. And it’s fairly easy. For this you need to use a Python script : PyMISP.

PyMISP

PyMISP is a Python library using the MISP Rest API. In essence this means that instead of crafting MISP API requests you can use Python request to interact with MISP.

Installing PyMISP is described on the GitHub page.

git clone https://github.com/CIRCL/PyMISP.git
cd PyMISP
python setup.py install

Basically it needs an API key and a URL. This can be set in any file you desire as long as you include it in your Python script that calls PyMISP. As an example you can use this in you Python scripts

from pymisp import PyMISP
from cudeso import misp_key
from cudeso import misp_url
from cudeso import misp_verifycert

This will import the API key and url from an external file.

Using PyMISP for OSINT

Once you have setup PyMISP you can use one of the example scripts to generate the OSINT feed. In the folder PyMISP/examples/feed-generator/ there are two files that you need to edit. Note that the OSINT generator script does not use the configuration script of PyMISP described above.

First you need to edit the file settings.py. The important settings to change are 

url = '<MISP-URL>'
key = '<MISP-automation-key>'
outputdir = 'output-dir'

The MISP url (url) defines where the script can find you MISP instance. The automation key (key) is the key set by your automation user. Finally the outputdir sets where you want to output the files. Note that this directory will hold all the MISP events. You have to make sure that this directory

  • is writable by the generate script
  • is a separate directory
  • accessible by the public if you want to publicize your feed (for example in your web directory).

Once you have set these settings you can run the feed generator script by issuing 

./generate.py

This will generate all the MISP events in the output directory set by “outputdir”.

Make your feed known!

If you are happy with your feed you should announce it to CIRCL to have it included as one of the OSINT feeds in the next update of MISP. The easiest way for doing so is via the MISP GitHub repository.

Existing open source feeds

The current release of MISP contains these OSINT feeds

  • https://www.circl.lu/doc/misp/feed-osint
  • http://www.botvrij.eu/data/feed-osint

I maintain the feed for botvrij.eu. If you want to include a description of a threat info then you can send me a MISP XML file or a pointer to a public description.

24 thoughts on “Using open source intelligence feeds, OSINT, with MISP”

    • Koen on said:

      You can do that for botvrij.eu ; the different files (NIDS, hashes, etc) are made available as download. For the MISP data, these are made available as JSON files, in theory you can just fetch them and postprocess the way you want.

      Reply ↓
    • Koen on said:

      Hi Douglas,

      Is this only with the feed botvrij.eu?
      Have you checked if the workers are running in the background?

      kr,

      koen

      Reply ↓
      • Douglas Molina on said:

        workers are running. When I do a ps-aux | grep queue. I see default, prio, email, cache as started. When I do systemctl status misp-workers I see them started in the background.

        I am running MISP over RHEL and I am running in a lot of problems

        Reply ↓
  6. Douglas Molina on said:

    Also, by chance do you have any configuration guides related to installing premium feed, or atleast point me in the right direction? We want to set up a feed with one of our agency partners who happens to have a threat connect platform and one with infoblox. This will be utilizing an API to retrieve the feeds.

    Reply ↓
  7. Douglas Molina on said:

    what does it mean to enrich an event? what actually happens? I tried to do it but i get an error that it has failed. no other information on why.

    Reply ↓
    • Koen on said:

      It means adding extra information to events based on existing attributes by querying data sources external to MISP. This querying happens via Python scripts includes in MISP-Modules.
      For example if you have a domain name in your event you can resolve this to an IP address (either passive DNS or current DNS). This IP address is then the enriched information that is added to the event. With the IP address is turn you can look up to which network provider it belongs (via whois) and then add the name and network block. Additionally the added IP address can link the event to other MISP events that containing the same IP address.
      Another “enrichment” is when you add a malware sample hash to a MISP event, then querying Virustotal for the file hash and then using the results from Virustotal like domain names and URLs as extra attributes for the event.

      See also for example https://www.circl.lu/assets/files/misp-training/brussels2016/misp-modules.pdf

      There is also a “Gitter” chat channel where you can also post questions concerning MISP : https://gitter.im/MISP/MISP

      Reply ↓
  8. Gary L Torrence on said:

    Does anyone have any information on how to access a MISP instance using only token logins related to CAC card access. This creates a token used by different applications to allow access. In this case of course access to an instance of MISP bypassing the login page authentication?

    Reply ↓
  9. Douglas Molina on said:

    I need to make some changes to a 3rd party feed that are not available in the GUI. I want the feeds published by dont want email alerts being sent. Where can i find the settings which i can make changes to via the CLI

    Reply ↓
  10. Rahul Sharma on said:

    I am exploring ways to use MISP as an intelligence repository system like feeding it with several attack patterns from Mitre that are relevant to our organization and possible mitigations. It can then able to correlate it with actual incidents giving the analyst first-hand information on the attack with possible mitigations. I am approaching this by creating an event having several attack patterns of groups like APT38, APT41, Lazarus Group, etc but still not getting any correlation with any of the events. Can somebody suggest the direction?

    Reply ↓
  11. Rahul Sharma on said:

    I am exploring ways to use MISP as an intelligence repository system like feeding it with several attack patterns from Mitre that are relevant to our organization and possible mitigations. It can then able to correlate it with actual incidents giving the analyst first-hand information on the attack with possible mitigations. I am approaching this by creating an event having several attack patterns of groups like APT38, APT41, Lazarus Group, etc as its attribute but still not getting any correlation with any of the events. Can somebody suggest the direction?

    Reply ↓
  13. Raghuman Rao on said:

    Hi,
    I am trying to update the feeds using PyMISP scripts, I am unable to do so as I am getting an error message as:
    Something went wrong (405): {‘name’: ‘You do not have permission to use this functionality.’, ‘message’: ‘You do not have permission to use this functionality.’, ‘url’: ‘/feeds/index’}

    Can you please help me out to resolve this.

    Reply ↓

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.