MISP web scraper

I published an article on the MISP project website on the MISP web scraper.

There are a lot of websites that regularly publish reports on new threats, campaigns or actors with useful indicators, references and context information. Unfortunately only a few publish information in an easily accessible and structured format, such as a MISP-feed. As a result, we often find ourself manually scraping these sites, and then copy-pasting this information in new MISP events. These tedious tasks are time-consuming and certainly not the most interesting aspect of CTI-work.

MISP-scraper is a Python script that

• Parses RSS feeds;
• Extracts the URLs from these feeds;
• Creates a MISP event for each URL. If the combination “event-URL” already exists then the event creation is skipped;
• Adds a MISP report (with the content of the URL) to the MISP event;
• And then uses the report feature to extract indicators and context from the web page;
• It is also possible to manually add URLs and outdated events are automatically deleted.

You can find MISP scraper on Github via https://github.com/cudeso/misp-scraper.