MSTICPy 1.6 & 1.7 — The Hackathon Edition ft Cybereason driver, IntSights TI, Sentinel features and more

MSTIC
7 min readMar 4, 2022

In recent weeks we have released MSTICPy 1.6 & 1.7 containing a range of features contributed during the MSTICPy January Hackathon. In this blog we are going to cover the features added in these releases including:

· Expanded Microsoft Sentinel features, including integration with new APIs

· Layering and subgrouping added to FoliumMap

· New VirusTotal features and Pivots

· Instance handling for Data providers, allowing for multiple versions of a data provider type to be included in config

· Features to make the Process Tree visualization schema agnostic

· Caching for OTRF datasets accessed via the Mordor driver

· A Data provider for Cybereason

· A TI provider for IntSights

· A new Sentinel Alert entity for processing Sentinel

· Some new sample notebooks

We will also have a roundup of other changes and improvements included in these releases.

You may have noticed that we did not have a v1.6.0 release and instead jumped to v1.6.1. This was due to some technical issues. This will hopefully be a one-off occurrence and in future we will try to keep to a linear release set. For this blog if we refer to the v1.6 release this means v1.6.1.

New Features

FoliumMap Clustering and Layering Support Added

The FoliumMap class now supports layering and subgrouping to make maps with a large number of items clearer and easier to read. This was contributed by @tj-senserva (from Senserva).

New Microsoft Sentinel Integrations

The Sentinel integration features of MSTICPy have been reworked to provide a new structure and to expand them to include support for more features including Watchlists and Analytics. As well as more features of existing integrations such as Incidents.

One important note is that the class has now been renamed to drop the Azure prefix and is now referenced as `MicrosoftSentinel`, however backwards compatibility has been maintained to allow for calling as `AzureSentinel` still. Several functions have been renamed to make naming more consistent, however aliases are in place to maintain backwards compatibility.

In addition, when instantiating a Sentinel object, you need to pass in the Workspace ID you wish to connect to:

MicrosoftSentinel(res_id=”subscriptions/fdee8146–8bcf-460f-86f3–3f788c285efd/resourceGroups/myRG/providers/Microsoft.OperationalInsights/workspaces/myWorkspace

This workspace is then used for all function calls under the object.

A full list of functionality with Sentinel now includes:

· Watchlists

o List watchlists in a workspace

o Create a new watchlist

o List items in a watchlist

o Add items to a watchlist

o Delete a watchlist

· Analytics

o List deployed analytic rules

o Create new analytic rule

o Delete an analytic rule

· Bookmarks

o List bookmarks

o Create a bookmark

o Delete a bookmark

· Incidents

o List incidents

o Create an incident

o Update an existing incident

· Hunting Queries

o List hunting queries

VirusTotal Features and Pivots

The VirusTotal V3 module has been expanded to add additional operations and data visualization:

· VT FileBehavior class lets you download and view file detonation data.

· VT ObjectBrowser lets you download and view basic attributes of a file object

· Pivot functions — these expose the VT3 Relationship APIs allowing you to pivot between
related IPs, URLs, Files and Domains.

VT File Behavior feature screenshots

Added instance handling to Dataproviders

Several providers now support instances — e.g. Splunk, Sumologic can support multiple
configuration entries in msticpyconfig.yaml with the use of an instance suffix, e.g.
Spunk-Env1, Splunk-Evn2.

Screenshot of the config file showing multiple Splunk configurations

This change also makes it possible to combine these instances into a single aggregate
data provider and run the same query across the combined instances. This also
works for multiple Microsoft Sentinel workspaces.

To add a connection to an existing QueryProvider instance you can use the `.add_connection()` function. If multiple connections exist for a single QueryProvider instance, then any queries executed by that provider will be run against all of the connected endpoints.

Screenshot of a query being run with a provider that has multiple connections configured

Caching for OTRF datasets accessed via the Mordor driver

The OTRF Security Datasets driver and browser now cache security data set and MITRE ATT&CK
data for quicker startup. By default, the cache files are stored in $HOME/.msticpy/mordor
or the location specified in the Dataproviders/Mordor section of the msticpyconfig.

Screenshot showing caching of Mordor data

Cybereason Dataprovider

This provider was contributed by Florian Bracq (AXA) and allows Cybereason users to run queries against their Cybereason instance. This provider follows the same model as existing MSTICPy data providers and includes a number of built in queries.

Connection details are stored in msticpyconfig.yaml in the following format:

Screenshot of Cybereason config block from msticpyconfig.yaml

And queries are run in the same format as other providers:

cybereason_prov.Connection.list_connections_from_process(
hostname=”hostname”,
pid=42
start=-10,
end=-2
)

IntSights TI Provider

Florian also contributed a new threat intelligence provider for the IntSights service. Again, this follows the same format and pattern as other providers already in MSTICPy

ti_lookup.lookup_ioc(ioc=”38.75.137.9", providers=[“IntSights”])

New SentinelAlert

This new entity creates an entity specific to the format provided by Microsoft Sentinel alerts. Whilst the existing Alert entity can be used it means some key elements included in Sentinel alerts are missed. This specific entity ensures these are captured in the entity for you. This entity is intended to replace the SecurityAlert entity.

Usage is the same as with other entities, it can be created from an existing entity or from an event, such as a Pandas Series.

SentinelAlert(src_event=alerts_df.iloc[0])

Screenshot showing creation of a SentinelAlert entityfrom a Pandas Series

Process Tree visualization Schema Expansion

Nicolas Bareil (@nbareil) added support to the Process Tree feature for Sysmon ProcessCreate events. In addition, minor changes were made to the Process Tree feature to make it more data source agnostic, this has been done by removing some instances of hard coded column names and to support instances where certain columns aren’t present in the dataset.

Screenshot of a Process Tree visualization

New Sample Notebooks

In these releases we also included a couple of new sample notebooks.

The first from Liam Kirton provides an example of working with Kusto data:

Screenshot of the Kusto notebook

The second from Ashwin Patil demonstrates how to use AWS S3 HoneyBucket logs for threat intelligence purposes:

Screenshot of the Honey Bucket notebook

Other improvements & changes of note

Splunk Queries Added

Joey Dreijer (@d3vzer0) contributed a range of new queries for Splunk meaning that the Splunk data provider now has built in queries for Alert and Authentication data.

Request replaced by HTTPX

Grant Versfeld (@grantversfeld) migrated MSTICPy from using Request to handle HTTP based connections, over to HTTPX. This move will mean MSTICPy can leverage the improved performance and async support offered by HTTPX

Documentation Overhaul

Our documentation has been updated and reformatted, making it easier to read and validate. One area of change is the data providers documents that now includes separate pages for each provider. In addition danlielc-evans updated the project Readme to make it more useful for first time users.

Setup reorganization

The package setup has been reorganized to use a setup.cfg and pyproject.toml file.

Blob storage connection string support

Luis Francisco Monge (@Lukky86) added the ability for users to provide a connection string when using the AzureBlobStorage feature. This provides additional flexibility to users when connecting to the Azure Blog Storage containers.

Pipeline Open to External Forks

Previously builds triggered from external forks would fail for a variety of reasons
(no access to pipeline secrets, MS-Internal-only build actions). This is now fixed so that
external contributors should expect a clean build.

You can find out more about the excellent external contributions as part of the Hackathon in this blog. You can also find out more about the changes and additions in the v1.6.1 release notes and the v1.7.0 release notes.

If you are not already, please follow MSTICPy on Twitter for all the latest updates.

We have several areas of future work under discussion, and we would love your input. Please look at our GitHub discussions and contribute your views.

--

--

MSTIC

This is the account of the Microsoft Threat Intelligence Center (MSTIC).