MSTICPy 1.6 & 1.7 — The Hackathon Edition ft Cybereason driver, IntSights TI, Sentinel features and more
In recent weeks we have released MSTICPy 1.6 & 1.7 containing a range of features contributed during the MSTICPy January Hackathon. In this blog we are going to cover the features added in these releases including:
· Expanded Microsoft Sentinel features, including integration with new APIs
· Layering and subgrouping added to FoliumMap
· New VirusTotal features and Pivots
· Instance handling for Data providers, allowing for multiple versions of a data provider type to be included in config
· Features to make the Process Tree visualization schema agnostic
· Caching for OTRF datasets accessed via the Mordor driver
· A Data provider for Cybereason
· A TI provider for IntSights
· A new Sentinel Alert entity for processing Sentinel
· Some new sample notebooks
We will also have a roundup of other changes and improvements included in these releases.
You may have noticed that we did not have a v1.6.0 release and instead jumped to v1.6.1. This was due to some technical issues. This will hopefully be a one-off occurrence and in future we will try to keep to a linear release set. For this blog if we refer to the v1.6 release this means v1.6.1.
New Features
FoliumMap Clustering and Layering Support Added
The FoliumMap class now supports layering and subgrouping to make maps with a large number of items clearer and easier to read. This was contributed by @tj-senserva (from Senserva).
New Microsoft Sentinel Integrations
The Sentinel integration features of MSTICPy have been reworked to provide a new structure and to expand them to include support for more features including Watchlists and Analytics. As well as more features of existing integrations such as Incidents.
One important note is that the class has now been renamed to drop the Azure prefix and is now referenced as `MicrosoftSentinel`, however backwards compatibility has been maintained to allow for calling as `AzureSentinel` still. Several functions have been renamed to make naming more consistent, however aliases are in place to maintain backwards compatibility.
In addition, when instantiating a Sentinel object, you need to pass in the Workspace ID you wish to connect to:
MicrosoftSentinel(res_id=”subscriptions/fdee8146–8bcf-460f-86f3–3f788c285efd/resourceGroups/myRG/providers/Microsoft.OperationalInsights/workspaces/myWorkspace
This workspace is then used for all function calls under the object.
A full list of functionality with Sentinel now includes:
· Watchlists
o List watchlists in a workspace
o Create a new watchlist
o List items in a watchlist
o Add items to a watchlist
o Delete a watchlist
· Analytics
o List deployed analytic rules
o Create new analytic rule
o Delete an analytic rule
· Bookmarks
o List bookmarks
o Create a bookmark
o Delete a bookmark
· Incidents
o List incidents
o Create an incident
o Update an existing incident
· Hunting Queries
o List hunting queries
VirusTotal Features and Pivots
The VirusTotal V3 module has been expanded to add additional operations and data visualization:
· VT FileBehavior class lets you download and view file detonation data.
· VT ObjectBrowser lets you download and view basic attributes of a file object
· Pivot functions — these expose the VT3 Relationship APIs allowing you to pivot between
related IPs, URLs, Files and Domains.
Added instance handling to Dataproviders
Several providers now support instances — e.g. Splunk, Sumologic can support multiple
configuration entries in msticpyconfig.yaml with the use of an instance suffix, e.g.
Spunk-Env1, Splunk-Evn2.
This change also makes it possible to combine these instances into a single aggregate
data provider and run the same query across the combined instances. This also
works for multiple Microsoft Sentinel workspaces.
To add a connection to an existing QueryProvider instance you can use the `.add_connection()` function. If multiple connections exist for a single QueryProvider instance, then any queries executed by that provider will be run against all of the connected endpoints.
Caching for OTRF datasets accessed via the Mordor driver
The OTRF Security Datasets driver and browser now cache security data set and MITRE ATT&CK
data for quicker startup. By default, the cache files are stored in $HOME/.msticpy/mordor
or the location specified in the Dataproviders/Mordor section of the msticpyconfig.
Cybereason Dataprovider
This provider was contributed by Florian Bracq (AXA) and allows Cybereason users to run queries against their Cybereason instance. This provider follows the same model as existing MSTICPy data providers and includes a number of built in queries.
Connection details are stored in msticpyconfig.yaml in the following format:
And queries are run in the same format as other providers:
cybereason_prov.Connection.list_connections_from_process(
hostname=”hostname”,
pid=42
start=-10,
end=-2
)
IntSights TI Provider
Florian also contributed a new threat intelligence provider for the IntSights service. Again, this follows the same format and pattern as other providers already in MSTICPy
ti_lookup.lookup_ioc(ioc=”38.75.137.9", providers=[“IntSights”])
New SentinelAlert
This new entity creates an entity specific to the format provided by Microsoft Sentinel alerts. Whilst the existing Alert entity can be used it means some key elements included in Sentinel alerts are missed. This specific entity ensures these are captured in the entity for you. This entity is intended to replace the SecurityAlert entity.
Usage is the same as with other entities, it can be created from an existing entity or from an event, such as a Pandas Series.
SentinelAlert(src_event=alerts_df.iloc[0])
Process Tree visualization Schema Expansion
Nicolas Bareil (@nbareil) added support to the Process Tree feature for Sysmon ProcessCreate events. In addition, minor changes were made to the Process Tree feature to make it more data source agnostic, this has been done by removing some instances of hard coded column names and to support instances where certain columns aren’t present in the dataset.
New Sample Notebooks
In these releases we also included a couple of new sample notebooks.
The first from Liam Kirton provides an example of working with Kusto data:
The second from Ashwin Patil demonstrates how to use AWS S3 HoneyBucket logs for threat intelligence purposes:
Other improvements & changes of note
Splunk Queries Added
Joey Dreijer (@d3vzer0) contributed a range of new queries for Splunk meaning that the Splunk data provider now has built in queries for Alert and Authentication data.
Request replaced by HTTPX
Grant Versfeld (@grantversfeld) migrated MSTICPy from using Request to handle HTTP based connections, over to HTTPX. This move will mean MSTICPy can leverage the improved performance and async support offered by HTTPX
Documentation Overhaul
Our documentation has been updated and reformatted, making it easier to read and validate. One area of change is the data providers documents that now includes separate pages for each provider. In addition danlielc-evans updated the project Readme to make it more useful for first time users.
Setup reorganization
The package setup has been reorganized to use a setup.cfg and pyproject.toml file.
Blob storage connection string support
Luis Francisco Monge (@Lukky86) added the ability for users to provide a connection string when using the AzureBlobStorage feature. This provides additional flexibility to users when connecting to the Azure Blog Storage containers.
Pipeline Open to External Forks
Previously builds triggered from external forks would fail for a variety of reasons
(no access to pipeline secrets, MS-Internal-only build actions). This is now fixed so that
external contributors should expect a clean build.
You can find out more about the excellent external contributions as part of the Hackathon in this blog. You can also find out more about the changes and additions in the v1.6.1 release notes and the v1.7.0 release notes.
If you are not already, please follow MSTICPy on Twitter for all the latest updates.
We have several areas of future work under discussion, and we would love your input. Please look at our GitHub discussions and contribute your views.
