MSTICPy Release 1.5 – Risk IQ integrations, Entity Investigation Graph, Kusto Support, & update Microsoft Defender for Endpoint Connector

MSTIC
5 min readJan 28, 2022

--

The 1.5.0 release of MSTICPy includes several major features & updates:

· A new visualization for graphing the relationship between entities in an investigation.

· Integration with RiskIQ as a Threat Intelligence provider.

· Updated support of Microsoft Defender as a query provider.

· Preview support for querying Azure Data Explorer/Kusto data sources.

· Support for Azure Sovereign Clouds in the Microsoft Graph query provider.

· Expanded Process Tree visualization now support Microsoft Defender data.

Entity Graph Visualization

Expanding on the existing security_alert_graph feature of MSTICPy the new EntityGraph visualization provides a way to visualize the relationships between Entities in an investigation. The graph can visualize any Entity supported by MSTICPy as well as Microsoft Sentinel Alerts and Incidents and is designed to help analysts understand the relationship between the key Entities that make up an investigation.

As with other visualizations in MSTICPy the EntityGraph is built using the Bokeh plotting library.

The graph can be initially created from an Alert or Incident or created simply from an Entity. From there it can be grown and expanded with additional Entities, and additional links, when they become part of the investigation.

from msticpy.vis.entity_graph_tools import EntityGraph

graph = EntityGraph(incident)

graph.plot()

A Bokeh generated graph showing an incident and associated entity nodes.

Users can also add notes, or free text, non-Entity based nodes to the graph to track additional information relevant to the investigation.

graph.add_note(name=”This is a note”,

description=”Notes allow for free form additions to the graph”,

attached_to=”Incident: Sample Incident”)

graph.plot()

A Bokeh generated graph showing an incident with associated entity nodes, with an additional notes node.

Similarly, Entities or links can be removed based on investigation updates:

graph.remove_link(“This is a note”, “Incident: Sample Incident”)

graph.plot()

Bokeh generated graph showing an incident with associated entity nodes, along with an unconnected notes node.

As well as plotting the graph on its own we can plot a timeline of events alongside the graph so you can see the relationships between Entities as well as the temporal elements of the Entities.

Graph.plot(timeline=True)

Bokeh generated graph with an incident note and attached entity nodes. Below the graph is a timeline showing when the nodes occured.

Once a graph has been created you can also export the nodes in the graph to a Pandas DataFrame so that the details can be easily used and connected to other MSTICPy features:

Graph.to_df()

Screenshot of a Jupyter Notebook showing a Pandas Dataframe representation of the nodes from a graph.

More details about the Entity Graph and its features can be found in the MSTICPy documentation.

RiskIQ Integration

With this release we are very excited to include a fantastic contribution from Mark Kendrick which brings integrations with RiskIQ to MSTICPy in the form of the RiskIQ threat intelligence provider. This provider provides access to RiskIQ Summary Card and Reputation datasets for IPs and hostnames with the same syntax and response format used with other TI Providers.

ti_lookup = TILookup()

lookup_result = ti_lookup.lookup_ioc(observable=”137.184.153.130", providers=”RiskIQ”)

ti_lookup.result_to_df(lookup_result).T

Screenshot of a Jupyter Notebook showing the output of a RiskIQ lookup.

This feature also adds a new way to register MSTICPy Pivot Functions used by threat intelligence providers. This means that we have Pivot functions available for all RiskIQ endpoints on Host, FNS, and IP Entities.

from msticpy.datamodel.entities import *

from msticpy.datamodel.pivot import Pivot

Pivot(namespace=globals())

(

Dns.RiskIQ.resolutions(“jquery.su”)

.query(“recordtype==’A’”)

.mp_pivot.run(Dns.RiskIQ.reputation, column=’resolve’, join=’left’)

.mp_pivot.run(IpAddress.RiskIQ.resolutions, column=’resolve’, join=’left’)

)

We don’t currently have extensive documentation for these features, but these will be added shortly.

Expanded Microsoft Defender data provider support

MSTICPy already had support for Microsoft Defender for Endpoint (MDE) as a data provider, however with this release we have updated this data provider to use the Microsoft 365 Defender APIs. This means that the new “M365D” provide you can query all the Defender data sets from a single provider.

m365d_provider = QueryProvider(“M365D”)

m365d_provider.connect()

This new provider has support for all the previously included MDE queries and you can add additional queries for the other datasets provided by M365D. We don’t currently have any pre-built queries for these new data sources, but we will look to add some in a later release.

If you want to continue to only use the MDE endpoint you can still create an “MDE” data provider which will only use the MDE API endpoints.

To use the M365D provider you will need to include connection details in your msticpyconfig.yaml file under the “MicrosoftDefender” heading. As with the existing provider it is expecting a ClientId and ClientSecret.

DataProviders:

# …

MicrosoftDefender:

Args:

ClientId: 66b9818a-26cd-4584–8eb0–7f7a499242aa

ClientSecret:

KeyVault:

TenantId: 8360dd21–0294–4240–9128–89611f415c53

For more information see the Defender provider MSTICPy documentation

Preview support for Kusto

In this release we are including a preview feature to provide the community with access to the capability for testing and feedback. This feature adds a data provider for Kusto (Azure Data Explorer), allowing KQL queries to be run against Kusto clusters. The data provider is named “Kusto” and is called in the same way as other data providers. When connecting with this data provider details of the customer cluster and database are provided in a connection string:

kusto_provider = QueryProvider(“Kusto”)

kusto_provider.connect(

“azure_data-Explorer://code;cluster=’my_cluster’;database=’my_db’;alias=’my_alias’”

)

If you use this feature and have any feedback please raise them as Issues on the MSTICPy GitHub repo or send them to msticpy@microsoft.com

Azure Sovereign Clouds in the Microsoft Graph query provider

As with other Azure features in MSTICPy we have added support to the SecurityGraph data provider to allow for connections to various Azure Sovereign Clouds. It will pull the cloud that is configured in the Azure section of your MSTICPy config file.

In addition, the SecurityGraph is now supported by the MSTICPy config file meaning that it can be configured user the “MicrosoftGraph” heading:

DataProviders:

# …

MicrosoftGraph:

Args:

ClientId: 66b9818a-26cd-4584–8eb0–7f7a499242aa

ClientSecret:

KeyVault:

TenantId: 8360dd21–0294–4240–9128–89611f415c53

Expanded Process Tree visualization now support Microsoft Defender data

The existing MSTICPy process tree visualization has been expanded to provide default support for process data provided by Microsoft Defender for Endpoint (MDE). This support is based on process data from MDE being collected from the DeviceProcessEvents table in Microsoft Sentinel.

Screenshot of a Jupyter Notebook showing the output of a process tree plot. The plot is rendered in Bokeh and shows processes, and child processes in a vertical hierarchy.

As with most releases this one also includes a number of minor fixes and improvements, more details of these can be found in the release notes: https://github.com/microsoft/msticpy/releases/tag/v1.5.0

--

--

MSTIC

This is the account of the Microsoft Threat Intelligence Center (MSTIC).