msticpy — 0.6.0/1 Release

MSTIC
3 min readJul 2, 2020

This is mostly housekeeping release — we needed to do a bit of updating and testing for the new Azure Machine Learning nteract-based notebook environment. This is going to be the default execution environment for Azure Sentinel Notebooks. Read more about nteract and about the Azure ML Notebooks feature.

New Features

Friendly Exceptions

We’ve introduced a set of friendly exceptions for common configuration-related problems that users might encounter while using msticpy in notebooks. In most cases, the cause is lack of a config item such as an API key or setting. Hitting one of these exceptions gives you a few lines of explanatory text plus links to documentation on readthedocs to help you fix it.

User-friendly exception example

In most cases the traceback is not relevant (since this is a config, rather than code-based error) so this is not shown for these exception types (although this can be re-enabled).

Time Series Analysis

Time Series Analysis using msticpy and Bokeh

We added time series decomposition functionality (using Statsmodel STL) so that you can do time series analysis on any data (previously we relied on Kql/Azure Sentinel to do the decomposition/analysis part). This function allows you to perform the analysis and anomaly visual on any time-stamped data in a pandas DataFrame. Ashwin Patil added documentation and a sample notebook describing how to use this.

Anomalous Sequences

Karishma Dixit added some more capability and documentation for the Anomalous Sequence feature. This allows you to identify unusual sequences of actions/activities in user sessions. Karishma wrote about this in a recent blog. Feature documentation is here and a Jupyter notebook demonstrating use of anomalous sequences is also available.

Timeline results for anomalous sessions (anomalies at bottom of graph)

Documentation and Readme updates

We’ve updated the GitHub README to add missing details, correct some things and added a few more images to illustrate the functionality and make it bit more attractive.

Also:

  • Added documentation for Azure Sentinel configuration for notebooks — how to use the config.json and msticpyconfig.yaml.
  • Updated a lot of the introductory sections to readthedocs to bring up-to-date and align with README
  • Replaced some badly-formatted tables in readthedocs docs
  • Anomaly Sequence and TimeSeries module/API docs not appear in the API doc tree

Other Changes

  • We replaced the use of the ipywidgets Output widget with IPython updatable display() objects. This is primarily for nteract/AML compatability. (The PR for this #71). Most Jupyter notebook users will be familiar with the IPython display() function. What is less well-known is that each call to display creates a display object that can be referenced and updated by later code.
Execute the first cell to display some text
Executing the second cell updates the display object from the previous cell
  • Added check_version() function that will check the current version against latest on PyPI
Checking that your msticpy version is up-to-date
  • Relaxing some version requirements in setup.py/requirements.txt to match AzureNotebooks resulting in quicker install #68
  • Adding new queries for Notebooklets project #67

--

--

MSTIC

This is the account of the Microsoft Threat Intelligence Center (MSTIC).