Jupyter Notebooks are remarkably versatile tools, even within Microsoft Sentinel’s data lake where current capabilities are limited. While Microsoft frequently highlights historical threat intelligence correlation and long-term threat hunting as notebook use cases notebooks unlock far more practical possibilities.
In a previous post, I explored the theoretical foundations of using notebooks for advanced data pipelines in Microsoft Sentinel. This follow-up takes a different approach: focusing on pratical solutions that solve problems I’ve encountered in production environments. Many of these scenarios were simply impossible to address using only traditional Sentinel SIEM capabilities.
The use cases covered here tackle cost optimization, complex log routing, and advanced analytics workflows. Each represents a practical pattern you can adapt to your own environment rather than theoretical architecture. (!!!)
1. Commitment Tier Optimization Through Backfilling
Microsoft Sentinel’s Commitment Tier pricing model offers big cost savings - you commit to a guaranteed daily ingestion volume, and pay that committed amount regardless of whether you actually use it. Overage charges apply only if you exceed your commitment. However, most organizations experience uneven ingestion patterns, with weekends and holidays generating significantly less traffic than normal weekdays.. This creates unused capacity within your committed allocation that you’ve already reserved but won’t utilize.
By strategically delaying lower-priority logs until low-traffic windows, you shift expensive analytics tier costs to already-paid capacity. Effectively getting storage for free.
There are two scenario in which this option can help:
- Delaying less critical logs
- Pushing some logs from Analytics, instead of keeping it in data lake.
Scenario and Solution
A client was collecting logs from their machines. But logs fomr Development environments was too expensive for them. They specifically asked whether logging can be natively deferred, but this was not possible with Sentinel SIEM. Now, they could wait until weekends when commitment tier capacity is available for ‘free’ ingestion.
A Notebook logic could be implemented on a daily or weekly (twice on weekends) basis. A schedule that runs on weekends would look like this:
- Ingest all the critical data as expexted to an analytics table (blue in the image)
- Ingest all the development data into a separate data lake table (various colors in the image)
- Weekend, end of day collect the data and push them into an analytics table (stacked colors in the image)
Delayed logging with weekend data reingestion
A potential inefficiency already appears in practice: Wednesday might have available commitment tier capacity, but logs don’t get pushed until the weekend because notebooks only check on weekends. You can accept this gap for simplicity, or create notebooks that process weekdays too - adding complexity but improving commitment tier utilization. The key takeaway is that commitment tier efficiency can be improved with the right strategy.