Microsoft has just introduced Sentinel Data Lake (SDL) in public preview, and there’s already a flurry of excitement in the cybersecurity world. Most blog posts so far focus on how to turn it on and when you might want to use it, but very few dig into how it will change your day-to-day experience - especially when it comes to how your data is organized and accessed.
Rather than providing step-by-step instructions and how-tos, this post will break down what the new data lake means at the table level: how data is structured, how the different components interact, and what you should consider if you want to enable the data lake for your existing Sentinel environment. If you’re looking for information on where to find specific data after enabling data lake and how the architecture is evolving, you’re in the right place.
The progression leading up to Sentinel Data Lake
To truly appreciate the value of Sentinel Data Lake, it’s important to understand how Sentinel table plans (tiers) have evolved - what existed before, what was already available, and what has changed with the introduction of the SDL.
Looking back, it’s clear that Microsoft had been moving toward alternative storage options for some time. Initially, there was only the single “Analytics” tier, but then the “Basic” log tier was introduced. This offered lower-cost storage and good performance, though at the expense of extra query costs and limited feature support. Despite its differences, “Basic” logs still functioned much like “Analytics” logs.
Not long after - and still a relatively new feature - Microsoft launched the “Auxiliary” tier. This log type gained more traction than “Basic” due to even lower storage costs but came with significant tradeoffs: additional limitations, reduced feature compatibility, and slower query performance. These restrictions were a telltale sign that the data was stored in a fundamentally different way - in fact, Auxiliary logs were already backed by Data Lake technology, though this wasn’t widely publicized.
So, the recent announcement isn’t just about offering another lower-cost log option; that already existed with Auxiliary logs. The true innovation with Sentinel Data Lake lies in Microsoft offering a cost-effective, easy-to-use logging option combined with direct access to the Data Lake. This empowers analysts to perform extensive data analysis, including use cases that demand large volumes of data—such as AI and machine learning workloads.
- Easy-to-use: Because you can easily switch a table from ‘Analytics’ tier to the new ‘Data Lake’ tier while you don’t have to change your logging infra at all. No connector change, no DCR change is needed.
- Direct access: Because with Data Lake you can query all your data via Jupyter notebooks hosted by MS instead of running unfriendly Search/Restore tasks.
Because the major change isn’t in how or where the data is stored, but rather in how it’s accessed and used, most organizations won’t need a complete overhaul of their logging architecture. If you’ve been using Analytics, Basic, or Auxiliary logs, you can generally keep your existing setup and simply extend it with the new capabilities Sentinel Data Lake provides.
Sentinel vs Sentinel Data Lake tiers
Enabling Data Lake introduces a new Table Management feature in Defender XDR, letting you work with the new ‘Data Lake’ tier and move tables between ‘Analytics’ and ‘Data Lake’.
Once activated, you’ll manage tables both from the Log Analytics workspace’s Table page and from Defender XDR’s Table Management. Even though a banner may state that table configuration must now be done in Defender XDR, you can still make changes in Log Analytics - and certain options remain exclusive to Log Analytics.
Let’s start by taking a look at the available table tiers:
- Basic table tier: Basic logs are not supported by Sentinel Data Lake. Though they appear in the Table Management page, they’re greyed out and not configurable. This is why ‘Basic’ tier is not listed on the SDL side of the image.
- Analytics tier: This is Sentinel’s core log tier, storing all critical security data and providing advanced analytics. It’s available and consistent in both Log Analytics and Defender XDR, with both platforms using the same terminology.
- Auxiliary / Data Lake tier: These terms refer to the same log type, despite the different names, which often causes confusion. An Auxiliary table created in Log Analytics will be listed as ‘Data Lake’ in Defender XDR. Also, changing a table from ‘Analytics’ to ‘Data Lake’ in Defender XDR will show it as ‘Auxiliary’ back in Log Analytics. Although the two may appear interchangeable, Microsoft now recommends against directly creating an Auxiliary log. Instead, you should create an Analytics table and then switch it to ‘Data Lake’ mode.
The diagram explains in which portal you can switch between the tiers.
To move a table from the Basic tier to Data Lake / Auxiliary, you’ll first need to switch it from Basic to Analytics in Log Analytics, and then from Analytics to Data Lake in Defender XDR.
Data location and mirroring
Table management is handled behind the scenes, with data storage, copying, and movement managed transparently to ensure a seamless experience. However, understanding where your data is stored remains important, as it affects default retention periods, available features, and potential limitations.
The log types have been divided into six groups to make the diagram easier to understand. Each group will be explained after the diagram.
Various logs types and their storage location
1. Supported Sentinel Analytics table
The Analytics tier is a key component of Sentinel, delivering the most value for security operations. That’s why it’s important to understand how its behavior changes once Data Lake is enabled:
- Automatic mirroring: Once Microsoft Sentinel Data Lake is enabled, all supported Analytics tables are automatically mirrored to the Data Lake tier at no additional cost, starting from the point of activation.
- No Retroactive Mirroring: Only data created after Data Lake is enabled will be copied; existing data is not mirrored retroactively.
- Default retention: By default, analytics data is mirrored to the data lake with the same retention period (no extra cost), but retention in the data lake can be extended for up to 12 years at a low cost.
- Switching tiers: You have the option to switch from Analytics to Data Lake tier only; in this case, data stops being ingested into the analytics tier, but any previously stored data in analytics remains available until it expires per retention settings.
In short, these logs are present both in Sentinel and in Sentinel Data Lake.
2. Unsupported Sentinel Analytics table
Not all Analytics tables are supported by Data Lake, which can lead to some issues:
- No Legacy Tables in Data Lake: Legacy custom tables that aren’t based on Data Collection Rules (DCR) are not supported. These tables can’t be switched to the Auxiliary/Data Lake tier and won’t be mirrored to the Data Lake.
- Table Management for Legacy Tables: While these unsupported tables can’t move to the Data Lake tier, you can still manage their retention settings in Defender XDR through the new Table Management feature.
- AzureDiagnostics Table: Surprisingly, AzureDiagnostics is not supported for the Data Lake tier. Since this table often contains large volumes of cloud-based logs, it’s important to know that Data Lake currently doesn’t support it.
These logs are only present in Sentinel.
3. Basic table
Basic tables have never gained much traction. While a few organizations tried them, the limited features often didn’t justify their -otherwise lower - cost, which may explain Microsoft’s lack of focus on this log type:
- No Data Lake Support: Basic logs aren’t mirrored to or compatible with the Data Lake tier. Like unsupported Analytics tables, they cannot use Data Lake.
- No Table Management in Defender XDR: Although Basic logs appear in Defender XDR, all configuration options are disabled. Any changes must be made in the Log Analytics workspace. [!!! image]
If you need Basic logs to work with Data Lake, switch their tier to Analytics in Log Analytics workspace first, and then you’ll be able to move them to the Data Lake tier.
The logs are only present in Sentinel
4. Sentinel Auxiliary table / Data Lake tier
Auxiliary logs are inherently part of the Data Lake. In the Defender XDR portal, Sentinel’s Auxiliary logs appear as Data Lake tier logs.
- Data Lake support: Since these logs are designed for the Data Lake, they are stored exclusively in this tier, making them a cost-effective storage option.
The logs are only present in Sentinel Data Lake.
5. Defender logs
These events are produced by the various Defender products.
- Stored in Defender: Defender generates these logs and retains them for 30 days within its own environment.
- No direct Data Lake support: These logs cannot be moved or configured as ‘Data Lake’ tier logs, so they are not stored or replicated in the Data Lake
- Table types:* Defender provides two types of log tables. The ‘XDR’ table type has a fixed 30-day retention and cannot be customized in the portal. Tables marked as ‘Sentinel’ can have their retention settings adjusted; if you select a retention period longer than 30 days, the logs are forwarded to Sentinel as Analytics logs, which increases storage costs.
- Streaming API: Adjusting retention and forwarding logs to Sentinel requires a Streaming API slot. However, there is a bug where the configuration may not correctly use this slot, which can result in a setup that becomes unchangeable.
When data is forwarded to Sentinel as ‘Analytics’ data, it is mirrored to the Data Lake, including these specific tables. So, you can send Defender data to ‘Data Lake’ tier this way. But, it is not possible to bypass the costly ‘Analytics’ tier.
[image !!!!]
Bug: Exporting to Sentinel requires a free Streaming API slot (max 5). However, when enabled via Table Managemenet, the slot isn’t properly reserved, allowing others to occupy all slots. If none are free, you can’t change the table configuration until a slot is freed.
The logs are only present in Defender; if exported, they appear in Sentinel and are mirrored to Data Lake.
6. Microsoft Asset logs
Microsoft designates certain logs as asset logs. Once Data Lake is enabled in your Azure tenant, these logs are automatically sent to the Data Lake.
The following types of data relate to your Microsoft assets:
- Microsoft Entra
- Microsoft 365
- Azure Resource Graph
The logs are only present in Sentinel Data Lake.
Data access
Now that we know where data is stored, it’s useful to understand how access changes - because it changes.
The following diagram shows where specific Sentinel data sets were accessible before and after enabling Sentinel Data Lake.
Where different Sentinel data sets can be accessed from
Key points:
- Analytics: Supported Analytics tables are now automatically mirrored to Data Lake, adding an extra access point once Sentinel Data Lake is enabled.
- Auxiliary: Previously, Auxiliary logs were accessible from the Advanced Hunting page in Defender XDR. Enabling Sentinel Data Lake removes this access, but makes the data available in Data Lake queries.
As per Microsoft’s documentation:
- Basic: Basic logs are not supposed to appear on the Advanced Hunting page, yet they are still there and being populated with fresh data.
- Auxiliary: Microsoft states that Auxiliary logs shouldn’t be available in Sentinel via the Azure Portal after enabling Azure Data Lake. However, in my experience, these logs have continued to appear and receive new entries. The diagram reflects this observation.
Since this differs from Microsoft’s official guidance, it’s uncertain whether this behavior will persist in the future.
To the end, I add an image that shows the same info but including Defender data and Microsoft Asset data as well to have the full picture. And also a similar one on table format.
Where different data sets can be accessed from
Where different data sets can be accessed from in table format
In the last image, green indicates data is queryable, red means it isn’t accessible, and yellow for Defender shows that enabling a longer retention and forwarding data to Sentinel makes it available in both Sentinel and SDL.
Summary
Here are key takeaways with a focus on recent changes and limitations:
- Legacy and non-DCR tables are not compatible with Data Lake. Recommend checking the full list of supported tables.
- Basic logs are not supported by Data Lake or by the new Table Management features.
- Once Sentinel Data Lake is enabled, Auxiliary data is no longer queryable from the Advanced Hunting page.
- Microsoft specifies that ‘Basic’ logs should no longer appear in the Advanced Hunting experience and ‘Auxiliary’ logs should vanish from Azure Portal Sentinel - but as of now, both are still accessible in these locations.
Because this feature is still in preview, further changes and improvements are likely. This information is current as of July 2025.
If you need assistance understanding the solution, assessing its impact, or configuring it for your environment, consider reaching out to BlueVoyant for support.