The best Commitment Tier for you

A SIEM is the foundation of a modern, well-working SOC. This also means a significant part of the SOC budget can be the cost of the SIEM. Azure Sentinel offers you various payment options based on your usage. Choosing the proper one can make a big difference and can save you a lot of money compared to the default setting. Even though choosing the best one is not too difficult, a lot of companies tend to pick a suboptimal one and then stick with it for a long time. I am here to help you opt for the most beneficial choice and tell you when to change it.

Azure gives you various discounts if you commit yourself to a given amount of ingested logs per day. This means you tell Microsoft that you are definitely going to use at least (for example) 100 GB of data per day and you are ready to pay for this amount regardless of the actual ingested data. If you push less data you still have to pay this 100GB per day. If you push more data than what you have committed to, then you have to pay for the over-usage. However, both the committed amount and the over-usage is cheaper than it would normally be. For your commitment, Microsoft gives you a discount. This is the reason why it is worth considering committing to a specific amount of ingested data.

Commitment Discount

In Azure, if you push some logs into it, you have to pay for the Log Analytics Workspace (LAW) usage and also for the Sentinel usage. There are Commitment Tiers and discounts for both solutions. The discount you get for the same Tier is different for LAW and Sentinel. Here are the commitment tiers and discounts Microsoft offers for Sentinel and LAW.

Even though the prices vary from location to location the discount percentages are consistent according to the pricing portal.

Please be aware, even though the discount percentages Microsoft shows for LAW on the portal are the same in case of every location, it seems like these values are not precise ones. Even though the shown numbers are the same, the actual discount varies from location to location, and sometimes the difference is significant. See the actual discounts at the end of this blog post. Thus, I let you define the actual price in my queries to provide the most precise numbers.

Once you ingest more logs, it is reasonable to commit to a higher tier and therefore get bigger and bigger discounts.

Also, important to note that you can pick different Commitment Tiers for your LAW and your Sentinel, they don’t have to be the same. Once you change it you can’t modify it again for a month. After that, you can adjust it at any time. This means even though the cost is calculated per day, and you commit to a given amount of data per day you won’t be able to change this every day. So, even though your weekends are possibly calmer and there are a lower number of events those days, you still have to pay for the committed amount.

Establishing baseline

We have different prices based on location. So, the same amount of ingested data is cheaper at one location and more expensive in another one. Also, Sentinel and LAW have different prices and we have differences due to the discounts provided by the commitment tiers. My queries were based on the discount percentages originally. But, as I stated before, these percentages are not correct, so I am giving you the opportunity to define the actual prices in my code for your location.

The discount percentages are the same in every location

For Sentinel the offered percentages are 0%, 50%, 55%, 57%, 58%, 60%, 61%, 63% and 65% respectively for Pay-As-You-Go, Commitment Tier 100, 200, 300, 400, 500, 1000, 2000, 5000.

For LAW the offered percentages are 0%, 15%, 20%, 22%, 23%, 25%, 26%, 28%, 30% for Pay-As-You-Go, Tier 100, 200, 300, 400, 500, 1000, 2000, 5000 respectively.

These percentages come off the Pay-As-You-Go price.

Unfortunately, the percentages you can see on the LAW pricing and Sentinel pricing page are not always correct. For example, the PAYG per GB price for LAW in Switzerland West is stated to be 4.664 USD/GB, and with a 15% discount the Commitment Tier 100 has a price of 2.53 USD/GB. But this is not 15%, it is closer to a 45% discount. This means, sometimes you should switch to a Commitment Tier from Pay-As-You-Go earlier or later than the (incorrect) percentage indicates.

Here is an example, in which the portal shows the same value as the 15% of two different costs:

Sentinel vs LAW cost

The ingested data per Gigabyte on the Pay-As-You-Go price at a given location is 1.15x more expensive for LAW than for Sentinel.

For example, East US location:

• PAYG LAW price in EUR: 1.94 EUR/GB
• PAYG Sentinel price in EUR: 1.69 EUR/ GB
• 1.69 x 1.15 = 1.9435. We have to round the numbers a little bit to get 1.94.

Another random example for UK South:

• LAW: 2.429 EUR/GB
• Sentinel: 2.11 EUR/GB
• 2.11 x 1.15 = 2.4265, which is close to 2.429

Here and there this 1.15x difference is not true. If you use any of those locations, you are advised to set the prices in the queries. For example, in EAST US 2 this value seems to be closer to 1.4x. I got these values from the pricing portal of Azure. Because of the inconsistency, I have not used this fact in my final queries, but the default prices - that can be modified by you - for the second query are still defined based on this 1.15x difference.

Price of the Commitment Tier

With Pay-As-You-Go you don’t have to pay any fixed upfront money. But when you choose a Commitment Tier you have to pay a fixed price. This cost is there even if you don’t ingest any data into LAW/Sentinel.

The price of a Commitment Tier is not completely random. You can calculate it by multiplying the gigabytes you are committed to with the effective price of the given tier. So practically, you pay the same price for the reserved gigabytes as you would pay for the pushed data to Sentinel/LAW above the reserved amount. Microsoft uses some rounding here, so this is just an approximation.

Like in Central US the Commitment Tier 200GB for LAW has an effective Per GB Price of 1.74 EUR. The price of the Commitment Tier itself is 347.58 per day that contains 200GB.

So, 200 GB x 1.74 EUR/GB = 348. The value we get with calculation is slightly higher than what MS defined. (347.45 ~= 348) This way, we can easily calculate the price of our usage with the following formula:

max(included_gb, used_gb) x effective_price 

When to switch tiers

The reason it is important to clarify is that this cost can be a big part of your overall budget and I have seen a lot of bad examples already. Explaining one of them at the end of this section.

Default percentages and values are provided in the queries, but at some locations, these values are not precise enough. To tell you at what GB you should switch to the next tier we need correct costs, so the ratios must be correct. If you use a location where the percentages seem to be not the same as the ones MS provided (0%,15%,20%,22%,23%,25%,26%,28%,30% - and this is frequently not the case) then define the costs in the effective_law_pricing and effective_sentinel_pricing variables.

Here is the code one can use to visualize the prices of the various Commitment Tiers with different amounts of ingested data (gitlab link):

let effective_law_pricing = dynamic([1.0,0.85,0.8,0.78,0.77,0.75,0.74,0.72,0.7]);  
// [payg,tier100,tier200,tier300,tier400,tier500,tier1000,tier2000,tier5000]  
let effective_sentinel_pricing = dynamic([1.0,0.5,0.45,0.43,0.42,0.40,0.39,0.37,0.35]);  
let start_gb = 0;   
let end_gb = 500;   
let step_size = (end_gb-start_gb)/50; //50 steps will be shown the graph, for more precise calculation set this value higher    
let sentinel_prices = range x from start_gb to end_gb step step_size   
| extend payg_sentinel_price = max_of(x,0.0) * effective_sentinel_pricing[0] 
| extend 100_sentinel_price = max_of(x,100.0) * effective_sentinel_pricing[1]  
| extend 200_sentinel_price = max_of(x,200.0) * effective_sentinel_pricing[2] 
| extend 300_sentinel_price = max_of(x,300.0) * effective_sentinel_pricing[3] 
| extend 400_sentinel_price = max_of(x,400.0) * effective_sentinel_pricing[4]  
| extend 500_sentinel_price = max_of(x,500.0) * effective_sentinel_pricing[5]  
| extend 1000_sentinel_price = max_of(x,1000.0) * effective_sentinel_pricing[6]   
| extend 2000_sentinel_price = max_of(x,2000.0) * effective_sentinel_pricing[7]    
| extend 5000_sentinel_price = max_of(x,5000.0) * effective_sentinel_pricing[8] ;   
let law_prices = range x from start_gb to end_gb step step_size   
| extend payg_sentinel_price = max_of(x,0.0) * effective_law_pricing[0]  
| extend 100_sentinel_price = max_of(x,100.0) * effective_law_pricing[1]  
| extend 200_sentinel_price = max_of(x,200.0) * effective_law_pricing[2]    
| extend 300_sentinel_price = max_of(x,300.0) * effective_law_pricing[3]   
| extend 400_sentinel_price = max_of(x,400.0) * effective_law_pricing[4]   
| extend 500_sentinel_price = max_of(x,500.0) * effective_law_pricing[5]   
| extend 1000_sentinel_price = max_of(x,1000.0) * effective_law_pricing[6]   
| extend 2000_sentinel_price = max_of(x,2000.0) * effective_law_pricing[7]    
| extend 5000_sentinel_price = max_of(x,5000.0) * effective_law_pricing[8] ;    
//sentinel_prices   
law_prices 

The code has two functions defined in it. Comment out the one you don’t want to use. Logically, sentinel_prices shows you the output based on Sentinel information, while the law_prices shows the information related to Log Analytics Workspace.

Part of the code you can modify:

1. effective_law_pricing and effective_sentinel_pricing: The percentages provided by MS are often not correct. Thus, it is the best if you define the prices of your location in these variables. The first element in the array is the price of the PAYG, the second element is the Commitment Tier 100 per GB price, the third is the Tier 200 per GB price, and so on.
2. start_gb: This will be the first entry on your graph. Because the smallest tier (0GB with Pay-As-You-Go) and the highest tier (5000 GB), has a huge difference, and the Commitment Tiers at the lower level are more frequently defined, so the graph can be hard to read. Thus, I recommend defining a smaller window. If you are interested when it is worth switching to Commitment Tier 200 from Tier 100, I recommend you using start_gb =0, end_gb=500.
3. step_size: Calculated automatically. This way, we will always have 50 steps between start and end. A higher number will show a more precise diagram, but it will be heavier and slower. Defining a smaller step will result in a more precise calculation.

The image (and calculation) shows at how many GB of ingested data it is worth switching to a higher tier. X marks the ingested data in GB, and in this case the threshold is 50 GB:

Visualization is not done automatically; you can configure on the Sentinel GUI what you actually want to see on the diagram. Click on the “Chart” button, pick the “Line” graph option and choose the dataset you want to see on the graph. In my case, I picked PAYG, Tier100, 200, 300:

One can even see on the image below that PAYG and Commitment Tier 100 has the same price at 50 ingested GB. After this going with Tier 100 would be the cheaper option. (This was a Sentinel calculation in East US).

Define the prices in the provided query and check when you should switch from one tier to another. The graph is going to tell you when you reached a point at which switching to another tier is beneficial for your.

The given GBs are approximate numbers, but this is not an problem here. Just because you reached any of these points you do not have to switch immediately. After changing tiers, you will not be able to modify the setting for 30 days. It is only worth upgrading if you know you will push this higher amount of data consistently. For example, because you have reached the point and you know your company is growing further. So, the decision should not solely be based on these thresholds, it should also depend on your plans and future.

I worked for a company that decided to create a Playbook that sends them an e-mail when they reached a specific threshold to tell them they should switch from PAYG to Commitment Tier 100. They did not do any calculations though, so the Playbook was created to send an e-mail when the daily ingested data – in 3 consecutive days – reach 100 GB. Their location was East US, which means even at 90 GB/day Commitment Tier 200 is better than PAYG. In this case, for Sentinel, they should have considered this switch (from PAYG to Tier 100) already at 50 GB. Obviously, we quickly modified the Playbook to alert them on time.

Why using Average is not a funky idea

In the previous section, I showed you how to define the numbers at which you should consider switching to a higher Commitment Tier. But that calculation only tells you whether you would have been in a better situation with a higher Commitment Tier for that day. So, when to change Tiers exactly?

There are a lot of code snippets on the internet showing you how to calculate your prices over a month. And these codes use the average function to give you the numbers (or just adds the usage together without calculating per day costs). And this is not always incorrect. For example, if you calculate your cost while you are using Pay-As-You-Go then you can freely use average for calculation. It can be also correct –but not always- if you use a Commitment Tier and your daily ingestion doesn’t vary too much, and the actual daily ingestion does not deviate much from the average value. But in other situations, it is often not the way to go.

Average in price calculation other than PAYG

Consider this example; a Sentinel calculation, so Tier100 has a 50% discount.

The price of Tier 100 was calculated with this code: =MAX(50;ingested_gb/2). So, one has to pay at least 50 money (currency does not matter), because that is the price of the commitment tier. And if there is over-usage, then the cost is going to be the used gigabytes with 50% discount. Again, the price of the gigabyte is 1. This also means that below 100GB/day you don’t have to pay for over-usage. As one can see, if we use the average value then we will be under 100 GB every day while using the real ingested data every day will cause some cost increase due to over-usage (over 100GB).

This won’t affect the Pay-As-You-Go-Price. But in case of Tier100 using the average is a mistake. Using the average daily ingestion will result in 7x50 = 350 money as cost, while using the real ingestion will result in 4x50 + 3x74.5 = 423.5 money. And this is a good 20% difference.

Average in case of Commitment Tier picking

This example shows the ‘Average’ issue between Commitment Tier 100 (50% discount) and Commitment Tier 200 (55% discount).

One can see that using the real ingested GBs and proper values favors Tier 100. While using the Average daily ingestion favors Tier 200. In a situation like this choosing Tier 100 would be the better choice.

Also, in our previous section, I showed when it is worth picking another tier based on the ingested GB. Here, the average daily ingestion is 371 GB. This is really high, which would indicate that the higher Tier is better. But this is not the case. The huge deviation on Monday from the daily average makes it untrue. The average is high because of that one day, but a higher tier won’t be used that much on other days causing it to be wasted.

So, using average this way can cause us to miscalculate our costs and misidentify a threshold. Still, if the variation and deviation are small, we can still use it for approximation or simple alerting. Also, the average calculation can be still useful in case of Pay-As-You-Go pricing tier.

Query for proper Commitment Tier selection

Because using ‘average’ was not satisfactory every time, especially in case of higher tiers, I decided to create a quick query myself. Due to the rounding in the costs, the output price can be different than the real one. However, the code is not for price calculation but for helping to pick the proper Commitment Tier. But, in case of higher Commitment Tiers this is still a better way to calculate your prices than using the average function.

The KQL query calculates your daily ingestion and its price including discount if there is any. This way you can define your approximate daily ingestion cost. It also tells you which option would have been the cheapest for you based on your historical data.

Query is available on gitlab (on gitlab you can find the most up-2-date version, but this post won’t be updated):

let starttime = 20d;
let endtime = 1d;
let SentinelPrice = dynamic([1.0,0.5,0.45,0.43,0.42,0.40,0.39,0.37,0.35]);
// [payg,tier100,tier200,tier300,tier400,tier500,tier1000,tier2000,tier5000]
//SentinelPrice[0] = PAYG price
// SentinelPrice[1] = Commitment Tier 100 price
let LAWPrice = dynamic([1.15,0.98,0.92,0.9,0.88,0.865,0.85,0.83,0.805]); // the cost difference between Sentinel per GB and LAW per GB is approximately x1.15 on most location, but not everywhere
let percentageIncrease = 0; //expected data ingestion increase
let MBPerDay = Usage
| where _TimeReceived > startofday(ago(starttime)) and _TimeReceived < startofday(ago(endtime))
| where IsBillable == true
| summarize MB_per_day = (sum(Quantity) * ((100.0 + percentageIncrease) / 100.0)) * 1.0 by bin(_TimeReceived,1d);
let SentinelPricing = MBPerDay
| extend PayAsYouGo = max_of(MB_per_day/1024.0, 0.0) * SentinelPrice[0]
| extend Commitment100 = max_of(MB_per_day/1024.0, 100.0) * SentinelPrice[1]
| extend Commitment200 = max_of(MB_per_day/1024.0, 200.0) * SentinelPrice[2]
| extend Commitment300 = max_of(MB_per_day/1024.0, 300.0) * SentinelPrice[3]
| extend Commitment400 = max_of(MB_per_day/1024.0, 400.0) * SentinelPrice[4]
| extend Commitment500 = max_of(MB_per_day/1024.0, 500.0) * SentinelPrice[5]
| extend Commitment1000 = max_of(MB_per_day/1024.0, 1000.0) * SentinelPrice[6]
| extend Commitment2000 = max_of(MB_per_day/1024.0, 2000.0) * SentinelPrice[7]
| extend Commitment5000 = max_of(MB_per_day/1024.0, 5000.0) * SentinelPrice[8];
let LAWPricing = MBPerDay
| extend PayAsYouGo = max_of(MB_per_day/1024.0, 0.0) * LAWPrice[0]
| extend Commitment100 = max_of(MB_per_day/1024.0, 100.0) * LAWPrice[1]
| extend Commitment200 = max_of(MB_per_day/1024.0, 200.0) * LAWPrice[2]
| extend Commitment300 = max_of(MB_per_day/1024.0, 300.0) * LAWPrice[3]
| extend Commitment400 = max_of(MB_per_day/1024.0, 400.0) * LAWPrice[4]
| extend Commitment500 = max_of(MB_per_day/1024.0, 500.0) * LAWPrice[5]
| extend Commitment1000 = max_of(MB_per_day/1024.0, 1000.0) * LAWPrice[6]
| extend Commitment2000 = max_of(MB_per_day/1024.0, 2000.0) * LAWPrice[7]
| extend Commitment5000 = max_of(MB_per_day/1024.0, 5000.0) * LAWPrice[8];
let pricing_per_service =SentinelPricing 
| union withsource=SourceTable LAWPricing
| extend SourceTable = iif(SourceTable == "union_arg0", "SentinelPrice", "LAWPrice")
| summarize cost_PayAsYouGo = sum(PayAsYouGo), cost_Commitment100 = sum(Commitment100), cost_Commitment200 = sum(Commitment200), cost_Commitment300 = sum(Commitment300), cost_Commitment400 = sum(Commitment400), cost_Commitment500 = sum(Commitment500), cost_Commitment1000 = sum(Commitment1000), cost_Commitment2000 = sum(Commitment2000), cost_Commitment5000 = sum(Commitment5000) by SourceTable
| extend cost_lowest = min_of(cost_PayAsYouGo, cost_Commitment100, cost_Commitment200, cost_Commitment300, cost_Commitment400,cost_Commitment500,cost_Commitment1000,cost_Commitment2000,cost_Commitment5000);
let pricing_per_day =SentinelPricing
| union withsource=SourceTable LAWPricing
| extend SourceTable = iif(SourceTable == "union_arg0", "SentinelPrice", "LAWPrice")
| summarize cost_PayAsYouGo = sum(PayAsYouGo), cost_Commitment100 = sum(Commitment100), cost_Commitment200 = sum(Commitment200), cost_Commitment300 = sum(Commitment300), cost_Commitment400 = sum(Commitment400), cost_Commitment500 = sum(Commitment500), cost_Commitment1000 = sum(Commitment1000), cost_Commitment2000 = sum(Commitment2000), cost_Commitment5000 = sum(Commitment5000) by _TimeReceived
| extend cost_lowest = min_of(cost_PayAsYouGo, cost_Commitment100, cost_Commitment200, cost_Commitment300, cost_Commitment400,cost_Commitment500,cost_Commitment1000,cost_Commitment2000,cost_Commitment5000);
let pricing =SentinelPricing
| union withsource=SourceTable LAWPricing
| extend SourceTable = iif(SourceTable == "union_arg0", "SentinelPrice", "LAWPrice")
| summarize cost_PayAsYouGo = sum(PayAsYouGo), cost_Commitment100 = sum(Commitment100), cost_Commitment200 = sum(Commitment200), cost_Commitment300 = sum(Commitment300), cost_Commitment400 = sum(Commitment400), cost_Commitment500 = sum(Commitment500), cost_Commitment1000 = sum(Commitment1000), cost_Commitment2000 = sum(Commitment2000), cost_Commitment5000 = sum(Commitment5000) by _TimeReceived, SourceTable
| extend cost_lowest = min_of(cost_PayAsYouGo, cost_Commitment100, cost_Commitment200, cost_Commitment300, cost_Commitment400,cost_Commitment500,cost_Commitment1000,cost_Commitment2000,cost_Commitment5000);
pricing_per_service //showing the optimal LAW and Sentinel costs
//pricing // optimal LAW and Sentinel pricing for each day
//pricing_per_day // optimal overall pricing per day (because you can choose different Commitment Tiers for LAW and Sentinel this function is not needed most of the time
//LAWPricing // only the LAW pricing for PAYG and Commitment tiers, it does not show the best one
//SentinelPricing // only the Sentinel pricing for PAYG and Commitment tiers, it does not show the best one
//MBPerDay // the amount of ingested non-free data per day

 

For a calculation like this, using the _TimeReceived (I am using this one) or the ingestion_time() value seems like a better fit than the TimeGenerated field.

The query can calculate multiple things. It is up to you which one you choose. To use another function just comment out the one you use right now and remove the comment mark “//” from before the function you want to use:

• pricing_per_service: shows the optimal Sentinel and LAW costs separately
• pricing: optimal LAW and Sentinel pricing separately per day
• pricing_per_day: optimal overall pricing per day – it adds the LAW and Sentinel prices together (because you can choose different Commitment Tiers for LAW and Sentinel this function is not needed most of the time, can be used for cost calculation purposes)
• SentinelPrice: Calculates the LAW pricing without choosing the best one. This function is there to be used in other queries.
• SentinelPricing: Calculates the Sentinel pricing without choosing the best one. This function is there to be used in other queries.
• MBPerDay: The amount of ingested non-free data per day

The “pricing_per_service” is the default function and this one was created to help you identify the best Commitment Tier. After the KQL query has run it shows the best price you can get based on your historical data. After that, you have to choose which Commitment Tier that price relates to (for now, in the future the code will tell you).

Example:

One can see on this picture above, that the optimal LAWPrice is related to Commitment Tier 200 while the optimal Sentinel price is the cost of the Commitment Tier 300.

Configure the following values before you start to use the script:

• starttime: Defines the start date of the query. Cost and best tier calculation will start from the start of the day.
• endtime: Defines the enddate of the query. Cost and best tier calculation will end at the start of the day.
• SentinelPrice: Contains the price of the various tiers for Sentinel in a list. Default values are defined based on the MS-provided percentages, but for proper calculation use the cost of your chosen location.
• LAWPrice: Contains the price of the various tiers for LAW in a list. Default values are defined based on the MS-provided percentages, but for proper calculation use the cost of your chosen location.
• PercentageIncrease: The calculator was created to tell you what would have been the best Tier for you based on your historical data. If you want to prepare for the future and you expect an increase in data ingestion you can use this constant. Example: Putting ‘10’ into this field will increase the daily ingestion rate by 10% in your calculation for each day.

Recommendations:

1. Whenever possible, define the exact prices of the Tiers for your location.
2. The query does not take any other discounts into account. Like the default free GBs in Log Analytics Workspace or the free GBs that come from Defender if you onboard specific devices.
3. MS offers its own KQL query for this calculation. It includes more discounts but also has some limitations compared to mine. Feel free to use whichever you choose.

Magic

As I expressed multiple times in this blog post, the actual discount and the number provided by MS are not the same. This means here and there you can get even bigger discounts than what is shown on the table. These are calculated based on the USD prices, there can be small differences in other currencies due to rounding.

Real discounts for Tier100 over Pay-As-You-Go Prices:

Please be aware, higher percentages do not mean the given locations are cheaper. The percentage is calculated based on the PAYG price and the Commitment Tier 100 effective per GB price at a specific location. A higher percentage means if you have a LAW instance at that location then you can benefit from switching to Tier100 from PAYG with a lower amount of ingested data.

Seems like the percentage values are correct in case of Sentinel.

I opened a QnA ticket to find out why the provided percentages and the real discounts differ heavily: https://docs.microsoft.com/en-us/answers/questions/506702/incorrect-percentage-values-on-the-azure-pricing-d.html waiting for MS to respond.