Powerful Capabilities of DCRs #1 - Direct type

DCRs are the modern backbone of data ingestion for Azure Sentinel, replacing legacy methods with a scalable, flexible, and consistent approach that uses a common data ingestion pipeline for all data sources. DCRs enable advanced filtering, transformation, and routing of data before it even hits your storage solution. In addition to these core features, DCRs offer a range of lesser-known and less frequently used functionalities that are well worth investigating for organizations looking to maximize their monitoring capabilities.

Beyond their core role in data collection, DCRs also offer additional features such as agent configuration, as well as the ability to pick up logs from alternative sources or forward them to storage destinations outside of just the Log Analytics workspace. This versatility makes DCRs a powerful tool for organizations seeking to optimize their monitoring infrastructure.

In upcoming posts, we will explore some of these lesser-known functionalities of DCRs, demonstrating how they can further enhance your data monitoring and management workflows.

Types of DCRs

Azure Monitor supports several different kinds of DCRs, each tailored for specific usage scenarios. The following table provides a comprehensive overview of all DCR types:

Most people are fairly familiar with the general log collection DCRs (Windows, Linux, WorkspaceTransforms, All). In this series of articles, I will highlight some of the less commonly known types and options, and share sample deployment code to assist with your testing.

This is going to be a four-part series:

1. Direct DCRs: In this post, I concentrate on the Direct DCR type, which simplifies log collection configuration for API-based sources.
2. Coming soon
3. Coming soon
4. Coming soon

Direct DCRs

Behavior of DCEs and DCRs

Direct Data Collection Rules (Direct DCRs) enable applications to send logs directly to a Log Analytics workspace using the Logs Ingestion API without requiring a separate Data Collection Endpoint (DCE). They are optimized for API-based log sources where agentless ingestion is preferred.

When picking up the logs from a machine via the AMA agent, the DCE (Data Collection Endpoint) usually has two separate roles:

Because of regional requirements (machine is not in the same region as your workspace), it is sometimes necessary to use two data collection endpoints (DCEs) for a single machine: one to retrieve configurations from and another to transmit data to. However, API-based data sources work differently than agent-based ones. API sources don’t need to fetch configurations because they send data directly to the DCE/DCR independently. Therefore, a single Data Collection Rule (DCR) with an integrated DCE is sufficient for API-based sources..

To illustrate this difference, let’s examine the code structure of a traditional non-Direct DCR setup: A non-Direct DCR without embedded DCE. The external DCE is attached to the DCR in this case.

The endpoints of an external DCE.

So, using Direct DCRs removes the need for conventional data collection endpoints. When you create a DCR with the “kind”: “Direct” setting, Azure automatically provisions embedded ingestion endpoints within the DCR. As a result, you only need to manage a single Azure resource - the DCR itself - while the DCE functionality is handled internally, accessible via the embedded ingestion endpoint URI.

Built-in log ingestion and metrics ingestion endpoints (DCE)

To send custom data to the Log Ingestion API, you require the appropriate permissions, the immutable ID of the DCR, the data collection endpoint URI, and the stream name (sometimes incorrectly referred to as the table name by certain tools). All of these required details are accessible when using a ‘Direct’ DCR, making this type of DCR fully suitable for custom API-based data ingestion scenarios. Thus, no additional enhancement needed for any of the existing data connectors which supported the separate DCE setup for API-based log forwarding.

Simplified ingestion without external DCE

Why Direct DCRs are not commonly used

1. Azure Portal’s table-creation GUI wizard defaults to creating DCRs without a kind attribute (‘All’ on the GUI) and requires a separate DCE. So, if you use the GUI for table creation you will ‘automatically’ have a different DCR type and a DCE created.
2. Official samples and documentation rarely use Direct DCRs; most tutorials focus on agent-based collection or DCE-backed DCRs.

My recommendations

Consider the following DCR management and implementation best practices to optimize your data collection strategy:

1. Create separate DCRs for multiple sources: While it’s technically possible to ingest multiple data streams through a single Direct DCR, I recommend creating individual DCRs when you have more than two sources. This approach simplifies both management and troubleshooting processes.
2. Use programmatic table creation: Create DCR-based tables programmatically rather than through the Azure Portal GUI to avoid unnecessary DCR and DCE resource creation.
3. Use the latest API version: Deploy using API version 2023-03-11 (the newest version at the time of writing) to access all configuration options and detailed information for Direct DCR types.

Getting started

Try the sample template: Use my ARM template available on GitLab to experiment with Direct DCR implementation. The template includes a default stream structure that you can customize before or after deployment.

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "dataCollectionRuleName": {
            "type": "string",
            "metadata": {
                "description": "Specifies the name of the Data Collection Rule to create."
            }
        },
        "location": {
            "defaultValue": "[resourceGroup().location]",
            "type": "string",
            "metadata": {
                "description": "Specifies the location in which to create the Data Collection Rule."
            }
        },
        "workspaceResourceId": {
            "type": "string",
            "metadata": {
                "description": "Specifies the Azure resource ID of the Log Analytics workspace (Sentinel) to use."
            }
        }
    },
    "resources": [
        {
            "type": "Microsoft.Insights/dataCollectionRules",
            "apiVersion": "2023-03-11",
            "name": "[parameters('dataCollectionRuleName')]",
            "location": "[parameters('location')]",
            "kind": "Direct",
            "properties": {
                "streamDeclarations": {
                    "Custom-DataTable": {
                        "columns": [
                            {
                                "name": "TimeGenerated",
                                "type": "datetime"
                            },
                            {
                                "name": "RawData",
                                "type": "string"
                            }
                        ]
                    }
                },
                "destinations": {
                    "logAnalytics": [
                        {
                            "workspaceResourceId": "[parameters('workspaceResourceId')]",
                            "name": "DestLogAnalytics"
                        }
                    ]
                },
                "dataFlows": [
                    {
                        "streams": [
                            "Custom-DataTable"
                        ],
                        "destinations": [
                            "DestLogAnalytics"
                        ],
                        "transformKql": "source",
                        "outputStream": "Custom-DataTable_CL"
                    }
                ]
            }
        }
    ],
    "outputs": {
        "dataCollectionRuleResourceId": {
            "type": "string",
            "value": "[resourceId('Microsoft.Insights/dataCollectionRules', parameters('dataCollectionRuleName'))]"
        },
        "immutableId": {
            "type": "string",
            "value": "[reference(resourceId('Microsoft.Insights/dataCollectionRules', parameters('dataCollectionRuleName'))).immutableId]"
        },
        "logsIngestionEndpoint": {
            "type": "string",
            "value": "[reference(resourceId('Microsoft.Insights/dataCollectionRules', parameters('dataCollectionRuleName'))).endpoints.logsIngestion]"
        },
        "metricsIngestionEndpoint": {
            "type": "string",
            "value": "[reference(resourceId('Microsoft.Insights/dataCollectionRules', parameters('dataCollectionRuleName'))).endpoints.metricsIngestion]"
        }
    }
}

Continuing the series

If you are interested in learning about some other lesser-known aspects of DCRs, check out another article in this four-part series:

1. Direct DCRs: In this current post I’m focusing on the Direct DCR kind that can simplify the log collection config for API-based sources.
2. Coming soon
3. Coming soon
4. Coming soon