Test Your Migration

In our blog series, we’ll review the steps required for migrating from the MMA (Microsoft Monitoring Agent), also known as the Log Analytics agent, to the AMA (Azure Monitor Agent).

In this third post we’ll review how to build your test plan and test your migration. 

Please ensure that you have read parts 1-Understand and 2-Plan before working through this document. 

The objective of this testing phase is to identify a pilot group of machines to test your rollout, including onboarding to Azure Arc (if non-Azure native), configuring Data Collection rules which will push the AMA to systems, and to ensure systems are connected into Sentinel and logging as expected. 

Note that this is a very prescriptive which makes for a much longer article than the others, so grab yourself a coffee and buckle up! 

Onboarding to Azure Arc (non-Azure systems) 

As discussed in the 2-Plan post, you should have decided how you are going onboard your systems to Azure Arc at this point. We recommend testing one machine with interactive onboarding to ensure successful permissions and Azure connectivity, then moving on to testing the full onboarding process with at-scale onboarding for this test/pilot group.  

At this time, follow your onboarding plan to onboard your pilot/test systems to Azure Arc. Within 15-or-so minutes of Azure Arc onboarding, the systems should appear in the Azure portal under Azure Arc -> Machines. 

Verifying Azure Arc onboarding success 

1. Navigate to Azure Arc in the search   
2. Select Servers on the left-hand side, then review to ensure the expected systems appear and are showing with a status of Connected, meaning that they have been been onboarded to Azure Arc successfully.

For troubleshooting, please see the Microsoft Learn article Troubleshoot Azure Connected Machine agent connection issues.

Deployment of Azure Monitor Agent (AMA) (all systems)

Note: If your organization is using user-assigned managed identities for cross-Azure resource access (key-vault or other shared resources), you will need to utilize Azure Policy to deploy the AMA and connect DCRs for log collection. The issue stems from the fact that attaching Data Collection Rules (DCRs) with the Azure UI to systems will cause a system-assigned managed identity to be created on systems which will disable the user-assigned managed identity, subsequently breaking the cross-Azure resource access. If this affects you, please reach out to your CSM for supplemental documentation for Azure policy.

Create a Windows DCR to Collect System/Application Events 

Multiple rules can be created to allow granularity for different verbosity rules, different filtering, and for multiple log analytics workspaces if needed. Rule creation can be summarized in three steps with detailed instructions further below.

• Assign resources (servers, Azure VMs & Azure Arc-connected) that are sending logs
• Select data sources (log events) to be monitored

• Select destinations (Log Analytics/Sentinel workspaces) to receive logs

  In the Azure portal, navigate to Monitor -> Data Collection Rule -> Create

Select the below Data Sources

Note: do not enable Security logging in this step as we want the Security logs to go to the Security Event table instead of the Event table.

Select Destination, which will be your Sentinel workspace

Create a Windows DCR to Collect Security Events

In the Azure portal, navigate to Sentinel -> workspace -> Data connectors -> select Windows Security Events Via AMA and click Open connector page – > +Create Data Collection Rule

Note: If you cannot see this connector under Data connectors, you may have to install/update “Windows Security Events” from the Content hub.

Under the basics tab, fill in the following:

• Rule Name
• Subscription
• Resource group

Then click next: Resource

Under Resource click +Add resource(s)

Select your resource(s) or Scope, then click Apply

Under the Collect Tab select common and click Next: Review +create and then click create

Create a Linux DCR to collect traditional Syslog Messages

Multiple rules can be created to allow granularity for different verbosity rules, different filtering, and multiple log analytics workspaces if needed. Rule creation can be summarized in the following three steps with detailed instructions further below.

• Assign resources (servers, Azure & ARC managed) that are sending logs
• Select data sources (log events) to be monitored
• Select destinations (log Analytics workspaces) to receive logs

In the Azure portal, navigate to Sentinel -> workspace -> Data connectors -> Windows Syslog via AMA – > Open connector page -> +Create data collection rule

Under the basics tab, fill in the following:

• Rule Name
• Subscription
• Resource Group

Under the Resources tab, select your syslog collector and any Linux machines that you would like to directly collect any logs from, then click Next: Collect>

Leave the checkbox selected to collect messages without PRI header then set all available facilities to the Minimum log level of LOG_INFO

Finally, select Next: Review + create > then Create. This will configure the data collection rule and queue up the AMA for installation on all selected systems.

Confirm AMA installation is successful on all systems

For Azure VMs

1. Navigate to Virtual Machines in the search bar 
2. Select the virtual machine in question

Navigate to Extensions + applications and you should see the AMA extension installed|

For Azure Arc-connected systems

1. Navigate to Azure Arc in the search  
2. Select Servers on the left hand side then the Azure Arc-connected machine in question
3. Navigate to Extensions and you should see the AMA extension installed.

Confirm Log collection from systems

Note: You may have to wait 15-30 minutes after configuring and attaching the DCRs above before logs start showing up in your Sentinel instance.

• We will start with Heartbeats. The Heartbeat table receives periodic heartbeats that include host and agent information from the underlying host. This lets us know that the AMA is installed and successfully pushing Heartbeats to the Sentinel instance.

  Within Microsoft Sentinel, navigate to Logs then execute the following queries where you should see your test systems:

  • For Windows systems

    Heartbeat

    | where Category == "Azure Monitor Agent"

    | where OSType == "Windows"

    | summarize arg_max(TimeGenerated,*) by Computer

     

  • For Linux systems

    Heartbeat

    | where Category == "Azure Monitor Agent"

    | where OSType == "Linux"

    | summarize arg_max(TimeGenerated,*) by Computer

  • Next, let’s verify that we’re seeing event logs/messages from our expected systems.
    • Within Microsoft Sentinel, navigate to Logs then execute the following queries where you should see your test systems. Note that these queries use the _ResourceId field which can help differentiate between the MMA and AMA on non-Azure systems; however, native Azure VMs also have this field even with the MMA so you will have to do additional testing after disabling/removing the MMA, ideally lowering the lookback period to only 1 day (1d) to ensure you’re still logging from systems as expected after cutover.

      • For Windows systems, the below query will show systems that are sending to the Heartbeat table but not sending to either the Event or SecurityEvent table. 

        let HeartBeatWindows=Heartbeat

        | where TimeGenerated >ago(30d)

        | where OSType == "Windows"

        | where Category == "Azure Monitor Agent"

        | extend ResourceId=_ResourceId

        | summarize by Computer, ResourceId;

        HeartBeatWindows

        | join kind=leftanti (SecurityEvent

        | where TimeGenerated >ago(30d)

        | extend ResourceId=_ResourceId

        | summarize by Computer, ResourceId) on ResourceId

        | summarize WindowsAMAmissingSecurityEvents=make_list(Computer)

        | extend ID=1

        | join kind=fullouter (HeartBeatWindows

        | join kind=leftanti (Event

        | where TimeGenerated >ago(30d)

        | extend ResourceId=_ResourceId

        | summarize by Computer, ResourceId) on ResourceId

        | summarize WindowsAMAissingSystemEvents=make_list(Computer)

        | extend ID=1) on ID

        | project-away ID, ID1

        For Linux systems, the below query will show systems that are sending to the Heartbeat table but not sending to the Syslog table.

        let HeartBeatLinux=Heartbeat

        | where TimeGenerated >ago(30d)

        | where OSType == "Linux"

        | where Category == "Azure Monitor Agent"

        | extend ResourceId=_ResourceId

        | summarize by Computer, ResourceId;

        HeartBeatLinux

        | join kind=leftanti (Syslog

        | where TimeGenerated >ago(30d)

        | extend ResourceId=_ResourceId

        | summarize by Computer, ResourceId) on ResourceId

        | summarize LinuxServerMissingSyslog=make_list(Computer)

Troubleshooting AMA installation or AMA Log Collection

Depending on the system that is failing to install the AMA or failing to log to Microsoft Sentinel, please follow the guidance for the individual Operating System:

• Windows VM
• Windows Arc-enabled server
• Linux VM
• Linux Syslog 

Additionally, there is both a Windows and a Linux troubleshooter that can be run to help identify issues.

If you are not utilizing BlueVoyant managed collectors and are standing up a Syslog collector to receive Syslog and Common Event Format logs from other hosts/devices, please review our part 3 supplemental blog post for Syslog Collector Testing.

All others, please see our next post that discusses Deployment.