Deploy, Assess, and Complete

In our blog series, we’ve reviewed the steps required for migrating from the MMA (Microsoft Monitoring Agent), also known as the Log Analytics agent, to the AMA (Azure Monitor Agent).

Time is short and organizations need to start migrating now. MMA will be retired on August 31, 2024.

The purpose of this final post is to Deploy, Assess, and Repeat until your migration is completed successfully.

At this point in our journey, we will run through our deployment plan built in Step 2, dividing our environment into bite-sized steps for migration. We will onboard on-premises systems with Azure Arc, ideally with an at-scale method as identified and tested in our 3. In the test phase, connect these hosts to the Data Collection Rules to push the AMA and begin collecting logs.

As a heads-up, this document will look very similar to our Testing document, with some minor changes. We are no longer creating new/fresh Data Collection Rules and instead attaching the existing DCRs to more systems.

As discussed in the 2-Plan post and tested in 3-Test, you should have decided and tested Arc onboarding. Follow your onboarding plan to onboard your remaining systems to Azure Arc. Within 15-or-so minutes of Azure Arc onboarding, the systems should appear in the Azure portal under Azure Arc -> Machines.

Verifying Azure Arc Onboarding Success

1. Navigate to Azure Arc in the search.
2. Select Servers on the left-hand side, then review to ensure the expected systems appear and are showing with a status of Connected, meaning that they have been been onboarded to Azure Arc successfully.

For troubleshooting, please see the Microsoft Learn article: Troubleshoot Azure Connected Machine agent connection issues.

Reminder: If your organization uses user-assigned managed identities for cross-Azure resource access (key-vault or other shared resources), you must utilize Azure Policy to deploy the AMA and connect DCRs for log collection. The issue stems from the fact that attaching Data Collection Rules (DCRs) with the Azure UI to systems will cause a system-assigned managed identity to be created on systems, which will disable the user-assigned managed identity, subsequently breaking the cross-Azure resource access. If this affects you, please contact your CSM for supplemental documentation for Azure policy.

Add Windows Servers to the DCR to Collect System/Application Events

1. In the Azure portal, navigate to Monitor -> Data Collection Rules, and select the Data Collection rule made for Windows System/Application Events in 3. Test

2. On the left side under configuration, select Resources then +Add

   

3. Select all of the hosts/scopes (leaving the ones selected that you would still like to collect from) that you would like to be a part of the DCR then click Apply

   

4. This will kick off the process to install the AMA extension and push the DCR to the hosts

Add Windows Servers to the DCR to Collect Security Events

1. In the Azure portal, navigate to Sentinel -> workspace -> Data connectors -> select Windows Security Event Via AMA and click Open connector page -> click the pencil icon (for edit) next to the Data Collection rule made for Windows Security Events in 3.Test
   

   

2. Click the Resources tab and Select all of the hosts/scopes (leaving the ones selected that you would still like to collect from) that you would like to be part of the DCR. Then click Next (repeat 2x) and click Create on the Review + Create tab

   

3. This will kick off the process to install the AMA extension and push the DCR to the hosts

Add Linux Servers to the DCR to Collect Syslog Events

1. In the Azure portal, navigate to Sentinel -> workspace -> Data connectors -> select Syslog via AMA and click Open connector page -> click the pencil icon (for edit) next to the Data Collection rule made for Windows Security Events in 3.Test
   

   

2. Click the Resources tab and Select all of the hosts/scopes (leaving the ones selected that you would still like to collect from) that you would like to be a part of the DCR then click through Next (repeat 2x) then click Create on the Review + create tab

   

3. This will kick off the process to install the AMA extension and push the DCR to the hosts.
   
   

For Azure VMs

1. Navigate to Virtual Machines in the search bar
2. Select the virtual machine in question

3. Navigate to Extensions + applications and you should see the AMA extension installed

   

For Azure Arc-Connected Systems

1. Navigate to Azure Arc in the search
2. Select Servers on the left hand side then the Azure Arc-connected machine in question
3. Navigate to Extensions and you should see the AMA extension installed
   

Note: You may have to wait 15-30 minutes after configuring and attaching the DCRs above before logs start showing up in your Sentinel instance.

1. We will start with Heartbeats. The Heartbeat table receives periodic heartbeats that include host and agent information from the underlying host. This lets us know that the AMA is installed and successfully pushing Heartbeats to the Sentinel instance.
   1. Within Microsoft Sentinel, navigate to Logs then execute the following queries where you should see your test systems:

      1. For Windows systems:

         Heartbeat
         | where Category == "Azure Monitor Agent"
         | where OSType == "Windows"
         | summarize arg_max(TimeGenerated,*) by Computer
         

      2. For Linux systems:

         Heartbeat
         | where Category == "Azure Monitor Agent"
         | where OSType == "Linux"
         | summarize arg_max(TimeGenerated,*) by Computer
         

2. Next, let’s verify that we’re seeing event logs/messages from our expected systems.

   1. Within Microsoft Sentinel, navigate to Logs then execute the following queries where you should see your test systems. Note that these queries use the _ResourceId field which can help differentiate between the MMA and AMA on non-Azure systems; however, native Azure VMs also have this field even with the MMA so you will have to do additional testing after disabling/removing the MMA, ideally lowering the lookback period to only 1 day (1d) to ensure you’re still logging from systems as expected after cutover.

      1. For Windows systems, the below query will show systems that are sending to the Heartbeat table but not sending to either the Event or SecurityEvent table

      let HeartBeatWindows=Heartbeat
      | where TimeGenerated >ago(30d)
      | where OSType == "Windows"
      | where Category == "Azure Monitor Agent"
      | extend ResourceId=_ResourceId
      | summarize by Computer, ResourceId;
      HeartBeatWindows
      | join kind=leftanti (SecurityEvent
      | where TimeGenerated >ago(30d)
      | extend ResourceId=_ResourceId
      | summarize by Computer, ResourceId) on ResourceId
      | summarize WinServerMissingSecurityEvents=make_list(Computer)
      | extend ID=1
      | join kind=fullouter (HeartBeatWindows
      | join kind=leftanti (Event
      | where TimeGenerated >ago(30d)
      | extend ResourceId=_ResourceId
      | summarize by Computer, ResourceId) on ResourceId
      | summarize WinServerMissingSystemEvents=make_list(Computer)
      | extend ID=1) on ID
      | project-away ID, ID1
      

      1. For Linux systems, the below query will show systems that are sending to the Heartbeat table but not sending to the Syslog table

      let HeartBeatLinux=Heartbeat
      | where TimeGenerated >ago(30d)
      | where OSType == "Linux"
      | where Category == "Azure Monitor Agent"
      | extend ResourceId=_ResourceId
      | summarize by Computer, ResourceId;
      HeartBeatLinux
      | join kind=leftanti (Syslog
      | where TimeGenerated >ago(30d)
      | extend ResourceId=_ResourceId
      | summarize by Computer, ResourceId) on ResourceId
      | summarize LinuxServerMissingSyslog=make_list(Computer)

Troubleshooting AMA Installation or AMA Log Collection

Depending on the system that is failing to install the AMA or failing to log to Microsoft Sentinel, please follow the guidance for the individual Operating System: 

• Windows VM 
• Windows Arc-enabled server 
• Linux VM 
• Linux Syslog  

Additionally, there is both a Windows and a Linux troubleshooter that can be run to help identify issues. 

At this point, we’ve successfully deployed the AMA and are collecting logs into Sentinel for a subset of our production systems. Head back up to the top of this blog and repeat as necessary until all systems have been onboarded successfully.

We recommend using the Azure Monitor Agent Migration Helper workbook to help track migration status. Please review and use to help track and ensure completion of your migration.

These guides walked through the bulk of Windows and Linux log collection; however, there are other Windows and Linux logs that you may want to pick up in your environment. To ensure that you are collecting all required logs that you were previously collecting with the MMA, we recommend performing a review your legacy Log Analytics data sources in Azure Monitor as well as your Sentinel log sources and tables. After identifying any additional logs/tables what you are collecting today with the MMA, ensure that you configure the AMA to collect the same logs. Microsoft has provided AMA collection guides for IIS logs, DNS logs, and custom Windows Event Logs (i.e. SQL) as well as DHCP and other custom text logs.

First, let us be the first to congratulate you for successfully migrating from the MMA to the AMA!

Thanks for joining on this journey to migrate from the MMA to the AMA. If you’d like to evaluate Azure Policy to continuously deploy the AMA and DCRs to Windows servers, please contact your CSM for any additional documentation.