Test Your Migration

As discussed in part 2, Plan, you should have decided how you are going to onboard your systems to Azure Arc at this point. We recommend testing one machine with interactive onboarding to ensure successful permissions and Azure connectivity, then moving on to testing the full onboarding process with at-scale onboarding for this test/pilot group.  

Now, follow your onboarding plan to onboard your pilot/test systems to Azure Arc. Within 15-or-so minutes of Azure Arc onboarding, the systems should appear in the Azure portal under Azure Arc -> Machines. 

Verifying Azure Arc Onboarding Success 

1. Navigate to Azure Arc in the search 
2. Select Servers on the left-hand side, then review to ensure the expected systems appear and are showing with a status of Connected, meaning that they have been been onboarded to Azure Arc successfully. 

For troubleshooting, please see the Microsoft Learn article: Troubleshoot Azure Connected Machine agent connection issues.

Note: If your organization is using user-assigned managed identities for cross-Azure resource access (key-vault or other shared resources), you will need to utilize Azure Policy to deploy the AMA and connect DCRs for log collection. The issue stems from the fact that attaching Data Collection Rules (DCRs) with the Azure UI to systems will cause a system-assigned managed identity to be created on systems which will disable the user-assigned managed identity, subsequently breaking the cross-Azure resource access. If this affects you, please reach out to your CSM for supplemental documentation for Azure policy.

Create a Windows DCR to Collect System/Application Events

1. Multiple rules can be created to allow granularity for different verbosity rules, different filtering, and for multiple log analytics workspaces if needed. Rule creation can be summarized in three steps with detailed instructions further below.
   1. Assign resources (servers, Azure VMs & Azure Arc-connected) that are sending logs
   2. Select data sources (log events) to be monitored

2. Select destinations (Log Analytics/Sentinel workspaces) to receive logsIn the Azure portal, navigate to Monitor -> Data Collection Rule -> Create
   

   

3. Select the below Data Sources
   Note: do not enable Security logging in this step as we want the Security logs to go to the Security Event table instead of the Event table.
   

   

4. Select Destination, which will be your Sentinel workspace
   

   

Create a Windows DCR to Collect Security Events

1. In the Azure portal, navigate to Sentinel -> workspace -> Data connectors -> select Windows Security Events Via AMA and click Open connector page – > +Create Data Collection Rule
   Note: If you cannot see this connector under Data connectors, you may have to install/update “Windows Security Events” from the Content hub.
   

   

2. Under the basics tab, fill in the following:
   1. Rule Name
   2. Subscription
   3. Resource Group

3. Then click next: Resource
   

   

4. Under Resource click +Add resource(s)
   

   

5. Select your resource(s) or Scope, then click Apply
   

   

6. Under the Collect Tab select common and click Next: Review +create and then click create
   

   

Create a Linux DCR to Collect Traditional Syslog Messages

1. Multiple rules can be created to allow granularity for different verbosity rules, different filtering, and multiple log analytics workspaces if needed. Rule creation can be summarized in the following three steps with detailed instructions further below.
   1. Assign resources (servers, Azure & ARC managed) that are sending logs
   2. Select data sources (log events) to be monitored
   3. Select destinations (log Analytics workspaces) to receive logs

2. In the Azure portal, navigate to Sentinel -> workspace -> Data connectors -> Windows Syslog via AMA – > Open connector page -> +Create data collection rule
   

   

3. Under the basics tab, fill in the following:
   1. Rule Name
   2. Subscription

   3. Resource Group
      

      

4. Under the Resources tab, select your syslog collector and any Linux machines that you would like to directly collect any logs from, then click Next: Collect>
   

   

5. Leave the checkbox selected to collect messages without PRI header then set all available facilities to the Minimum log level of LOG_INFO
   

   

6. Finally, select Next: Review + create > then Create. This will configure the data collection rule and queue up the AMA for installation on all selected systems.

For Azure VMs:

1. Navigate to Virtual Machines in the search bar
2. Select the virtual machine in question

3. Navigate to Extensions + applications and you should see the AMA extension installed
   

   

For Azure Arc-Connected Systems

1. Navigate to Azure Arc in the search
2. Select Servers on the left-hand side then the Azure Arc-connected machine in question

3. Navigate to Extensions and you should see the AMA extension installed
   

   

Note: You may have to wait 15-30 minutes after configuring and attaching the DCRs above before logs start showing up in your Sentinel instance.

1. We will start with Heartbeats. The Heartbeat table receives periodic heartbeats that include host and agent information from the underlying host. This lets us know that the AMA is installed and successfully pushing Heartbeats to the Sentinel instance.
   1. Within Microsoft Sentinel, navigate to Logs then execute the following queries where you should see your test systems:

      1. For Windows systems:

         Heartbeat
         | where Category == "Azure Monitor Agent"
         | where OSType == "Windows"
         | summarize arg_max(TimeGenerated,*) by Computer

      2. For Linux systems

         Heartbeat
         | where Category == "Azure Monitor Agent"
         | where OSType == "Linux"
         | summarize arg_max(TimeGenerated,*) by Computer

2. Next, let’s verify that we’re seeing event logs/messages from our expected systems.
   1. Within Microsoft Sentinel, navigate to Logs then execute the following queries where you should see your test systems. Note that these queries use the _ResourceId field which can help differentiate between the MMA and AMA on non-Azure systems; however, native Azure VMs also have this field even with the MMA so you will have to do additional testing after disabling/removing the MMA, ideally lowering the lookback period to only 1 day (1d) to ensure you’re still logging from systems as expected after cutover.

      1. For Windows systems, the below query will show systems that are sending to the Heartbeat table but not sending to either the Event or SecurityEvent table:

         let HeartBeatWindows=Heartbeat
         | where TimeGenerated >ago(30d)                                                    
         | where OSType == "Windows"
         | where Category == "Azure Monitor Agent"
         | extend ResourceId=_ResourceId
         | summarize by Computer, ResourceId;
         HeartBeatWindows
         | join kind=leftanti (SecurityEvent
         | where TimeGenerated >ago(30d)
         | extend ResourceId=_ResourceId
         | summarize by Computer, ResourceId) on ResourceId
         | summarize WindowsAMAmissingSecurityEvents=make_list(Computer)
         | extend ID=1
         | join kind=fullouter (HeartBeatWindows
         | join kind=leftanti (Event
         | where TimeGenerated >ago(30d)
         | extend ResourceId=_ResourceId
         | summarize by Computer, ResourceId) on ResourceId
         | summarize WindowsAMAissingSystemEvents=make_list(Computer)
         | extend ID=1) on ID
         | project-away ID, ID1

      2. For Linux systems, the below query will show systems that are sending to the Heartbeat table but not sending to the Syslog table.

         let HeartBeatLinux=Heartbeat
         | where TimeGenerated >ago(30d)
         | where OSType == "Linux"
         | where Category == "Azure Monitor Agent"
         | extend ResourceId=_ResourceId
         | summarize by Computer, ResourceId;
         HeartBeatLinux
         | join kind=leftanti (Syslog
         | where TimeGenerated >ago(30d)
         | extend ResourceId=_ResourceId
         | summarize by Computer, ResourceId) on ResourceId
         | summarize LinuxServerMissingSyslog=make_list(Computer)

Troubleshooting AMA Installation or AMA Log Collection

Depending on the system that is failing to install the AMA or failing to log to Microsoft Sentinel, please follow the guidance for the individual Operating System:

• Windows VM
• Windows Arc-enabled server
• Linux VM
• Linux Syslog 

Additionally, there is both a Windows and a Linux troubleshooter that can be run to help identify issues.

If you are not utilizing BlueVoyant managed collectors and are standing up a Syslog collector to receive Syslog and Common Event Format logs from other hosts/devices, please review our supplemental blog post for Syslog Collector Testing.

All others, please see our next post that discusses Deployment.