Supplement – Syslog Collector Testing

This blog expands upon the original Data Collection Rule creation and testing process to account for the specific requirements of Syslog + CEF collectors in your environment. 

Note: This is ONLY for clients that are not utilizing BlueVoyant managed collectors. 

Please ensure that you have read parts 1-Understand, 2-Plan, and 3-Test before working through this document. 

The first step is to come up with a replacement and cutover strategy depending on the risk appetite of the business to lose logs during the cutover. The recommendation to minimize lost logs would be to stand up an identical collector installation to your current environment, including a front-end load balancer if on exists for the MMA deployment, then start pointing test/quiet/non-critical devices and ensure that logs are coming in as expected, progressing into cutting over all of your devices. Alternatively, you can install the AMA on the same collectors as the MMA, but this could cause problems with double-ingestion of data, breaking ingestion entirely due to configuration changes, or even overloading the system’s resources causing the system to hang/crash. 

Create a Linux DCR to collect CEF (Common Event Format) Logs 

Note: This is a requirement if you are not using a BlueVoyant managed syslog collector.  If you are running your own Syslog server for collection of network/firewall logs, there are a couple things to keep in mind: 

1. The AMA is more performant than the MMA, so you may need less collectors if you have a multi-collector deployment (redundancy or scaling purposes). Please review the Microsoft Learn article for AMA Performance. 
2. The AMA handles Syslog & CEF collection differently than the MMA so you’ll need to review the options for avoiding data ingestion duplication available on Microsoft Learn. 
   1. The foremost recommendation is to dedicate certain facilities to CEF collection and certain facilities to Syslog collection. This provides the most straight-forward option, conserving collector resources and network bandwidth, but requires your devices to be configurable to send to specific target facilities. The configuration image in this guide (below) suggests configuring the CEF DCR to the local0-local7 facilities, while editing the Syslog DCR (created above) to only collect the traditional (non local*) Syslog facilities. 
   2. If changing the facility isn’t possible with your log sources, you can use an ingest time transformation within your Syslog DCR (created above) to drop CEF messages, while the CEF DCR (to be created below) will drop Syslog messages. 

Configuration 

1. Multiple rules can be created to allow granularity for different verbosity rules, different filtering, and multiple log analytics workspaces if needed. Rule creation can be summarized in the following three steps with detailed instructions further below.
   1. Assign resources (servers, Azure & ARC managed) that are sending logs 
   2. Select data sources (log events) to be monitored 
   3. Select destinations (log Analytics workspaces) to receive logs 
2. In the Azure portal, navigate to Sentinel -> workspace -> Data connectors -> Common Event Format (CEF) via AMA – > Open connector page -> +Create data collection rule 
   Note: If you cannot see this connector under Data connectors, you may have to install/update “Common Event Format” from the Content hub.  
3. Under the basics tab, fill in the following: 
   1. Rule Name 
   2. Subscription 
   3. Resource Group  
4. Under the Resources tab, select your syslog collector and any Linux machines that you would like to directly collect any logs from, then click Next: Collect> 
5.  Leave the checkbox selected to collect messages without PRI header then set all ONLY the local0-local7 facilities to the Minimum log level of LOG_INFO 
6. Select Next: Review + create > then Create. This will configure the data collection rule and queue up the AMA for installation on all selected systems. 
7. Finally, you will need to run the Microsoft provided script to configure rsyslog/syslog-Ng on your collectors to listen/receive remote logs. 
8. The last step of this onboarding process helps us avoid Full Disk scenarios which will prevent the AMA from functioning. We recommend that you set the rsyslog configuration not to store remote logs as a Full Disk scenario disrupts the function of the AMA. If using rsyslog , one such resolution is to modify your configuration file encapsulating your existent logging configuration as illustrated below with the yellow highlights: 

• rsyslog: /etc/rsyslog.conf 

if ($fromhost-ip == '127.0.0.1') then { 

auth,authpriv.* /var/log/auth.log ... 
... 
*.emerg :omusrmsg:* 

} 

Troubleshooting CEF Log Collection 

Let’s verify that we’re seeing event logs/messages from our expected systems. 

Note: You will need to either have test log data (example below) or have cutover a CEF system to see these logs. A Logger script to generate a CEF message is also included immediately below: 

logger -p local4.warn -P 514 -n 127.0.0.1 --rfc3164 -t CEF "0|BV-test|LOCAL|common=event-format-test|end|TRAFFIC|1|rt=$common=event-formatted-receive_time" 

Within Microsoft Sentinel, navigate to Logs then execute the following query where you should see CEF logs from your test systems. Note that these queries use the _ResourceId field which can help differentiate between the MMA and AMA on non-Azure systems; however, native Azure VMs also have this field even with the MMA so you will have to do additional testing after disabling/removing the MMA, ideally lowering the lookback period to only 1 day (1d) to ensure you’re still logging from systems as expected after cutover. 

• For Linux systems, the below query will show systems that are sending to the Heartbeat table but not sending to the Syslog table 

let HeartBeatLinux=Heartbeat 

| where TimeGenerated >ago(30d) 

| where OSType == "Linux" 

| where Category == "Azure Monitor Agent" 

| extend ResourceId=_ResourceId 

| summarize by Computer, ResourceId; 

HeartBeatLinux 

| join kind=leftanti (Syslog 

| where TimeGenerated >ago(30d) 

| extend ResourceId=_ResourceId 

| summarize by Computer, ResourceId) on ResourceId 

| summarize LinuxServerMissingSyslog=make_list(Computer) 

• For Linux systems, the below query will show systems that are sending to the Heartbeat table but not sending to the         CommonSecurityLog table 

let HeartBeatLinux=Heartbeat 

| where TimeGenerated >ago(30d) 

| where OSType == "Linux" 

| where Category == "Azure Monitor Agent" 

| extend ResourceId=_ResourceId 

| summarize by Computer, ResourceId; 

HeartBeatLinux 

| join kind=leftanti (CommonSecurityLog 

| where TimeGenerated >ago(30d) 

| extend ResourceId=_ResourceId 

| summarize by Computer, ResourceId) on ResourceId 

| summarize LinuxServerMissingCEF=make_list(Computer) 

Not getting either Syslog or CEF messages with the above commands? Assuming that you are getting local Syslog messages in the Part3-Test document following the Linux VM troubleshooting, review the Linux Syslog troubleshooting documentation on Microsoft Learn. Additionally, if you are getting Linux containers in the above queries you can add additional filtering based upon ResourceType. 

Thanks for joining us for this supplemental Syslog/CEF deployment doc. Please see our next post that discusses Deployment.