Microsoft introduced a set of classes supporting diagnostics monitoring in the November 2009 Azure Services SDK. The classes are composed into two namespaces:

• Microsoft.WindowsAzure.Diagnostics
• Microsoft.WindowsAzure.Diagnostics.Management

The Diagnostics namespace comprises classes for the configuration and collection of diagnostic information. The Diagnostics.Management namespace comprises classes supporting local and remote management of the collection of diagnostics information and its transfer to persistent storage in Azure Tables and Blobs. This post is focused on the Diagnostics namespace and a subsequent post will consider the Diagnostics.Management namespace.

MSDN has a good section on Implementing Windows Azure Diagnostics for Hosted Services that includes the following sections:

• Initializing and Configuring Diagnostics Data Sources
• Transferring Diagnostic Data to Windows Azure Storage
• Managing Windows Azure Diagnostics Remotely

Matthew Kerner gave an excellent presentation on Windows Azure Monitoring, Logging, and Management APIs at PDC 2009 – he has posted his samples here. The November 2009 Update of the Windows Azure Platform Training Kit has a good hands-on-lab Windows Azure: Deploying and Monitoring Applications in Windows Azure with Example 3 being Monitoring Applications in Windows Azure. Sumit Mehrotra, from the Azure team, has a couple of posts – here and here – looking at Azure Diagnostics.

UPDATE 12/1/2010: This post describes some configuration changes to Azure Diagnostics implemented in the Azure SDK v1.3 release.

UPDATE 5/6/2011. I uploaded a Windows Azure Diagnostics sample to the MSDN Code Samples Gallery that demonstrates the features described in this post.

Diagnostics Information

Azure Diagnostics captures the following information:

• Directories – file-based data buffers (including IIS Logs, failed request logs and crash dumps)
• Diagnostic infrastructure logs – logs the diagnostics infrastructure itself
• Logs – Windows Azure logs
• Performance counters – Windows Performance Counters
• Windows Event Logs

Note that developers can create and configure their own user-defined file-based data buffers.

The diagnostic information is captured in data buffers in the local storage of each instance. Diagnostic infrastructure logs, IIS Logs, (Windows Azure) Logs are stored by default – the others must be initiated programmatically. Once captured, diagnostic information can be transferred either by schedule or on-demand to persistent Azure Storage.  Directories (file-based data buffers) are stored in Blobs and all other diagnostics information is is stored in Tables.

The data buffers containing the diagnostics information are configured through several data-buffer configuration classes. Diagnostics monitoring is configured through the DiagnosticMonitor and DiagnosticMonitorConfiguration classes.

Data Buffer Configuration

The data buffers are configured through a simple set of classes which persist the configuration in an Azure blob named wad-control-container. The configuration for each instance of each role is persisted in an individual blob in the container named with a combination of the deployment Id, role name and instance Id of the current role instance. For example:

deployment(301)/WebRole1/deployment(301).NovemberStorageClient.WebRole1.0

The diagnostics configuration classes can be invoked either from the running instance of the Azure Service or through, for example,  a remote desktop application interacting with the configuration through the diagnostics management and diagnostics configuration classes. Every Azure instance hosts a MonAgentHost.exe executable to manage the Azure diagnostics configuration and management process.

DiagnosticDataBufferConfiguration

DiagnosticDataBufferConfiguration is an abstract base class for the various diagnostics configuration classes representing data buffers. It is declared:

public abstract class DiagnosticDataBufferConfiguration {
    // Constructors
    protected DiagnosticDataBufferConfiguration();

// Properties
    public Int32 BufferQuotaInMB { get; set; }
    public TimeSpan ScheduledTransferPeriod { get; set; }
}

The DiagnosticsDataBufferConfiguration properties are exposed through its derived classes each of which represents a data buffer containing a specific type of diagnostics information. These derived classes are:

• BasicLogsBufferConfiguration
• DirectoriesBufferConfiguration
• PerformanceCountersBufferConfiguration
• WindowsEventLogsBufferConfiguration

BufferQuotaInMB specifies the maximum quota for an data buffer. The maximum overall quota is specified by DiagnosticMonitorConfiguration.OverallQuotaInMB. By default the overall quota is 4GB and there is no maximum quota specified for the individual buffers. (When an individual quota is reached I believe the policy is to delete the oldest records.) ScheduledTransferPeriod is the TimeSpan in minutes between scheduled transfers of the data buffer to Azure Storage.

BasicLogsBufferConfiguration

The BasicLogsBufferConfiguration class is derived from DiagnosticDataBufferConfiguration and supports configuration of (basic windows) logs. It is declared:

public class BasicLogsBufferConfiguration : DiagnosticDataBufferConfiguration {
    // Constructors
    public BasicLogsBufferConfiguration();

// Properties
    public LogLevel ScheduledTransferLogLevelFilter { get; set; }
}

The BasicLogsBufferConfiguration adds a ScheduledTransferLogLevelFilter to the BufferQuotaInMB and ScheduledTransferPeriod of the base DiagnosticDataBufferConfiguration class. ScheduledTransferLogLevelFilter specifies a filter level to limit the log information persisted to Azure Tables. LogLevel is the following enumeration:

public enum LogLevel {
    Undefined,
    Critical,
    Error,
    Warning,
    Information,
    Verbose
}

Note that when a filter level, here or another data buffer class, is set to a value from this enumeration it implicitly includes any levels earlier in the enumeration. Undefined has the same effect as Verbose.

The Logs are persisted in an Azure Table named:

• WadLogsTable

DirectoriesBufferConfiguration

The DirectoriesBufferConfiguration class is derived from DiagnosticsDataBufferConfiguration and supports configuration of file-based data buffers such as IIS Logs, failed request logs, crash dumps, and user-defined diagnostics. DirectoriesBufferConfiguration is declared:

public class DirectoriesBufferConfiguration : DiagnosticDataBufferConfiguration {
    // Constructors
    public DirectoriesBufferConfiguration();

// Properties
    public IList<DirectoryConfiguration> DataSources { get; }
}

The DirectoriesBufferConfiguration adds a list of DataSources to the BufferQuotaInMB and ScheduledTransferPeriod of the base DiagnosticDataBufferConfiguration class. The DataSources are instances of the DirectoryConfiguration class declared:

public class DirectoryConfiguration {
    // Constructors
    public DirectoryConfiguration();

// Properties
    public String Container { get; set; }
    public Int32 DirectoryQuotaInMB { get; set; }
    public String Path { get; set; }
}

Each directory has its own quota, DirectoryQuotaInMB, which is 1GB by default. Path specifies the absolute path of the directory containing the file-based data buffer such as the Crash Dumps (and eventually the IIS Logs). that contains the log Container names the Azure Blob container into which the diagnostic log will be transferred for persistence. The Azure provided file-based data buffers are transferred into the following containers by default:

• wad-iis-failedreqlogfiles
• wad-iis-logfiles
• wad-crash-dumps

Note that the IIS Failed Request Logs require that IIS be configured to capture the information. The feature is not turned on by default for performance reasons. The additional configuration comprises an additional section in the web.config file and is documented here.

Note that although this section implies that IIS Logs are configured like other Directories this does not appear to be true. Currently, the scheduled transfer of IIS Logs is configured through BasicLogsBufferConfiguration rather than DirectoriesBufferConfiguration. Although not confirming this, Matthew Kerner’s presentation states: Coming Soon: IIS Logs generated in role’s local data directory. I suspect that once this change happens IIS Logs will be configured through DirectoriesBufferConfiguration rather than BasicLogsBufferConfiguration.

UPDATE 3/25/2010: IIS Logs are now configured through DirectoriesBufferConfiguration.

PerformanceCountersBufferConfiguration

The PerformanceCountersBufferConfiguration class is derived from DiagnosticsDataBufferConfiguration and supports configuration of performance counters data buffers. PerformanceCountersBufferConfiguration is declared:

public class PerformanceCountersBufferConfiguration : DiagnosticDataBufferConfiguration {
    // Constructors
    public PerformanceCountersBufferConfiguration();

// Properties
    public IList<PerformanceCounterConfiguration> DataSources { get; }
}

The PerformanceCountersBufferConfiguration adds a list of DataSources to the BufferQuotaInMB and ScheduledTransferPeriod of the base DiagnosticDataBufferConfiguration class. The DataSources are instances of the PerformanceCounterConfiguration class declared:

public class PerformanceCounterConfiguration {
    // Constructors
    public PerformanceCounterConfiguration();

// Properties
    public String CounterSpecifier { get; set; }
    public TimeSpan SampleRate { get; set; }
}

CounterSpecifier specifies a Windows Performance Counter to be captured. This specification is in special syntax defined for Windows Performance Counters. An example of this syntax is:

@”Processor(*)% Processor Time”

This captures the % of processor time for all processors. Note that the names of performance counters on a development machine can be found by typing the following at a command prompt:

typeperf.exe /q

SampleRate is a TimeSpan specifying when the performance counter is measured. The SampleRate is rounded to the nearest second.

The Windows Performance Counters are persisted in an Azure Table named:

• WadPerformanceCountersTable

Note that each measurement of a performance counter is stored as one entity in the table.

WindowsEventLogsBufferConfiguration

The WindowsEventLogsBufferConfiguration class is derived from DiagnosticsDataBufferConfiguration and supports configuration of Windows Event Log data buffers. WindowsEventLogsBufferConfiguration is declared:

public class WindowsEventLogsBufferConfiguration : DiagnosticDataBufferConfiguration {
    // Constructors
    public WindowsEventLogsBufferConfiguration();

// Properties
    public IList<String> DataSources { get; }
    public LogLevel ScheduledTransferLogLevelFilter { get; set; }
}

The WindowsEventLogsBufferConfiguration adds a list of DataSources and a ScheduledTransferLogLevelFilter to the BufferQuotaInMB and ScheduledTransferPeriod of the base DiagnosticDataBufferConfiguration class. ScheduledTransferLogLevelFilter specifies a filter level taken to limit the log information persisted to Azure Tables. The filter level comes from the LogLevel enumeration. The DataSources are Strings in a special syntax defined for specifying how to consume events from Windows Event Logs. This syntax allows all events in an Event Log to be retrieved or only a filtered list of events.

The Windows Event Logs are persisted in an Azure Table named:

• WadWindowsEventLogsTable

Crash Dumps

There is one type of diagnostics data that is not configured through the normal data buffer configuration technique described here – crash dumps. These are initiated through the CrashDumps class declared:

public static class CrashDumps {
    // Methods
    public static void EnableCollection(Boolean enableFullDumps);
    public static void EnableCollectionToDirectory(String directory, Boolean enableFullDumps);
}

EnableCollection() enables either full or mini crash dumps. This data is placed in the CrashDumps directory in the same location as the other file-based Directories. EnableCollectionToDirectory() allows the specification of an alternative directory in local storage.

Note that ASP.Net handles almost all exceptions so that it is almost impossible to create a crash dump in a web role.

Diagnostics Monitoring

Diagnostics monitoring is supported by two classes: DiagnosticMonitor which controls the diagnostics monitoring process; and DiagnosticMonitorConfiguration which associates the data buffer configuration with the diagnostics monitoring process.

DiagnosticMonitorConfiguration

DiagnosticMonitorConfiguration associates the data buffer configuration with the diagnostics monitoring process. DiagnosticMonitorConfiguration is declared:

public class DiagnosticMonitorConfiguration {
    // Constructors
    public DiagnosticMonitorConfiguration();

// Properties
    public TimeSpan ConfigurationChangePollInterval { get; set; }
    public BasicLogsBufferConfiguration DiagnosticInfrastructureLogs { get; set; }
    public DirectoriesBufferConfiguration Directories { get; set; }
    public BasicLogsBufferConfiguration Logs { get; set; }
    public Int32 OverallQuotaInMB { get; set; }
    public PerformanceCountersBufferConfiguration PerformanceCounters { get; set; }
    public WindowsEventLogsBufferConfiguration WindowsEventLog { get; set; }
}

DiagnosticInfrastructureLogs, Directories, Logs, PerformanceCounters and WindowsEventLog provide access to an instance of each of the five Azure Diagnostics data buffers. ConfigurationChangePollInterval specifies the TimeSpan dictating when the diagnostics monitor polls for configuration changes. OverallQuotaInMB specifies the total amount of local storage allocated to all Azure Diagnostics data buffers.

DiagnosticMonitor

DiagnosticMonitor controls the Azure Diagnostics process by providing methods to start and stop diagnostics monitoring and to transfer the diagnostics logs to persistent Azure Storage. DiagnosticMonitor is declared:

public class DiagnosticMonitor {
    // Properties
    public static Boolean AllowInsecureRemoteConnections { get; set; }
    public String LocalDataDirectory { get; }

// Methods
    public static DiagnosticMonitorConfiguration GetDefaultInitialConfiguration();
    public void Shutdown();
    public static DiagnosticMonitor Start(String diagnosticsStorageAccountConfigurationSettingName,
       DiagnosticMonitorConfiguration initialConfiguration);
    public static DiagnosticMonitor Start(CloudStorageAccount storageAccount,
       DiagnosticMonitorConfiguration initialConfiguration);
    public static DiagnosticMonitor Start(String diagnosticsStorageAccountConfigurationSettingName);
    public void UpdateStorageAccount(CloudStorageAccount storageAccount);
}

A DiagnosticMonitor object is created using one of the static Start() methods parameterized by various combinations of CloudStorageAccount, DiagnosticMonitorConfiguration and storage account configuration setting name. The Start() methods actually starts the Monitoring Agent Host, MonAgentHost.exe, which provides for out-of-service diagnostics monitoring and management. Either a diagnosticsStorageAccountConfigurationSettingName String or a CloudStorageAccount must be supplied to provide the authentication required to access Azure Storage both to save the diagnostics configuration information and to persist the Azure Diagnostics data buffers. The Start() methods can be invoked either with a supplied initial DiagnosticMonitorConfiguration or using a default initial DiagnosticMonitorConfiguration. A common pattern is to retrieve a copy of the default initial configuration using the static GetDefaultInitialConfiguration() method, modify this configuration, and then use the modified configuration when Start() is invoked.

The basic example is:

DiagnosticMonitor diagnosticMonitor = DiagnosticMonitor.Start(“DiagnosticsConnectionString”);

This starts the Azure Diagnostics monitoring agent host and uses the Azure Storage configuration information identified by the name, DiagnosticsConnectionString, in the Azure Service configuration file. This name is merely a convention and is not mandatory.

A more sophisticated example is:

protected void StartDiagnosticMonitor()
{
    DiagnosticMonitorConfiguration diagnosticMonitorConfiguration =
       DiagnosticMonitor.GetDefaultInitialConfiguration();
    diagnosticMonitorConfiguration.Directories.ScheduledTransferPeriod = TimeSpan.FromMinutes(5.0);
    diagnosticMonitorConfiguration.Logs.ScheduledTransferPeriod = TimeSpan.FromMinutes(1.0);
    diagnosticMonitorConfiguration.WindowsEventLog.ScheduledTransferPeriod = TimeSpan.FromMinutes(5.0);
    diagnosticMonitorConfiguration.PerformanceCounters.ScheduledTransferPeriod = TimeSpan.FromMinutes(1.0);

diagnosticMonitorConfiguration.WindowsEventLog.DataSources.Add(“Application!*”);
    diagnosticMonitorConfiguration.WindowsEventLog.DataSources.Add(“System!*”);

PerformanceCounterConfiguration performanceCounterConfiguration = new PerformanceCounterConfiguration();
    performanceCounterConfiguration.CounterSpecifier = @”Processor(*)% Processor Time”;
    performanceCounterConfiguration.SampleRate = System.TimeSpan.FromSeconds(1.0);
    diagnosticMonitorConfiguration.PerformanceCounters.DataSources.Add(performanceCounterConfiguration);

CloudStorageAccount cloudStorageAccount = CloudStorageAccount.DevelopmentStorageAccount;
    DiagnosticMonitor diagnosticMonitor = DiagnosticMonitor.Start(
       cloudStorageAccount, diagnosticMonitorConfiguration);
}

This example retrieves the DiagnosticMonitor default initial configuration. It then specifies in this configuration various transfer periods to persistent Azure Storage for Directories, Logs, Windows Event Logs. and Windows Performance Counters. It adds Event Log data sources for the Windows Application and System Event Logs to the configuration. Then it configures the collection of a Windows Performance Counter for the %processor time for all processors with data collection taking place every second. Finally, it retrieves the built-in DevelopmentStorageAccount and starts the Monitoring Agent Host by invoking the DiagnosticMonitor Start() method.

Note. I got an error in the Start() method when I tried to use a CloudStorageAccount created as follows:

CloudStorageAccount cloudStorageAccount = CloudStorageAccount.FromConfigurationSetting(“DiagnosticsConnectionString”);

This happened even though the Azure Service configuration file had the following:

<Setting name=”DiagnosticsConnectionString” value=”UseDevelopmentStorage=true” />

and CloudStorageAccount.SetConfigurationSettingPublisher() had been invoked.

This problem could easily be due to user error.

The AllowInsecureRemoteConnections property specifies whether or not the Monitoring Agent Host can communicate with Azure Storage using http instead of https. LocalDataDirectory  gets the full path in local storage where the Azure Diagnostics data files and configuration are stored.

The Shutdown() method stops the Monitoring Agent Host and stops the collection of diagnostics data. UpdateStorageAccount() allows new CloudStorageAccount information to be supplied to the Monitoring Agent Host.

Tracing to the Log

Azure has a special trace listener, DiagnosticMonitorTraceListener, that writes Trace information to the Azure log files. An Azure role is configured to use the DiagnosticMonitorTraceListener by placing the following in the web.config or app.config file as appropriate:

<system.diagnostics>
    <trace>
        <listeners>
            <add type=”Microsoft.WindowsAzure.Diagnostics.DiagnosticMonitorTraceListener,
                 Microsoft.WindowsAzure.Diagnostics, Version=1.0.0.0, Culture=neutral,
                 PublicKeyToken=31bf3856ad364e35″ name=”AzureDiagnostics”>
                <filter type=”” />
            </add>
        </listeners>
    </trace>
</system.diagnostics>

Tracing is then performed by using static methods in the System.Diagnostics.Trace class. For example, TraceInformation() writes an informational trace to the Azure logs while TraceWarning() writes a warning trace to the event logs. These log levels can later be used to filter messages during analysis of the trace output. The Trace class also provides basic Write() and WriteIf() methods, the latter writing a trace depending on the Boolean value of a parameter. For example, the following writes the instance Id of the current instance as an informational trace to the Azure logs:

protected void WriteTraceInformation()
{
    String roleInstance = RoleEnvironment.CurrentRoleInstance.Id;
    System.Diagnostics.Trace.TraceInformation(“Role instance: ” + roleInstance);
}

UPDATE 10/13/2010 – A post on the Dot Net Solutions blog warns that Trace.WriteLine() issues only Verbose messages.

Accessing Logs

The Azure Diagnostics logs are accessible once they have been persisted to Azure Storage. They can then be viewed using an application like Azure Storage Explorer or Cloud Storage Studio. They can also be downloaded from Azure Storage to a local system for further analysis.

UPDATE 3/25/2010

Confirmed that IIS Logs are now configured through DirectoriesBufferConfiguration.