Troubleshooting and Common Questions

This chapter describes how to view and manage log files and diagnostics for components of the auditing infrastructure on UNIX computers. This chapter also describes how to identify and resolve common problems you might encounter when auditing user activity or managing the auditing infrastructure.

Checking the Status of the UNIX Agent

After you install and enable auditing for a UNIX computer, you can check the status of the agent using the dainfo command to verify the connection to the correct installation. For example, the agent might not automatically connect to the installation if you use an installation name other than DefaultInstallation.

To check the status of the agent and the auditing infrastructure, run the following command as a user with root privileges:

dainfo --diag

The --diag option returns detailed information about the local computer and about the installations, audit stores, trusted collectors, trusted agents, and the active audit store database that the agent is sending its data to. The diagnostic output also includes details about the Active Directory location and object identifier for each installation.

Configuring the Installation for an Agent

If the command indicates that the status is offline or the installation is not configured, use dacontrol to explicitly identify the correct installation. For example:

dacontrol -i installation_name

You can then rerun dainfo --diag to verify the installation is configured correctly. Note that you cannot use dacontrol to connect to a different installation name if the installation is configured using the Installation group policy. In a secure installation, the Installation group policy identifies the Active Directory location that contains the service connection point object for the installation. If you are not using group policy to identify the installation, you can manually configure agents and collectors to use a specific installation name.

Checking for Disconnected Agents using Audit Manager

You can also use Audit Manager to see the status of all agents in the installation. I any agent is listed as Disconnected, you should check whether the audited computer is shut down, If the audited computer is not shut down, the agent might be outside the scope of any audit store or unable to find a collector. Use the diagnostic services to check communication between components.

Starting and Stopping the UNIX Agent

In most cases, the UNIX agent is automatically started when an audited computer is first powered on and remains running until the audited computer is shut down. Starting the agent when a computer starts up ensures the agent can capture activity for all shell sessions.

Although you typically start and stop the dad process as part of a computer's startup and shutdown scripts, you can also start the agent directly from the command line on a local computer.

If the agent is not running, run the following command to start it:

/usr/share/centrifydc/bin/centrifyda start

Detecting the Server Suite Installation Status

If you're encountering any issues with your Server Suite installation, you can run the dacheck program on your UNIX computers. The dacheck command detects the following errors in your Server Suite installation:

  • Auditing binaries linkage problems
  • Disk space
  • DNS, collector, dad, adclient health
  • Logging status
  • Auditing file permissions/ownership
  • Auditing installation configuration
  • If ActiveDirectory joined
  • Auditing database integrity
  • If root in user.ignore and other criteria that affect root login
  • var/centrifyda, /tmp write permission
  • nsswitch.conf (or method.cfg, user.cfg for AIX)
  • Selinux status
  • Nscd (pwgrd) status
  • User's cdax/real shell existence, permission, ownership.
  • DNS Reverse lookup for collector's hostname
  • Report Domain Controller

To check the status of the agent and the auditing infrastructure, run the following command as a user with root privileges:

  • dacheck

The dacheck command is available in the same location as the adcheck command: /usr/share/centrifydc/bin.

Viewing and Changing Log File Settings

Log files are text files that record information about operations performed by auditing components on a local computer. If you have administrative privileges on a computer, you can open log files with any text editor.

You can view log files, change the location of the log file, and change the level of detail recorded in the log file from the Log Settings dialog box. Depending on the computer you are using, there are different ways to open the Log Settings.

Enabling Detailed Logging for Linux and UNIX Computers

In most cases, troubleshooting auditing-related issues requires information about the operation of the agent, the collector service, and database activity. For performance reasons, you should only enable agent logging when you need to capture detailed information about agent operations. For troubleshooting purposes, however, you can use the dadebug command to turn on detailed logging.

To enable audit-related logging on audited Linux or UNIX computers

  1. Switch to the root user.

  2. Run the dadebug clear command to remove any existing detailed logging from previous operations.

    dadebug clear

  3. Run the dadebug on command to enable detailed logging on for audit-related agent operations.

    dadebug on

    Detailed messages are recorded in the /var/log/centrifydc.log file. You can view the contents of the log file with a text editor. In most cases, however, you should collect additional information and send all of the logged information to Delinea Support.

  4. Restart the auditing service.

    /usr/share/centrifydc/bin/centrifyda restart

  5. Run the dainfo diagnostic command and save the output to a text file.

    dainfo --diag > /tmp/dainfo.txt

  6. Run the adinfo diagnostic command and save the output to a text file.

    adinfo --diag > /tmp/adinfo.txt

  7. Stop detailed logging of audit-related activity.

    dadebug off

  8. Send an email to Delinea Support with the log files and the agent configuration file as an attachment.

    /var/log/centrifydc.log /tmp/dainfo.txt/tmp/adinfo.txt /etc/centrifyda/centrifyda.conf

To check whether detailed logging is enabled:

  1. Run dadebug without parameters to see if detailed logging is currently enabled.

    dadebug

    Centrify DirectAudit debug logging is on.

  2. Run addebug without parameters to see if detailed logging is currently enabled.

    addebug

  3. Run addebug off to disable logging, if needed.

Enabling Detailed Logging for the Collector Service

If you are troubleshooting an auditing-related issue, you should enable detailed logging for the collector service on the computers where the collector service runs.

To enable detailed logging on a collector:

  1. Log on to a computer with a collector service.

  2. Click Start > All Programs > Centrify Server Suite 2021.1 > Centrify Audit & Monitoring Service > Audit Collector Control Panel to open the Collector Control Panel.

  3. Click the Troubleshooting tab.

  4. Click Options, change the logging level to Trace messages, then click Apply.

  5. Note the log folder location or click Browse to specify a different location for the log file, then click OK.

  6. Click View Log to view the current log file.

    From the log file window, you can also click File > Save As to save the log file.

  7. Click Close to close the Collector Control Panel.

  8. Send an email to Centrify Support with the log file from the location specified in Step 5 as an attachment.

  9. Open the Collector Control Panel, click the Troubleshooting tab, click Options, change the logging level back to its default setting ofInformational messages, then click OK.

Enabling Detailed Logging for Auditing Consoles

In most cases, troubleshooting auditing-related issues requires information about the operation of the agent and the collector or database activity. However, in some cases, it might be necessary to capture detailed information about the operation of Audit Manager or Audit Analyzer.

To capture detailed information for Audit Manager:

  1. Log on to a computer with the Audit Manager console.

  2. Click Start > All Programs > Centrify Server Suite 2021.1 > Audit Manager to

    open the Audit Manager console.

  3. Select the Audit Manager node, right-click, then click Log Settings.
  4. Change the logging level to Trace messages, then click Apply.

  5. Note the log folder location or click Browse to specify a different location for

    the log file, then click OK.

  6. Send an email to Centrify Support with the log file from the location specified in

    Enabling detailed logging for the collector service as an attachment.

  7. Right-click Audit Manager, click Log Settings, change the logging level back to its default

    setting of Warning messages, then click OK.

To capture detailed information for Audit Analyzer:

  1. Log on to a computer with the Audit Analyzer console.
  2. Click Start > All Programs > Start > All Programs > Centrify Server Suite 2021.1 > Audit Analyzer> to open the Audit Analyzer console.
  3. Select the Audit Analyzer node, right-click, then click Options.
  4. Change the logging level to Trace messages, then click Apply.

  5. Note the log folder location or click Browse to specify a different location for the

    log file, then click OK.

  6. Send an email to Centrify Support with the log file from the location specified in

    Enabling detailed logging for the collector service as an attachment.

  7. Right-click Audit Analyzer, click Options, change the logging level back to its

    default setting of Warning messages, then click OK.

Tracing Database Operations

Database traces are used to help diagnose problems in the management database or audit store databases. For example, database traces can help to identify inconsistencies caused by hardware errors or network interruptions. After you enable database tracing, Audit Manager tracks all of the SQL statements and debug messages from the audit management database or audit store, and records the information in the database server.

Tracing database operations affects database performance. You should only activate a database trace if you require this information for troubleshooting. Before you start a database trace, try to reduce the load on the database instance as much as possible, then only perform the actions needed to reproduce the issue you are troubleshooting. Turn off database tracing as soon as you have logged the activity you need for the analysis of database operations. The trace for each database can take up to 800MB of server disk space. After you turn off database tracing, restart the SQL Server instance to reset the disk space.

Starting a Database Trace

You can start a database trace for a management database or an audit store database.

To start database tracing:

  1. Open Audit Manager.

  2. Select an installation name, right-click, then click Properties.

  3. Click the Database Trace tab.

    This tab displays basic information about the management databases and audit store databases for the selected installation. In the Trace Status column, you can see whether tracing is enabled or disabled for each database.

  4. Select a management or audit store database in the list, then click Enable to start tracing on the database selected.

  5. Click OK, then perform the database actions for which you want to capture information.

Stopping the Database Trace

You should turn off database tracing immediately after you have logged the activity you need for the analysis of database operations.

To stop database tracing:

  1. Open Audit Manager.
  2. Select the installation name, right-click, then click Properties.
  3. Click the Database Trace tab.

  4. Select the management or audit store database that has tracing enabled, then

    click Disable to stop tracing on the database selected.

  5. Click Export to save the database trace from the selected databases to a file

    with comma-separated values (.csv).

  6. Follow the prompts displayed in the Export Database Trace wizard to save the information to a file.

Exporting the Database Trace for a Management Database

The Export Database Trace wizard prompts you for different information depending on whether the database trace is for a management database or an audit store database. For example, if you generate a database trace for a management database then click Export, the Export Database Trace wizard prompts you for user accounts.

To export the database trace:

  1. Select a start date and time for the From filter and an end date and time for the To filter, then click Next.

  2. Click Add to search for and select users, then click Next.

    By default, you can search for users in the entire directory, you can click Object Types or Locations to change the scope of the search scope, or click Advanced specify other criteria.

  3. Accept the default folder location or click Browse to select a different location, then click Next.

  4. Review your selections, then click Next.

    By default, the wizard save the file as installation_name.csv and opens the file location.

  5. Click Finish, then click OK to close the installation properties.

Exporting the database trace for audit store databases

When you select an audit store from the lower area of the Database Trace tab on the Properties page and click the lower Export button, the wizard opens with a date/time Export Criteria page. On the second page, the wizard asks you to pick the domain and computer.

To export the database trace:

  1. Select a start date and time for the From filter and an end date and time for the To filter, then click Next.

  2. Click Add to search for and select collectors, then click Next.

    By default, you can search for computers in the entire directory, you can click Object Types or Locations to change the scope of the search scope, or click Advanced specify other criteria.

  3. Click Add to search for and select management database computers, then click Next.

  4. Accept the default folder location or click Browse to select a different location, then click Next.

  5. Review your selections, then click Next.

    By default, the wizard save the file as audit_store_name.csv and opens the file location.

  6. Click Finish, then click OK to close the installation properties.

Delegating Database Trace Management

You can delegate the authority to manage database tracing by granting the Manage Database Trace permission to other users for a management database or an audit store database.

Stopping Auditing on a Computer

Several actions can directly or indirectly stop auditing on a computer. For example:

  • Someone powers down the audited computer.
  • Someone logs in on the audited computer and stops the agent.

  • The audited computer is moved to a different audit store, causing the initial audit store to

    consider the audited computer disconnected.

  • The administrator checks the Define trusted audited computer list on the Advanced tab of an

    Audit Store Properties page, and does not include the audited computer on that list.

Resuming Auditing if the Agent Stops

If the dad service stops running for any reason, audited shell sessions will stop working and you will be prompted to resume or quit auditing. If you resume auditing, the cdawatch process attempts to start dad and connect to the installation. However, if you have manually stopped the dad process, for example by running /usr/share/centrifydc/bin/centrifyda stop, you must manually restart the agent.

If you decide to quit auditing when the dad service has stopped running, you are prompted to confirm that you want to terminate the session before the session ends.

Allowing Users to Log in when Auditing is Stopped

If auditing is required but the agent is not running, users might be prevented from logging in. You can log in as a user with root privileges and either restart the agent or temporarily disable auditing using dacontrol -d to allow users to log in.

You can also run dainfo --diag or check the log file to get more information. For example, if the adclient process is not running, you might be unable to restart auditing.

If you cannot immediately correct the problem, you can temporarily disable all auditing.

Determining Collector Status and Connectivity

You can use the Collector Control Panel to generate a complete diagnostic check of the collector. The diagnostic report includes detailed information about the current status of the collector and the installation and audit store to which the collector sends data.

To generate diagnostics on a collector:

  1. Log on to a computer with a collector service.
  2. Click Start > All Programs > Centrify Server Suite 2021.1 > Centrify Audit & Monitoring Service > Audit Collector Control Panel to open the Collector Control Panel.
  3. Click the Troubleshooting tab.
  4. Click Diagnostics.

The results display in a Diagnostic Information window. If connections are successful and components are configured correctly, you should see results similar to this:

Establishing connection with Collector: Success  
 Getting collector's current status: Running  
 Getting Collector's current Installation: DefaultInstallation (locally configured)  
 Getting Collector's current Audit Store: Data Source=pysql.py.dev\\CENTRIFYSUITE;Initial Catalog=AuditStoreWindows-2018-11-02
 Machine IP address(es): 10.140.16.59  
 Machine is joined to: py.dev  
 Forest: py.dev  
 Using Domain Controller: pydc.py.dev  
 Is Global Catalog Available: True  
 Using Global Catalog: pydc.py.dev  
 Machine is in site: Default-First-Site-Name@py.dev  
 Installations:  
 DefaultInstallation  
 AD Object: py.dev/Program Data/Centrify/DirectAudit/Vegas-Installation-d97d8fc9-7876-4f5b-b161-4a7b3736b8ec
 Object GUID: 1563b2e1-1ea3-4307-ac51-7c92fdf5cb8a  
 Installation ID: f7b36b73-0384-4283-b6f7-63a1cdb77b17  
 Audit Stores:  
 AuditStoreUNIX  
 Site(s): (Default-First-Site-Name@py.dev)  
 Subnet(s): None configured  
 Affinity: UNIX  
 Trusted Agents: None configured  
 Trusted Collectors: None configured  
 Audit Store Active Database:  
 Data Source=pysql.py.dev\\CENTRIFYSUITE  
 Initial Catalog=AuditStoreUNIX-2018-11-02  
 Additional Connection Parameters=\<none\>  
 AuditStoreWindows  
 Site(s): (Default-First-Site-Name@py.dev)  
 Subnet(s): None configured  
 Affinity: Windows  
 Trusted Agents: None configured  
 Trusted Collectors: None configured  
 Audit Store Active Database:  
 Data Source=pysql.py.dev\\CENTRIFYSUITE  
 Initial Catalog=AuditStoreWindows-2018-11-02  
 Additional Connection Parameters=\<none\>  
 Machine's Installation: DefaultInstallation (locally configured)  
 This machine's Audit Store is 'AuditStoreWindows' based on preferred Audit Store (locally configured)
 Attempting to connect to Audit Store:  
 Data Source=pysql.py.dev\\CENTRIFYSUITE  
 Initial Catalog=AuditStoreWindows-2018-11-02  
 Integrated Security=TRUE  
 Pooling=True  
 Max Pool Size=1000  
 Encrypt=True  
 TrustServerCertificate=True  
 Additional Connection Parameters=\<none\>  
 Connected to Audit Store successfully  
   
 Done.

You can copy the results to a file and send them to Centrify Support for help.

Resolving Connectivity Issues between a Collector and an Audit Store

If the diagnostic report or the Collector Configuration wizard indicates that the collector cannot connect to an audit store database, check the following:

  • Verify the account you logged in with has permission to add a collector.

  • Verify the collector service has permission to connect to the active audit store database. You can grant this permission from Audit Manager.

  • Check whether the SQL Server instance needs to be restarted. For example, make sure the SQL Server instance is not waiting for a restart to complete ASP.NET registration changes.

  • Check whether there is a firewall between the collector and the SQL Server instance blocking access.

  • Check whether SQL Server is configured to allow named pipes and TCP/IP connections.

  • Check whether SQL Server is configured to allow remote connections.

  • Compare the site or subnet that the collector is configured to use with the scope of the audit store. For example, make sure the audit store site or subnet matches the site or subnet in the audit store properties.

    AuditStore
    Site(s): (Default-First-Site-Name@pistolas.org)
    Subnet(s): None configured

Resolving Authentication Issues

If you configure the collector service to use an Active Directory account instead of the local system account, you might encounter problems with Kerberos authentication when the collector attempts to connect to the audit store database. Kerberos authentication uses the service principal names (SPN) registered for the SQL Server account to authenticate a service. When the collector (client) wants to connect to SQL Server, it locates an instance of the service, composes an SPN for that instance, connects to the service, and presents the SPN for the service to authenticate. If the collector service account does not have any SPNs, the Kerberos authentication request fails.

To resolve this problem, go to KB-1311 in the Centrify Knowledge Base, select Attachments, and click View > Open > Run to run the checkspn.vbs script on a computer that is joined to Active Directory.

The user who is running this command must have permission to register the SPN on the service account.

By default, this script runs in report-only mode. It checks whether the required SPNs are present on the service account in question and issues a prompt to fix it, if not. This script registers the SPN in the service account servicePrincipalName attribute in the format:

MSSQLSvc/<FQDN>:<tcpport>

Monitoring Collector Performance Counters

If you have enabled auditing and installed the collector service on a local Windows computer, you can add audit-specific performance counters to Performance Monitor to help you analyze and resolve audit-related issues. When you install the collector, the performance counters are added automatically, if you uninstall the collector, the counters are also automatically removed from Performance Monitor.

To add Server Suite performance counters:

  1. Log on to a computer with a collector service.

  2. Click Start > Administrative Tools > Performance Monitor.

  3. Expand Monitoring Tools and select Performance Monitor.

  4. Click the green plus (+) icon in the toolbar.

  5. Find the Audit Collector from the list, and expand it to show the list of available performance counters.

    The performance counters generally fall into one of three categories; agent information, packet volume, and data loads. For example, if you add the counter # Connected Agent,you will be able to view the number of agents currently connected. If you add thecounter # Unix Meta Message Packet, you will be able to view the number of Unix metamessage packets. If you add the counter, Bytes Unix Command, you will be able to view Unix command data in bytes.

  6. Choose the performance counter you would like to add and click Add.

  7. Repeat Step 6 until you have added the counters you want to monitor.

  8. Click OK.

Managing Microsoft SQL Server Databases

Managing an audit installation requires permission to create new SQL Server databases on a SQL Server instance. In a production environment, this is an ongoing process to keep databases small and efficient. Because the management of the audit databases is not a onetime setup operation, Delinea recommends that you have at least one dedicated SQL Server instance for the audit administrator to use. The audit administrator should also be a member of the SQL Server system administrator role to ensure full control over the databases created and archived.

Selecting SQL Server or Windows Authentication

When you configure the Microsoft SQL Server instance to use for auditing, you must specify the type of authentication to use. The appropriate type of authentication depends on how your production environment is configured. For example, if you have a firewall between components or one-way trust relationship between forests, you must allow SQL Server authentications.

To support the auditing infrastructure, you can use the following types of authentication:

  • Windows authentication for creating new databases.

  • Windows authentication or both SQL Server authentication and Windows authentication for

    connections between collectors and audit stores.

  • Windows authentication or both SQL Server authentication and Windows authentication for

    connections between audit stores and the audit management database.

  • SQL Server authentication for collectors in an untrusted forest and an audit store in a

    trusted forest.

  • SQL Server authentication for audit store databases in a trusted forest and audit management

    database in an untrusted forest.

If you choose Windows authentication, you can perform actions with your own logon account or using another Windows account name and password.

Connecting to an Installation or Database

If you unable to connect to the SQL Server database, the problem might be caused by one of the following issues:

  • A firewall blocking access to the SQL Server instance.
  • TCP/IP has not been enabled for the SQL Server instance of SQL Server
  • Remote connections have not been enabled for the SQL Server instance.

For information about areas to check, see the following article:

https://blog.sqlauthority.com/2009/05/21/sql-server-fix-error-provider-named-pipes-provider-error-40-could-not-open-a-connection-to-sql-server-microsoft-sql-server-error/

Assigning the Service Principal name for SQL Server

If you get error messages when performing database operations, such as creating a new audit management database using Audit Manager, the problem is likely because the service principal name (SPN) for the SQL Server instance is assigned to the wrong Active Directory container.

  • If the SQL Server startup account is a local system account, the appropriate container is

    the computer name.

  • If it is any other account, the appropriate container is the SQL Server startup account.

Because authentication tries to use the first SPN it finds, make sure that no SPNs are assigned to inappropriate containers. Usually this error occurs when the administrator does not remove a manually added SPN from the Active Directory container after changing the SQL Server service account.

For help troubleshooting this problem, read the following article:

https://support.microsoft.com/en-us/help/811889/how-to-troubleshoot-the-cannot-generate-sspi-context-error-message

Publishing Installation Information in Active Directory

The default location for publishing audit installation information in Active Directory is:

domain/Program Data/Centrify/DirectAudit

In most cases, this location is accessible to any administrative user. If you cannot access the publication location, check the following:

  • Make sure you have permission to publish information to Active Directory.
  • Verify that the publication location exists in Active Directory.
  • Check the network for problems.

Moving a service connection point from its published location can result in connection problems. If you delete the default publication location and add a new publication location, you might not have permissions on the new location. If you do not have the appropriate permissions on the new location, ask the Active Directory administrator to grant you such permissions before running any of the wizards to reconfigure agents and collectors.

A new location might not be reflected immediately in the list current published locations. However, this has no any adverse effects apart from not being able to see the published location.

Monitoring File System Disk Space Usage

Like most software applications, Centrify Agents require adequate disk space to be available to operate properly. For example, agents read and write temporary files to authenticate processes and ensure data integrity. If your operating system does not have enough disk space to accommodate these temporary files, the agent might be unable to run and prevent users from logging on or activity from being audited.

To prevent problems with disk space allocation, you should monitor key directories, such as the /tmp and /var directories, to ensure free space is available. The disk space required by different directories depends on the configuration and operating systems of the computer and the Active Directory environment. However, if any directory approaches 100% of its allocation, you should allocate more disk or remove older files to free up space for continued operation.