ArcSight CEF format

The Common Event Format (CEF) standard format, developed by ArcSight, enables vendors and their customers to quickly integrate their product information into ArcSight ESM.

CEF defines a syntax for log records comprised of a standard header and a variable extension, formatted as key-value pairs.

When syslog is used as a transport mechanism, CEF uses the following format, comprised of a syslog prefix, a header, and an extension:

Jan 18 11:07:53 host CEF:Version|Device Vendor|Device Product|Device
Version|Device Event Class
ID|Name|Severity|[Extension]

The following example illustrates a general CEF message using syslog transport:

Sep 19 08:26:10 host
CEF:0|Centrify|Centrify_Cloud|1.0|Cloud.core|Cloud.core.MfaSummary|5|src=10.0.0.1 dst=2.1.2.2 spt=1232

Using CEF without wyslog

Syslog applies a syslog prefix to each message, no matter what device it arrives from, which contains the date and hostname:

Jan 18 11:07:53 host CEF:Version|…

However, if an event producer is unable to write syslog messages, it is still possible to write the events to a file. In this case, begin the message with the format shown below, and omit the syslog prefix:

CEF:Version|Device Vendor|Device Product|Device Version|Device Event Class ID|Name|Severity|[Extension]

Sample Python functions for CEF creation

This section describes a set of sample Python functions for generating CEF-formatted CP events.

There are three main functions in this package:

  • fetch_oauth_token()

  • query_events()

  • cef_generator()

Using the functions to demonstrate sample usage

Prerequisite: Python 3.5 or above

Follow these steps:

  1. Download the Python code from https://github.com/centrify/centrify-hparcsight-integration-sample/

  2. Install pip packages in requirement.txt.

  3. Provide the values for tenant, siem_username, and siem_password in config.ini.

  4. Execute sample_usage.py to generate CEF-formatted CP events for one hour:

    python3.5 sample_usage.py

The following example shows a CEF message for a Self-Service App Launch CIS Event:

Copy
CEF:0|Centrify|Centrify_Cloud|1.0|Cloud.Saas.Application|Cloud.Saas.Application.SelfServiceAppLaunch|5|dhost=AAA0056 duser=cloudadmin@persistent.com01 msg=User cloudadmin@persistent.com01 launched Instagram from 103.6.32.100 shost=103.6.32.100 src=103.6.32.100 rt=1525844566655 deviceProcessName=centrify-syslog-writer dvchost=dinesh-VirtualBox dtz=Africa/Abidjan requestContext=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36 Edge/15.15063 externalId=772a4a904e82da87.W00.0315.1aa20afe647f09c dpriv=WebRole destinationServiceName=CDS suid=c2c7bcc6-9560-44e0-8dff-5be221cd37ee cs1=Instagram cs1Label=applicationId cs2=Instagram cs2Label=applicationName cs3=Web cs3Label=applicationType cs4=103.6.32.100 cs4Label=clientIPAddress cs5=65f79bb1-4f91-4496-9991-d148da16cc3e cs5Label=internalSessionId cs6=0d10a24f4c57434198fb3ad4559cc48b cs6Label=azDeploymentId directoryServiceNameLocalized=Centrify Directory threadType=RestCall azRoleId=WebRole_IN_0 internalTrackingID=d3a0713b610146ca916155efca2be690 authMethod=UserPassword requestIsMobileDevice=False directoryServiceUuid=09B9A9B0-6CE8-465F-AB03-65766D33B05E requestDeviceOS=Windows level=Info

You can customize the usage or the APIs per your application needs.

Note:   CEF has a predefined set of keys.

CEF mapping of CP events

This section provides detailed information about how the CEF fields have been mapped from the CP event fields in the Python application described above.

CEF header

Header Field CP Event Field

Version

‘0’

Device Vendor

‘Centrify’

Device Product

‘Centrify_Cloud’

Device Version

‘1.0’

Device Event Class ID

Variable — depends on the event.
For example: ‘Cloud.Saas.Application’

Name

Variable — depends on the event.
For example: Cloud.Saas.Application.SelfServiceAppLaunch’

Severity

Variable — depends on the Level field in event.
For example: ‘5’ for Info, ‘10’ for Error.

CP ArcSight CEF extension

The CEF Extension contains a collection of key-value pairs. The keys are predefined and are referred to as the ArcSight Extension Dictionary. (CEF Fields)

Common properties in CP events

This section lists the CEF field mapping of CP events, which are part of the CEF extension.

These properties are common to all events of the Centrify Platform and Privilege Services.

 

ArcSight CEF Field CP Event Field

The common properties are those listed below in bold.

Destination Host Name

Tenant

Destination User Name

NormalizedUser

Message

EventMessage

Source Host Name

RequestHostName

Source Address

FromIPAddress

Device Receipt Time

whenoccurred_epoch_ms

(This is the event timestamp in UTC)

Device Process Name

‘centrify-syslog-writer’ (can be configured in cef_mapping.ini)

Device Host Name

Hostname of machine running the python app

Device Time Zone

‘Africa/Abidjan’

Note:   (This time zone is chosen mainly to set UTC offset to 0)

The keys in the common properties section below are added in the CEF message only if no event-specific CEF mapping is specified for an event in the mapping configuration file, which is enclosed with the Sample Python application for CEF creation.

Device Custom String 1

AuthMethod

Device Custom String1 Label

'authMethod'

Device Custom String2

RequestIsMobileDevice

Device Custom String2 Label

'requestIsMobileDevice'

Device Custom String3

DirectoryServiceUuid

Device Custom String3 Label

'directoryServiceUuid'

Device Custom String4

RequestDeviceOS

Device Custom String4 Label

'requestDeviceOS'

Device Custom String5

Level

Device Custom String5 Label

Level

Event-specific properties in CP

This section lists the event-specific properties mapped to ArcSight Fields. All events (whether they are listed below or not) will have the first nine common properties, identified in the table above, mapped in an ArcSight.CEF message. The message is generated when you use the Sample Python functions described earlier in this document.

Any CEF key appearing in event-specific mapping will override the CEF key mapping in the common properties section. For example, the Cloud.Server.ManualAccount.SessionStart event, Destination host (Dhost), and Destination User(duser) will be ‘ComputerName’ and ‘AccountName’, which will overwrite the common properties mapped for dhost and duser.

EventType=Cloud.Core.MfaSummary

ArcSight CEF Field CP Event Field

Reason

MfaReason

Outcome

MfaResult

RequestContext

RequestUserAgent

ExternalId

ID

Dpriv

AzRoleName

DestinationServiceName

DirectoryServiceName

Device Custom String 1

MfaInitiator

Device Custom String1 Label

‘mfaInitiator’

Device Custom String2

FactorsLocalized

Device Custom String2 Label

‘factorsLocalized’

Device Custom String3

ProfileName

Device Custom String3 Label

‘profileName’

Device Custom String4

FailReason

Device Custom String4 Label

‘failReason’

Device Custom String5

MfaUnlock

Device Custom String5 Label

‘mfaUnlock’

Device Custom String6

ForgotPassword

Device Custom String6 Label

‘forgotPassword’

Device Custom Number1

Factorcount

Device Custom Number1 Label

‘factorCount’

Device Custom Number2

SecurityQuestionAnswerCount

Device Custom Number2 Label

‘securityQuestionAnswercount’

Note:   The remaining fields in an event that are not mapped to CEF keys will still be added in the CEF message with their CP-event field keys. These custom non-CEF keys will not be available for reporting in ArcSight, but they can viewed as part of the raw event message.

EventType=Cloud.Saas.Application.AppLaunch

ArcSight CEF Field CP Event Field

RequestContext

RequestUserAgent

ExternalId

ID

Dpriv

AzRoleName

DestinationServiceName

DirectoryServiceName

Suid

UserGuid

Device Custom String 1

ApplicationID

Device Custom String1 Label

‘applicationId’

Device Custom String2

ApplicationName

Device Custom String2 Label

‘applicationName’

Device Custom String3

ApplicationType

Device Custom String3 Label

‘applicationType’

Device Custom String4

TemplateName

Device Custom String4 Label

‘templateName’

Device Custom String5

InternalSessionId

Device Custom String5 Label

‘internalSessionId’

Device Custom String6

AzDeploymentId

Device Custom String6 Label

azDeploymentId

EventType=Cloud.Saas.Application.GatewayAppLaunch

EventType=Cloud.Saas.Application.SelfServiceAppLaunch

 

ArcSight CEF Field CP Event Field

RequestContext

RequestUserAgent

ExternalId

ID

Dpriv

AzRoleName

DestinationServiceName

DirectoryServiceName

Suid

UserGuid

Device Custom String 1

ApplicationID

Device Custom String1 Label

‘applicationId’

Device Custom String2

ApplicationName

Device Custom String2 Label

‘applicationName’

Device Custom String3

ApplicationType

Device Custom String3 Label

‘applicationType’

Device Custom String4

ClientIPAddress

Device Custom String4 Label

‘clientIPAddress’

Device Custom String5

InternalSessionId

Device Custom String5 Label

‘internalSessionId’

Device Custom String6

AzDeploymentId

Device Custom String6 Label

azDeploymentId

 

EventType=Cloud.Server.ManualAccount.SessionStart

EventType= Cloud.Server.LocalAccount.SessionStart

 

ArcSight CEF Field CP Event Field

Src

FromIPAddress

Suser

NormalizedUser

Dhost

ComputerName

Duser

AccountName

RequestContext

RequestUserAgent

ExternalId

ID

Dpriv

AzRoleName

DestinationServiceName

DirectoryServiceName

Suid

UserGuid

Device Custom String 1

UserType

Device Custom String1 Label

‘userType’

Device Custom String2

SessionType

Device Custom String2 Label

‘sessionType’

Device Custom String3

AuthorityName

Device Custom String3 Label

‘authorityName’

Device Custom String4

JumpType

Device Custom String4 Label

‘jumpType’

Device Custom String5

DirectoryServiceNameLocalized

Device Custom String5 Label

‘directoryServiceNameLocalized’

Device Custom String6

AuthoritySource

Device Custom String6 Label

‘authoritySource’

 

EventType=Cloud.Server.LocalAccount.PasswordExport

EventType= Cloud.Server.DomainAccount.PasswordExport

 

ArcSight CEF Field CP Event Field

Src

FromIPAddress

Suser

NormalizedUser

Dhost

ComputerName

Duser

AccountName

RequestContext

RequestUserAgent

ExternalId

ID

Dpriv

AzRoleName

DestinationServiceName

DirectoryServiceName

Suid

UserGuid

Device Custom String 1

UserType

Device Custom String1 Label

‘userType’

Device Custom String2

AuthorityID

Device Custom String2 Label

‘authorityID’

Device Custom String3

AuthorityName

Device Custom String3 Label

‘authorityName’

Device Custom String4

AzRoleId

Device Custom String4 Label

‘azRoleId’

Device Custom String5

DirectoryServiceNameLocalized

Device Custom String5 Label

‘directoryServiceNameLocalized’

Device Custom String6

CheckedOut

Device Custom String6 Label

‘checkedOut’

Device Custom Date1

WhenDueBack

Device Custom Date1 Label

‘whenDueBack’

 

EventType=Cloud.Core.Server.CpsTileLaunch

ArcSight CEF Field CP Event Field

RequestContext

RequestUserAgent

ExternalId

ID

Dpriv

AzRoleName

DestinationServiceName

DirectoryServiceName

Suid

UserGuid

Device Custom String 1

UserType

Device Custom String1 Label

‘userType’

Device Custom String2

ApplicationType

Device Custom String2Label

‘applicationType’

Device Custom String3

ApplicationName

Device Custom String3Label

‘applicationName’

Device Custom String4

ApplicationID

Device Custom String4Label

‘applicationId’

Device Custom String5

DirectoryServiceNameLocalized

Device Custom String5Label

‘directoryServiceNameLocalized’

Device Custom String6

InternalTrackingID

Device Custom String6Label

‘internalTrackingID’

EventType=Cloud.Core.AdaptiveMfa.RiskAnalysis

Only Common properties.

Alternate approach for creating the Common Extension Format (CEF)

In case you are using the CP REST APIs directly in your application and generating your own (Undefined variable: centrify-cloud-vars.platform)syslog messages in a generic non-CEF format having key=value pairs separated by a delimiter, then ArcSight SmartConnector will need to be installed and configured to collect these (Undefined variable: centrify-cloud-vars.platform) syslog.

These logs will need to be parsed into CEF format by creating ArcSight FlexConnector, to enable the (Undefined variable: centrify-cloud-vars.platform) events to be usable for SIEM in ArcSight. The only downside to using a FlexConnector is that ArcSight does not officially certify it.