Journey Manager (JM) The transaction engine for the platform. | System Manager / DevOps | 21.05 This feature was updated in 21.05.
Splunk is popular software for searching, monitoring, and analyzing machine-generated data, such as log files, which you can use it to have insights into errors and perform proactive monitoring and alerts for your applications.
Although Manager doesn't provide a direct integration with Splunk, you can configure a System Event Publisher with an AWS SQS adapter to push log records to an AWS queue. Then, you integrate Splunk with the AWS SQS, so it pulls Manager log records from there.
The Splunk integration is summarized below:
It is important to understand that you will deal with messages as events, not raw exports of log data, so you need to distinguish what an event and log record mean in this context.
Downside of such integration is that an event may contain redundant data, such as link to a transaction, within a set of messages (events) for the same transaction, which can dramatically increase amount of raw data to be stored on a client's side. The logic importing this data to Splunk should consider that.
It means that, for instance, a submission status change event will have event logs and error logs linked to the submission. If the submission status changes multiple times (and it often does), all event messages to the AWS SQS will contain all submission's event and error log records up to that milestone. In short, you will be getting duplicates of the early recorded log records.
This solution creates significant amount of data in Splunk, as compared to Manager logs.
If you want to analyze the Groovy service log records in Splunk, you need to know that the Manager's GroovyLogger class internally has a SLF/Log4J logger which writes messages to a service.log file. The default logging level is ERROR, which means DEBUG, INFO and WARN messages are not logged to disk. The reason for this default setting is to prevent PIIPersonally Identifiable Information (PII) is information about an individual that can be used to distinguish or trace an individual‘s identity, such as name, social security number, date and place of birth, mother‘s maiden name, or biometric records; and any other information that is linked to an individual.
In Europe, PII is known as personal data. data being logged to disk.
If this is configured to INFO or DEBUG level, then GroovyLogger statements at those levels will be logged to the server.log file as well. From here they can be feed into Splunk or other log management systems, regardless of the Groovy Service Logging configurations for the database.
The System Event Publisher can feed Groovy Service Log Data message to the AWS SQS endpoint to support integration with other log management systems without having to write PII data to disk. The Groovy Service Log Data messages will include the same content as would be included in the Groovy Service Log Data records, so it cannot be used as a complete substitute to the Groovy Service Log records.
Next, learn about the system logs.