Data Retention Management Overview

   Journey Manager (JM) The transaction engine for the platform.  |    System Manager / DevOps  |  All versions This feature is related to all versions.

Manager has a high volume of user form transactions and associated resources, such as data records and log files, that amounts to significant data even during a rather short period of operation. This can have a detrimental effect on the overall system performance, which, in turn, can negatively impact user experience and various third party integrations. To maintain the system optimal performance over time, Manager provides data storage, retention and purging capabilities that you can adjust to your company’s data requirements. This is instrumental in achieving the following:

• Control the growth of database tables
  In Manager, the Transaction table is a heavy weight table used to store a lot of information about in progress transactions and is not designed for long term storage. When a transaction is completed an entry is created in the Transaction History table which is designed to provide a long term audit trail. Purging data and records quickly from the Transaction table will help maintain performance.
• Maintain high system performance
• Purge transaction form data which contains sensitive information
  Form transactions can include sensitive business or personal data, such as a form user's Personally Identifiable InformationPersonally identifiable information (PII) is any data that could potentially identify a specific individual. Any information that can be used to distinguish one person from another and can be used for de-anonymizing anonymous data can be considered PII. (PII), depending on how transactions have been designed. Purging this data after delivery provides a stronger security story.

Data retention management is an automated process - what you need to do is to adjust the default data retention configuration and periodically review it to make sure the data is purged as per your SLAsA service-level agreement (SLA) is a commitment between a service provider and a client. Particular aspects of the service – quality, availability, responsibilities – are agreed between the service provider and the service user..

Manager purges the following data:

• Transaction Data - the forms XML, Data Extracts, Attachments and Receipts. This will include the majority of the users personal information.
• Transaction Records - the actual record from the Transaction table. The main reason to keep this record in place for short period of time is for troubleshooting.
• Transaction History Records - the audit log of transaction activities. For systems with high throughput there is a argument for archiving these records for long term storage and purging these records from Manager.
• System Logs - numerous areas of system logs whose purge time can the controlled separately for troubleshooting purposes.

Manager collects data retention information for each transaction including whether any PII data has been purged or is due to be purged and details about the data retention stage reached in the life cycle of this transaction.

You can set the data retention and storage settings globally and override important settings for individual organizations.

Data Retention Configuration - Global Level

In addition to data retention, you can also configure submission data storage. This means that you can choose how and where submission-related data, such as submission data, attachments and submission history data for saved forms, is stored and retrieved by Manager.

By default, submission-related data is stored in a Manager database. However, it can be configured and changed at runtime as well. The functionality is encapsulated in a set of service definitions of type Submission Data Storage. The storage service that is marked as the default for this service type is used when storing submission data.

Data Retention Configuration - Organization Level

You can override important form-related settings for each organization individually.

The data retention policies you can configure are the maximum age of form submissions in general as well as delivered (including undeliverable) and saved (including abandoned) submission data. If you would like the global settings to apply for this organization, select "Use system default".

You can also set the data storage encoding for the organization if it is different to the system-wide setting. We recommend that you use the Compressed / Encrypted option for maximum security and minimal storage space requirements.

Finally, for additional security you can set a rollover interval for the security key for the organization. The security key is used by Manager to encrypt submission-related data. All encrypted data is associated with its security key, so data encrypted before a security key change will still be accessible. If you choose a value other than "Never", a new security key will be automatically generated by a background job when required.

Manager has two modes of data retention policy, strict and relaxed, that determines the length of time transactional data is retained. The selected policy can be customized at the environment organization, or form levels.

Manager uses various scheduled jobs that automatically purge transactions that have reached their maximum age.

You can check the data retention summary, which is useful for monitoring the overall performance of the data retention management and assists in diagnosing and tuning of any data retention management issues.

Next, learn about data retention policies.