Data Retention Management Overview

   Journey Manager (JM) The transaction engine for the platform.  |    System Manager / DevOps  |  All versions This feature is related to all versions.

Manager has a high volume of user transactions associated with forms and their resources, such as data records and log files. This amounts to significant data even during a rather short period of operation. This can have a detrimental effect on the overall system performance, which, in turn, can negatively impact user experience and various third party integrations. To maintain the system optimal performance over time, Manager provides data storage, retention and purging capabilities that you can adjust according to your company’s data policies. This is instrumental in achieving the following:

Control the growth of database tables

In Manager, the Transaction table is a heavy weight table used to store a lot of information about in progress transactions and is not designed for long term storage. When a transaction is completed an entry is created in the Transaction History table which is designed to provide a long term audit trail. Purging data and records quickly from the Transaction table will help maintain high system performance.

Reduced Exposure of PII

Purge transaction form data which contains sensitive information

Form transactions can include sensitive business or personal data, such as a form user's Personally Identifiable InformationPersonally identifiable information (PII) is any data that could potentially identify a specific individual. Any information that can be used to distinguish one person from another and can be used for de-anonymizing anonymous data can be considered PII. (PII), depending on how transactions have been designed. Purging this data after delivery provides a stronger security story.

Data retention management is an automated process - what you need to do is to adjust the default data retention configuration and periodically review it to make sure the data is purged as per your SLAsA service-level agreement (SLA) is a commitment between a service provider and a client. Particular aspects of the service – quality, availability, responsibilities – are agreed between the service provider and the service user..

Manager purges the following data:

• Transaction data: the forms XML, Data Extracts, Attachments and Receipts. This will include the majority of the users personal information.
• Transaction records: the actual record from the Transaction table. The main reason to keep this record in place for short period of time is for troubleshooting.
• Transaction history records: the audit log of transaction activities. For systems with high throughput there is a argument for archiving these records for long term storage and purging these records from Manager.
• System logs: numerous areas of system logs whose purge time can the controlled separately for troubleshooting purposes.

Manager collects data retention information for each transaction including whether any PII data has been purged or is due to be purged and details about the data retention stage reached in the life cycle of this transaction.

Manager allows you to configure data retention on the global (system), organization and form levels that gives you flexibility in overriding important settings for individual organizations. Let's look at this in more detail.

Global Level

You can define data retention on the global (system) level, which is applicable to an entire server or environment, including all server nodes. This configuration is initially set to default values that depend on your environment’s data retention policy mode: strict or relaxed.

In addition to data retention, you can also configure submission data storage. This means that you can choose how and where submission-related data, such as submission data, attachments and submission history data for saved forms, is stored and retrieved by Manager.

By default, submission-related data is stored in a Manager database. However, it can be configured and changed at runtime as well. The functionality is encapsulated in a set of service definitions of type Submission Data Storage. The storage service that is marked as the default for this service type is used when storing submission data.

Organization Level

By default, Manager applies global data retention configuration defined for the entire server to its organizations, unless an organization has its own data retention configured. However, you can customize retention policies for all transactions associated with forms, for example, saved and finished transaction's PIIPersonally Identifiable Information (PII) is information about an individual that can be used to distinguish or trace an individual‘s identity, such as name, social security number, date and place of birth, mother‘s maiden name, or biometric records; and any other information that is linked to an individual. data.

The data retention policies you can configure are the maximum age of form submissions in general as well as delivered (including undeliverable) and saved (including abandoned) submission data.

You can also set the data storage encoding for the organization if it is different to the system-wide setting. We recommend that you use the Compressed / Encrypted option for maximum security and minimal storage space requirements. For more information, see File System Submission Data Storage Service.

Finally, for additional security you can set a rollover interval for the security key for the organization. The security key is used by Manager to encrypt submission-related data. All encrypted data is associated with its security key, so data encrypted before a security key change will still be accessible. If you choose a value other than Never, a new security key will be automatically generated by a background job when required.

Form Level

Manager applies global data retention policies, defined for the entire server or individual organizations, to its forms. However, you can customize transaction retention policies on the form level, for example, saved and finished transaction's PIIPersonally Identifiable Information (PII) is information about an individual that can be used to distinguish or trace an individual‘s identity, such as name, social security number, date and place of birth, mother‘s maiden name, or biometric records; and any other information that is linked to an individual. data. These settings then apply to all submissions made for this form.

Manager has two modes of data retention policy, strict and relaxed, that determines the length of time transactional data is retained. The selected policy can be customized at the environment organization, or form levels.

Manager uses various scheduled jobs that automatically purge transactions that have reached their maximum age.

The diagram below summarizes all this functionality.

You can check the data retention summary, which is useful for monitoring the overall performance of the data retention management and assists in diagnosing and tuning of any data retention management issues.

Next, learn about data retention policies.