Configure Data Retention

   Journey Manager (JM) The transaction engine for the platform.  |    System Manager / DevOps  |   22.04 This feature was updated in 22.04.

Manager comes with highly customizable Data Retention Management that allows you to implement various customer requirements pertaining to their data retention policies.

To configure data retention:

1. Select System > Data Retention Management.

   

Note

You must have the Service Edit permission to view the Data Retention Management tab.

2. Select the Retention Settings tab to configure transaction records and system logs retention settings.
3. Configure Transaction Records. Transaction records are created the moment a user opens a form. The status of the form changes as the user saves, cancels, and submits the form. Each transaction corresponds to the user entered data, and is associated with other details, such as submission history, submission properties, data extracts, and attachments. In relation to data retention, transactions incur two major concerns:
• Transactions are continually generated and contain a large amount of data. This can lead to large database sizes and degraded performance if data is not purged regularly.
• Transactions contain user-entered Personally Identifiable Information. This can pose a security risk. Manager comes with several data security measures. It is recommended to ensure that PII is purged as soon as transactions have finished.

Generally, you want to keep your purge settings strict. Using the strict mode will ensure that transaction data is purged regularly, maximizing your system’s performance and security. Manager supports this approach by a frequent job that checks for and purges any PII data and completed submissions that had been completed more than the maximum age time frame (according to your data retention management settings). The transactions eligible for deletion are completed transactions (successful submissions) and abandoned transactions. That is, the Manager must recognize that the transaction is in a finished state.

4. The Enforce System/Org Thresholds checkbox determines whether to constrain the settings that can be overridden on the organization or form level, so select it to prevent setting a more lenient override for a corresponding setting at the organization or form level. Furthermore, where there is a stricter corresponding setting at the organization level, you will be prevented from setting a more lenient corresponding override at the form level. For example, if the Enforce System/Org Thresholds checkbox is ticked and the global Finished Transaction PII Data policy is set to 14 days, you cannot enter values greater than 14 days for the Finished Transaction PII Data policy on the organization or form level. In addition, if an organization's value of the Finished Transaction PII Data policy is 10 days, you cannot configure any of this organization’s forms to have a Finished Transaction PII Data policy value greater than 10 days.
   
   Thresholds are only relevant to the Saved Transactions and Finished Transaction PII Data settings.
Note

Even if Enforce System/Org Thresholds is selected, changes to global or organization policies DO NOT affect the values set on lower levels. For example, if an organization has a Finished Transaction PII Data policy value of 14 days and you change the global Finished Transaction PII Data policy value to 7 days, the organization value will remain the same until the organization settings are edited. This avoids problems around accidental changes to global policies trickling through the entire system.

The following settings will initially be defaulted according to your environment’s data retention policy mode, strict or relaxed. The individual settings may be customized as explained.

5. The Saved Transactions setting allows you to specify the number of days after which a transaction that was explicitly saved by a user is automatically abandoned by the system. The number of days before abandonment will reset each time the form is saved by the user. This means that the allowable length of time between when the form is first opened and its eventual submission may be extended by the user resuming the form regularly to complete and save the entered data.

Once a saved transaction is abandoned, it is considered a finished transaction like when a user completes and successfully submits a form. Note that the transaction will not be deleted at this point, but as a finished transaction, the countdown for PII data deletion and submission record deletion starts ticking. This setting can be customized on both the organization and form levels.

Warning

Be aware that 24-hour automatic abandonment of transactions can happen even they are rendered but never actually captured any data, or transactions that are never user-saved, even though they are background-saved, for example, on page navigation.

6. The Finished Transaction PII Data setting determines how many days personally identifiable submission data (XML data, attachment data, submission extract data and submission properties) will be kept once a transaction has finished (completed or abandoned). This data takes up a large amount of space and poses some security risk due to containing user entered data. The transaction record itself will stay around after data deletion (see the next setting below). This setting can be customized on both the organization and form levels.
7. The Finished Transactions setting determines how many days transaction records and any associated details will be kept once a transaction has finished. If PII data has been deleted for a submission, the header records remain until this policy kicks in, allowing administrators to find the submission in the logs.
8. Edit the number of days in the Finished Collaboration Jobs field to specify how long collaboration jobs and their submissions are kept. The default is 90 days.
9. Edit the number of days in the Collaboration Jobs Abandonment field to specify how long to keep collaboration jobs before automatic abandonment. The Transaction Processor service abandons a collaboration job when either its Last Processed Time or Creation Time parameter expires, that is, when either time is greater than the above specified period of time. The default is 60 days.
   Note

   We recommend understanding the abandonment process for collaboration jobs as well as transactions and tasks before configuring this property.

    |  22.04 This feature was introduced in 22.04.

   Note

   You can disable collaboration job abandonment by setting the Max Collaboration Job Abandon Age Days to 0 in the Data Retention Management Service.

   Note

   You can override this value on the organization level.

10. The Transaction History setting determines how long transaction history records are kept, counting from the time the submission is started. The default is 180 days.
11. Select how long an email queue item is kept from the Email Queue dropdown list. Time is counted from the moment the email is created.  |  18.05 This feature was introduced in 18.05.
12. Error Log setting determines how long error log entries and data that are not associated with a submission will be kept, counting from the time that the error occurred.
13. Event Log setting determines how long event log entries that are not associated with a submission will be kept, counting from the time that the event was logged.
14. Groovy Service Log setting determines how long log entries for Groovy service invocations will be kept, counting from invocation time.
15. Security Log Audit setting determines how long audit log entries are kept, counting from the time the log was created. The audit log can be crucial to tracking down changes made by administrators, so on non-test servers, we recommend that you allow adequate time before purging this log.
16. Schedule Job History setting determines how long Manager keeps scheduled job history entries, which are created every time a scheduled job runs.
17. User Login History setting determines how long events relating to user session, such as login, logout, session expiry, and account lock etc., are kept.
18. Click Save to update the changes.
19. Click Apply Policy to apply the retention policies right away and purge records exceeding the maximum age.

Next, learn how to configure organization data retention.