Global Data Retention Configuration

   Journey Manager (JM) The transaction engine for the platform.  |    System Manager / DevOps  |  24.10This feature was updated in 24.10

Manager comes with highly customizable Data Retention Management that allows you to implement various customer requirements relating to their data retention policies. You can define data retention on the global (system) level, which is applicable to an entire server or environment, including all server nodes. This configuration is initially set to default values that depend on your environment’s data retention policy mode: strict or relaxed. Then, you can refine data retention configuration per an organization and, further, per form.

Global data retention configuration properties are grouped into the following sections:

Transaction Records

Transaction records are created the moment a user opens a form. The status of the form changes as the user saves, cancels, and submits the form. Each transaction corresponds to the user entered data, and is associated with other details, such as submission history, submission properties, data extracts, and attachments. In relation to data retention, transactions incur two major concerns:

• Transactions are continually generated and contain a large amount of data. This can lead to large database sizes and degraded performance if data is not purged regularly.
• Transactions contain user-entered Personally Identifiable Information. This can pose a security risk. Manager comes with several data security measures and we recommend purging PII data as soon as transactions have finished.

Generally, you want to keep your purge settings strict, which ensures that transaction data is purged regularly, maximizing your system’s performance and security. Manager supports this approach by a frequent job that checks for and purges any PII data and completed submissions that had been completed more than the maximum age time frame. The transactions eligible for deletion are transactions in a finished state, such as completed transactions (successful submissions) and abandoned transactions. For more information, see the Transaction Records sections.

System Logs

This section allows you to define how long to keep various logs that Manager collects as per transaction life cycle.

Data Retention Configs

This section allows you to configure additional data retention settings.

To configure global data retention:

1. Select System > Data Retention Management and click the Retention Settings tab.

   

   Note

   You must have the Service Edit permission to view this configuration.

Transaction Records (max age)

2. Select a size of a commit batch, which is used for transaction purging, from the Commit Batch dropdown list. The value must be within the fetch limit specified below. The default is 1000.
3. Select the Enforce System/Org Thresholds checkbox to ensure that data retention policies on lover levels (organizations and forms) can't exceed policies on higher levels (system or organizations). This prevents you from configuring a more lenient policies at the organization and form levels.

   Furthermore, where there is a stricter corresponding setting at the organization level, you will be prevented from setting a more lenient corresponding override at the form level.

   For example, if the global Finished Transaction PII Data policy is set to 14 days, you can't enter values greater than 14 days for the Finished Transaction PII Data policy for the organization or form level. In addition, if an organization's value of the Finished Transaction PII Data policy is 10 days, you can't configure any of this organization’s forms to have the same policy be greater than 10 days.

   Note

   This setting is only relevant to the Saved Transactions and Finished Transaction PII Data policies, as only they can be overridden on all 3 levels.

   Note

   Even if Enforce System/Org Thresholds is selected, changes to global or organization policies don't affect the values already set on lower levels. For example, if an organization has the Finished Transaction PII Data policy set as 14 days and you change the global Finished Transaction PII Data policy to 7 days, the organization's value will remain unchanged until you start editing it. This logic avoids problems around accidental changes of global policies that would start trickling through the entire system.

4. Edit the maximum age of user saved transactions, after which they will be automatically abandoned, in the Saved Transactions field. The enforced range of this value is:
   • Min: 1 day
   • Recommended: 30 days
   • Max (Strict mode): 180 days
   • Max (Relaxed mode): 730 days
   Note

   Manager enforces the maximum limits of data retentions to provide required system stability.

   The abandonment date is reset each time the form is saved by the user. This means that the allowable length of time between when the form is first opened and its eventual submission may be extended by the user resuming the form regularly to complete and save the entered data.

   Once a saved transaction is abandoned, it is considered a finished transaction like when a user completes and successfully submits a form. Note that the transaction will not be deleted at this point, but as a finished transaction, the countdown for PII data deletion and submission record deletion starts ticking.

   Note

   You can override this setting on both the organization and form levels.

   Warning

   Be aware that 24-hour automatic abandonment of transactions can happen even they are rendered but never actually captured any data, or transactions that are never user-saved, even though they are background-saved, for example, on page navigation.

5. Edit the amount of time to keep PII transaction data for after the transaction is finished (Completed or Abandoned) in the Finished Transaction PII Data field. The enforced range of this value is:
   • Min: 1 day
   • Recommended: 30 days
   • Max (Strict mode): 30 days
   • Max (Relaxed mode): 180 days

   Transaction PII data consists of user entered information, such as XML data, attachment data, PDF receipts, submission data extracts and submission properties, which takes up a large amount of space and poses some security risk due to containing user entered data. The transaction record itself will stay after data deletion for a period of time defined with Finished Transactions.

   Note

   You can override this setting on both the organization and form levels.

6. Edit a number of days transaction records and any associated details are kept once a transaction has finished in the Finished Transactions field. If PII data has been deleted for a submission, the header records remain in the system until this policy kicks in, allowing administrators to find the submission in the logs.
7. Edit the number of days in the Finished Collaboration Jobs field to specify how long collaboration jobs and their submissions are kept. The default is 90 days.
8. Edit the number of days in the Collaboration Jobs Abandonment field to specify how long to keep collaboration jobs before automatic abandonment. The enforced range of this value is:
   • Min: 1 day
   • Recommended: 30 days
   • Max (Strict mode): 180 days
   • Max (Relaxed mode): 180 days

   The Transaction Processor service abandons a collaboration job when either its Last Processed Time or Creation Time parameter expires, that is, when either time is greater than the above specified period of time. The default is 60 days.

   Note

   We recommend understanding the abandonment process for collaboration jobs as well as transactions and tasks before configuring this property.

    |  22.04 This feature was introduced in 22.04.

   Note

   You can disable collaboration job abandonment by setting the Max Collaboration Job Abandon Age Days to 0 in the Data Retention Management Service.

   Note

   You can override this value on the organization level.

9. Edit the number of days to keep PII collaboration job data for after the collaboration job has been finished (Completed, Canceled, or Expired) in the Finished Collaboration Jobs PII Data field. The collaboration job PII data consists of job properties. The enforced range of this value is:
   • Min: 1 day
   • Recommended: 30 days
   • Max (Strict mode): 30 days
   • Max (Relaxed mode): 730 days

System Logs (max age)

10. Select a number of days to keep transaction history records, counting from the time the submission is started, from the Transaction History dropdown list. The default is 180 days.
11. Select how long an email queue item is kept from the Email Queue dropdown list. Time is counted from the moment the email is created.  |  18.05 This feature was introduced in 18.05.
12. Select how long error log entries and data that are not associated with a submission are kept, counting from the time that the error occurred, from the Error Log dropdown list.
13. Select how long event log entries that are not associated with a submission are kept, counting from the time that the event was logged, from the Event Log dropdown list.
14. Select the amount of time in days when billing records become eligible for purging from the Billing Records dropdown list. Default is 60 days.  |  24.10 This feature was introduced in 24.10
15. Select the amount of time in days when Event Inbox records (both Delivered and Failed) become eligible for purging from the Event Inbox dropdown list. Default is 3 days.  |  24.10 This feature was introduced in 24.10
16. Select the amount of time in days when Event Outbox records (both Delivered and Failed) become eligible for purging from the Event Outbox dropdown list. Default is 3 days.  |  24.10 This feature was introduced in 24.10
17. Select how long log entries for Groovy service invocations are kept, counting from invocation time, from the Groovy Service Log dropdown list.
18. Select how long scheduled job history entries, which are created every time a scheduled job runs, are kept from the Schedule Job History dropdown list.
19. Select how long audit log entries are kept, counting from the time the log was created, from the Security Audit Log dropdown list. The audit log can be crucial to tracking down changes made by administrators, so on non-test servers, we recommend allowing adequate time before purging this log. The default is 1 year.
20. Select the maximum age to keep Security Manager log records for from the Security Manager Log dropdown list. The default is 30 days.
21. Select the amount of time to keep T.Field App device log, sync log and sync call records for from the T.Field App Logs dropdown list. The default is 30 days.
22. Select how long events relating to user sessions, such as login, logout, session expiry, and account lock, are kept, from the User Login History dropdown list. The default is 180 days.

Data Retention Configs

23. Select the required purge mode from the PII Search Purge Mode dropdown list to specify when PII searching metadata, such as data extracts and transactions properties, will be deleted:
    • Form Completed: the smallest set of data to be deleted.
    • Delivery Completed: a bigger set of data to be deleted.
    • PII Purge Time: the biggest set of data to be deleted and it takes the longest time to do it.
    The default is Delivery Completed.
24. Select the Purge Submission Error Log With PII checkbox to delete all PIIPersonally Identifiable Information (PII) is information about an individual that can be used to distinguish or trace an individual‘s identity, such as name, social security number, date and place of birth, mother‘s maiden name, or biometric records; and any other information that is linked to an individual. In Europe, PII is known as personal data. data of a submission from the error log.
25. Click Save to update the changes.
26. Click Apply Policy to apply the retention policies right away and permanently delete records exceeding the maximum age.
27. Click Reset Defaults to reset this configuration with the recommended default values.  |  24.10 This feature was introduced in 24.10

Next, learn how to configure organization data retention.