This page is for data modellers or administrators who need to manage stream data retention and deletion.
How to Configure Stream Data Retention and Deletion
...
When a data modeller creates a stream they should set up the Data Retention Settings to specify the:
...
- either one or more specific streams. You may choose to do this, for example:
- to manage the data in several related streams
- to run frequently on a stream that contains a large amount of data.
- to run occasionally on streams with low volumes of data that change rarely.
- or have the All Streams option ticked. The task will run on all streams:
- that are not in another stream-data-delete task
- and have Data Retention Settings configured.
Managing
...
Stream Sets
| Anchor | ||||
|---|---|---|---|---|
|
The following table shows the different combinations of settings possible in stream properties → Data Retention Settings. It assumes that a stream currently contains 8 stream sets:
...
- N: a number of days
- X: a number of stream sets
null: indicates no value has been entered for this option.
...
| Archive After X Days | Keeping Latest Y StreamSets | Resulting Streams Archived/RetainedAction taken when stream-data-delete task runs |
|---|---|---|
| null | null | No stream sets will be deleted.All streams sets are retained. |
| 0 | null | All stream sets will be deletedare deleted. |
| 1 | null | The last day of valid steam sets will be are retained. All earlier stream sets will be are deleted. In our example the 2 latest most recent stream sets will be retained with the earliest 6 older stream sets deleted. |
| N | null | All stream sets which are older than N days before the latest valid stream set will be are deleted. |
| null | 0 | All stream sets will be deletedare deleted. |
| null | 1 | The last most recent valid stream set will be is retained, all other stream sets will be deletedare deleted. |
| null | X | The most recent Y X valid stream sets will be are retained, all others stream sets will be are deleted. |
| 0 | 0 | All stream sets will be deletedare deleted. |
| 0 | 1 | The last most recent valid stream set will be is retained, all other stream sets will be deletedare deleted. |
| 1 | 0 | The last day of valid steam sets will be are retained. All earlier older stream sets will be are archived. |
| 1 | 1 | The last day of valid steam sets will be are retained regardless of if there are more than 1. If there are no stream sets in the last day, then the first previous stream set will be is retained instead. |
| N | X | Will retain Retains the maximum number of active stream sets in the data, such that no conflicting stream sets will be are deleted. |
Superseded Stream Sets
| Anchor | ||||
|---|---|---|---|---|
|
In the case where only the If only Keep Superseded for N Days and Keep Superseded for X StreamSets fields are populated, the same logic in the table above will apply applies to the superseded records. Note that again archiving will always retain the maximum superseded stream sets in the data such that no conflicting stream sets will be archived.In the cases where a mixture of the full archive fields Keep for X Days, Keep for Y StreamSets' and the superseded archive fields Keep Superseded for X Days, Keep Superseded for Y StreamSets are populated, then the full archive values will be first applied and the resultant stream item records will be archived and deleted. Only then will the remaining stream sets use the Keep Superseded ... values to apply a further condition to archive and delete any remaining non qualifying
If a stream has values for all the properties
- Keep for N Days
- Keep for X StreamSets
- Keep Superseded for N Days
- Keep Superseded for X StreamSets
then the values for stream sets are applied first, deleting full stream sets. Then the stream values for superseded records are applied to delete any remaining superseded records.