Filters

The filter mechanism lets the agent read, process, and write data. Filters for files and parcels can read textual data and binary data, respectively, compress data, encrypt data, write binary data, decompress data, unencrypt data.

Agents read and send data from one to another in chunks that are known as parcels. A parcel is the amount of raw data a sender agent reads before it applies any parcel filters and before it sends the data across the network to the receiver agent. The default parcel size is approximately 500 KB. So, for example, the process of transferring 10 MB of data involves the reading and sending of 20 parcels.

You can create your own filters and corresponding filter objects for use during transfers, or you can use several predefined file and parcel filters.

Valid DTSCLI file filters include the following types:

Directory filters
Encryption filters
File attribute filters

When adding the file filters, parcel filters can also be needed. To transfer a directory, file filters must be added to the dtscli command. If you try to transfer a directory without adding the directory filter, you get the following error message:

"Failed to open input data <c:\dtstemp> error=<Permission denied>".

Valid DTSCLI parcel filters include the following types:

Binary (default) or text
Compress
Encrypt

The two main parcel filters that are used are either binary or text. Use the text filter if transferring text files between two different types of computers (Windows and UNIX, for example). This ensures that the text format is preserved; otherwise, always use binary filters.

Note: Binary and text filters are mutually exclusive: A transfer can either be text or binary but not both.

Filters usually work in pairs. The write filter must reverse an action that is performed by the read filter. Filters are applied in the following order:

File Read—Works on a file when reading the source data
Parcel Read—Works on a buffer of data when reading the source data
Parcel Write—Works on a buffer of data when writing the destination data
File Write—Works on a file when writing the destination data

Read the filters of the same type (file or parcel) are applied in the same order that they are specified on the command line. Write the filters of the same type are applied in the reverse order that they are specified on the command line.

DTS also supports external file and parcel filters.

Example: Transfer a Directory

The following example transfers a directory using a short format for the directory filter:
```
dtscli -agent ipath=Gold::c:\dtstemp 
```
```
"rpath=Silver::c:\dtstemp" 
"f_filters=dir"
```

This example performs the same operation but uses a long format:

dtscli -agent "ipath=Gold::c:\dtstemp"

"rpath=Silver::c:\dtstemp" 
"f_filters=DIRTREE_READ:DIRTREE_WRITE"
"p_filters=binary"

Example: Transfer a File as Text

The following example transfers the file from Gold to Silver using the text filter and a short format:

dtscli -agent "ipath=Gold::c:\dtstemp\file.src"

"rpath=Silver::c:\dtstemp\file.dest" 
"p_filters=text"

This example performs the same operation but uses a long format:

dtscli -agent "ipath=Gold::c:\dtstemp\file.src"

"rpath=Silver::c:\dtstemp\file.dest" 
"p_filters=TEXT_READ:TEXT_WRITE"

Note: Only use text filters for transferring text files.

Note: For detailed information, see the DTSCLI Command Reference Guide.

More information:

Support for External Filters

Creating Transfer Objects