Weik.io

File Transfer Patterns

Common file transfer patterns and use cases

Use file transfer patterns to implement common data movement scenarios between storage systems.

Rclone Commands

MFT uses rclone for file transfers. Each command provides different transfer behavior.

copy

Copies files from source to destination without deleting files from destination.

command: copy

Behavior:

  • Copies new and modified files
  • Keeps existing files in destination
  • Does not delete anything
  • Safe for backups and one-way transfers

Use cases:

  • Backups
  • Data replication
  • Log aggregation

sync

Makes destination identical to source by copying files and deleting files in destination not present in source.

command: sync

Behavior:

  • Copies new and modified files
  • Deletes files in destination not in source
  • Makes destination match source exactly
  • Use with caution - can delete data

Use cases:

  • Directory mirroring
  • Website deployments
  • Maintaining exact replicas

move

Copies files to destination then deletes them from source.

command: move

Behavior:

  • Copies files to destination
  • Deletes files from source after successful copy
  • Ensures files are moved, not duplicated
  • Use when source storage is limited

Use cases:

  • Processing pipelines
  • Archiving data
  • Clearing source directories

File Filtering

Use glob patterns to filter files for transfer:

filters: "*.csv"              # Single extension
filters: "*.{csv,json,xml}"   # Multiple extensions
filters: "report_*.csv"       # Prefix matching
filters: "data_2025-*.csv"    # Pattern matching

Examples:

Transfer only CSV files:

filters: "*.csv"

Transfer multiple file types:

filters: "*.{csv,json,xml}"

Transfer files with specific prefix:

filters: "report_*.csv"

Transfer files matching date pattern:

filters: "data_2025-*.csv"

Common Use Cases

Daily Database Backups

Export database dumps and transfer to cloud storage:

apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: db_backup_to_cloud
spec:
  source:
    name: backup_server_smb
    path: backups/database/
  destination:
    name: backup_s3
    path: database/
  command: copy
  schedule: 0 0 2 * * ?
  filters: "*.sql.gz"

Partner File Exchange

Transfer files to partner SFTP servers:

apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: invoice_to_partner
spec:
  source:
    name: company_smb
    path: exports/invoices/
  destination:
    name: partner_sftp
    path: incoming/
  command: move
  schedule: 0 0 */2 * * ?
  filters: "invoice_*.xml"

The move command removes source files after successful transfer, preventing duplicate processing.

Cloud-to-Cloud Migration

Transfer files between cloud storage providers:

apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: azure_to_s3_migration
spec:
  source:
    name: weikio_blob
    path: archive/
  destination:
    name: backup_s3
    path: migration/
  command: copy
  schedule: 0 0 3 * * ?

Bidirectional Sync

Keep directories synchronized between storage systems:

apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: document_sync
spec:
  source:
    name: company_smb
    path: shared/documents/
  destination:
    name: weikio_blob
    path: documents/
  command: sync
  schedule: 0 */4 * * * ?

The sync command makes destination match source exactly. Files present only in destination are deleted.

What’s Next