Weik.io

Managed File Transfers Overview

Automate and monitor file transfers with Weik.io MFT

Managed File Transfer (MFT) automates file transfers between storage systems with scheduling, monitoring, and optional file processing. MFT definitions reference CoreSystem resources as source and destination endpoints.

What is MFT?

MFT provides automated file transfer capabilities:

  • Schedule transfers using cron expressions
  • Transfer files between any rclone-supported storage systems
  • Monitor transfer status and history
  • Process files during transfer with Python scripts
  • Filter files using glob patterns

Security Best Practices

NEVER hardcode credentials in YAML definitions. Always use secure secret management for passwords, API keys, and access tokens.

Weik.io supports two primary methods for managing secrets in MFT definitions:

Azure Key Vault (Recommended for Azure Environments)

Use Azure Key Vault to store and retrieve secrets. Reference secrets using the {{azure:secret-name}} syntax:

parameters:
  pass: '{{azure:smb-password}}'
  key: '{{azure:azure-storage-key}}'
  access_key_id: '{{azure:aws-access-key}}'

Configure Azure Key Vault access in your MFT definition:

parameters:
  camel.vault.azure.tenantId: '{{env:AZURE_TENANT_ID}}'
  camel.vault.azure.clientId: '{{env:AZURE_CLIENT_ID}}'
  camel.vault.azure.clientSecret: '{{env:AZURE_CLIENT_SECRET}}'
  camel.vault.azure.vaultName: kv-integrations-prod
additionalPackages: camel-azure-key-vault

Weik.io Variables (Platform-Native Secret Management)

Use Weik.io Variables for centralized secret management. Reference variables using the {{sys:VARIABLE_NAME}} syntax:

parameters:
  pass: '{{sys:SMB_PASSWORD}}'
  key: '{{sys:AZURE_STORAGE_KEY}}'
  access_key_id: '{{sys:AWS_ACCESS_KEY}}'

Create variables using the Weik.io CLI:

apiVersion: weik.io/v1alpha1
kind: Variable
metadata:
  name: SMB_PASSWORD
spec:
  value: "your-secure-password"
  isSecret: true

See Using Variables for complete documentation.

Additional Security Practices

Use Read-Only Credentials When Possible

  • Restrict source systems to read-only access
  • Grant write access only where required

Rotate Credentials Regularly

  • Update secrets in Key Vault or Variables
  • CoreSystems automatically use updated values

Monitor Access Patterns

  • Review transfer logs regularly
  • Set up alerts for unusual activity

CoreSystems as Storage Backends

MFT definitions use CoreSystem resources to define source and destination storage systems. CoreSystems are reusable storage backend definitions configured with rclone-compatible parameters.

Key Concepts

CoreSystem Resources

  • Define storage backends once, reference in multiple MFTs
  • Configure with rclone-compatible parameters
  • Support 40+ storage providers (S3, Azure Blob, SFTP, SMB, etc.)
  • Store credentials securely using variables

MFT Definitions

  • Reference CoreSystems by name for source and destination
  • Define transfer schedule, filters, and processing
  • Configure transfer behavior with parameters

Note: In future versions, MFTs may be defined without requiring separate CoreSystem resources.

Simple Example

Define a CoreSystem for Azure Blob storage and an MFT to transfer files from SMB:

apiVersion: weik.io/v1alpha1
kind: CoreSystem
metadata:
  name: company_smb
spec:
  category: storage
  type: smb
  title: "Company SMB"
  parameters:
    host: fileserver.company.local
    user: integrations
    pass: '{{azure:smb-password}}'
---
apiVersion: weik.io/v1alpha1
kind: CoreSystem
metadata:
  name: weikio_blob
spec:
  category: storage
  type: azureblob
  title: "Weikio Azure Blob"
  parameters:
    account: weikio
    key: "{{azure:azure-storage-key}}"
---
apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: daily_backup
spec:
  source:
    name: company_smb
    path: data/employees.csv
  destination:
    name: weikio_blob
    path: backups/
  command: copy
  schedule: 0 0 2 * * ?
  requirements:
    Location: OnPremise
  parameters:
    camel.vault.azure.tenantId: '{{env:AZURE_TENANT_ID}}'
    camel.vault.azure.clientId: '{{env:AZURE_CLIENT_ID}}'
    camel.vault.azure.clientSecret: '{{env:AZURE_CLIENT_SECRET}}'
    camel.vault.azure.vaultName: kv-integrations-prod
  additionalPackages: camel-azure-key-vault

This example creates two CoreSystems (SMB and Azure Blob) and an MFT that copies a file daily at 2 AM.

Supported Storage Types

MFT supports 40+ storage providers through rclone, including:

Cloud Storage

  • AWS S3
  • Azure Blob Storage
  • Google Cloud Storage
  • Backblaze B2

File Transfer

  • SFTP
  • FTP/FTPS
  • SMB/CIFS
  • WebDAV

Enterprise Storage

  • Box
  • Dropbox
  • OneDrive
  • Google Drive

See rclone documentation for complete list and configuration parameters.

Monitoring Transfers

Monitor MFT operations:

  • View transfer history in UI
  • Check agent logs for detailed execution
  • Set up alerts for transfer failures
  • Track file counts and transfer durations

What’s Next

Learn more about MFT features: