Weik.io

Managed File Transfers Overview

Automate and monitor file transfers with Weik.io MFT

Managed File Transfer (MFT) automates file transfers between storage systems. You can schedule transfers, monitor them, and run custom processing scripts along the way. MFT definitions reference CoreSystem resources as their source and destination endpoints.

What is MFT?

MFT provides the following capabilities:

  • Schedule transfers using cron expressions
  • Move files between any storage system supported by Rclone
  • Check transfer history and status
  • Process files in transit with Python scripts
  • Filter which files to move using glob patterns

Security practices

Do not hardcode credentials in YAML files. Use a secret manager for passwords, API keys, and tokens.

Managing secrets

Weik.io provides two ways to handle secrets in MFT definitions:

Azure Key Vault

If you use Azure Key Vault, pull secrets directly using the {{azure:secret-name}} syntax:

parameters:
  pass: '{{azure:smb-password}}'
  key: '{{azure:azure-storage-key}}'
  access_key_id: '{{azure:aws-access-key}}'

To enable this, configure the Key Vault access at the bottom of the MFT definition:

parameters:
  camel.vault.azure.tenantId: '{{env:AZURE_TENANT_ID}}'
  camel.vault.azure.clientId: '{{env:AZURE_CLIENT_ID}}'
  camel.vault.azure.clientSecret: '{{env:AZURE_CLIENT_SECRET}}'
  camel.vault.azure.vaultName: kv-integrations-prod
additionalPackages: camel-azure-key-vault

Weik.io Variables

If you do not use an external vault, you can use built-in Weik.io Variables. Reference them using the {{sys:VARIABLE_NAME}} syntax:

parameters:
  pass: '{{sys:SMB_PASSWORD}}'
  key: '{{sys:AZURE_STORAGE_KEY}}'
  access_key_id: '{{sys:AWS_ACCESS_KEY}}'

You create these variables using the Weik.io CLI:

apiVersion: weik.io/v1alpha1
kind: Variable
metadata:
  name: SMB_PASSWORD
spec:
  value: "your-secure-password"
  isSecret: true

For details, see Using Variables.

Additional security practices

  • Use read-only credentials where possible. A system that is only a source does not need write access.
  • Rotate credentials. When you update secrets in Key Vault or Variables, CoreSystems automatically pick up the new values.
  • Monitor the logs. Set up alerts to detect failing transfers.

Using CoreSystems for storage

MFT definitions do not configure storage directly. Instead, they reference CoreSystem resources. A CoreSystem is a reusable storage configuration that uses Rclone parameters.

How it fits together

CoreSystem resources

  • Define a storage backend once, then reuse it across multiple MFTs.
  • Use standard Rclone parameters.
  • Support over 40 storage providers (S3, Azure Blob, SFTP, SMB, and others).
  • Store credentials securely using variables.

MFT definitions

  • Reference CoreSystems by name for the source and destination.
  • Set the schedule, filters, and any custom processing.

Note: Weik.io is evaluating support for defining MFTs without separate CoreSystem resources in future versions to simplify the setup.

Example

The following example defines a CoreSystem for Azure Blob storage, one for an SMB share, and an MFT that copies a file between them every night at 2 AM:

apiVersion: weik.io/v1alpha1
kind: CoreSystem
metadata:
  name: company_smb
spec:
  category: storage
  type: smb
  title: "Company SMB"
  parameters:
    host: fileserver.company.local
    user: integrations
    pass: '{{azure:smb-password}}'
---
apiVersion: weik.io/v1alpha1
kind: CoreSystem
metadata:
  name: weikio_blob
spec:
  category: storage
  type: azureblob
  title: "Weikio Azure Blob"
  parameters:
    account: weikio
    key: "{{azure:azure-storage-key}}"
---
apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: daily_backup
spec:
  source:
    name: company_smb
    path: data/employees.csv
  destination:
    name: weikio_blob
    path: backups/
  command: copy
  schedule: 0 0 2 * * ?
  requirements:
    Location: OnPremise
  parameters:
    camel.vault.azure.tenantId: '{{env:AZURE_TENANT_ID}}'
    camel.vault.azure.clientId: '{{env:AZURE_CLIENT_ID}}'
    camel.vault.azure.clientSecret: '{{env:AZURE_CLIENT_SECRET}}'
    camel.vault.azure.vaultName: kv-integrations-prod
  additionalPackages: camel-azure-key-vault

Supported storage

Because MFT runs on Rclone, it supports over 40 storage providers. The most common ones are:

Cloud

  • AWS S3
  • Azure Blob Storage
  • Google Cloud Storage
  • Backblaze B2

Protocols

  • SFTP
  • FTP/FTPS
  • SMB/CIFS
  • WebDAV

Enterprise drives

  • Box
  • Dropbox
  • OneDrive
  • Google Drive

See the Rclone documentation for the full list and parameter details.

Monitoring transfers

To follow the state of your transfers:

  • Check the transfer history in the UI.
  • Review the agent logs to debug a failure.
  • Set up alerts to avoid watching the logs manually.

Next steps