Weik.io

Managed File Transfers Overview

Automate and monitor file transfers with Weik.io MFT

Managed File Transfer (MFT) automates file transfers between storage systems. You can schedule transfers, monitor them, and run custom processing scripts along the way. Under the hood, MFT definitions reference CoreSystem resources as their source and destination endpoints.

What is MFT?

MFT handles the heavy lifting of moving files around:

  • Schedule transfers using cron expressions
  • Move files between any storage system supported by rclone
  • Check transfer history and status
  • Process files in transit with Python scripts
  • Filter which files to move using glob patterns

Security practices

Don’t hardcode credentials in your YAML files. It’s an easy mistake to make, but you should always use a secret manager for passwords, API keys, and tokens.

Managing secrets

Weik.io gives you two main ways to handle secrets in your MFT definitions:

Azure Key Vault (Best if you’re already in Azure)

If you use Azure Key Vault, you can pull secrets directly using the {{azure:secret-name}} syntax:

parameters:
  pass: '{{azure:smb-password}}'
  key: '{{azure:azure-storage-key}}'
  access_key_id: '{{azure:aws-access-key}}'

To make this work, configure the Key Vault access at the bottom of your MFT definition:

parameters:
  camel.vault.azure.tenantId: '{{env:AZURE_TENANT_ID}}'
  camel.vault.azure.clientId: '{{env:AZURE_CLIENT_ID}}'
  camel.vault.azure.clientSecret: '{{env:AZURE_CLIENT_SECRET}}'
  camel.vault.azure.vaultName: kv-integrations-prod
additionalPackages: camel-azure-key-vault

Weik.io Variables (Built-in)

If you don’t want to use an external vault, you can use Weik.io Variables. Reference them using the {{sys:VARIABLE_NAME}} syntax:

parameters:
  pass: '{{sys:SMB_PASSWORD}}'
  key: '{{sys:AZURE_STORAGE_KEY}}'
  access_key_id: '{{sys:AWS_ACCESS_KEY}}'

You create these variables using the Weik.io CLI:

apiVersion: weik.io/v1alpha1
kind: Variable
metadata:
  name: SMB_PASSWORD
spec:
  value: "your-secure-password"
  isSecret: true

For the full details, check out Using Variables.

A few more security tips

  • Use read-only credentials where you can. If a system is just a source, it doesn’t need write access.
  • Rotate your credentials. When you update secrets in Key Vault or Variables, CoreSystems automatically pick up the new values.
  • Keep an eye on the logs. Set up alerts so you know if transfers start failing or acting weird.

Using CoreSystems for storage

MFT definitions don’t configure storage directly. Instead, they reference CoreSystem resources. A CoreSystem is basically a reusable storage configuration that uses rclone parameters.

How it fits together

CoreSystem Resources

  • You define a storage backend once, then reuse it across multiple MFTs.
  • They use standard rclone parameters.
  • They support over 40 storage providers (S3, Azure Blob, SFTP, SMB, etc.).
  • They let you store credentials securely using variables.

MFT Definitions

  • You point to CoreSystems by name for your source and destination.
  • Here is where you set the schedule, filters, and any custom processing.

Note: We’re looking at allowing MFTs to be defined without separate CoreSystem resources in future versions to simplify the setup.

A quick example

Here’s how you define a CoreSystem for Azure Blob storage, one for an SMB share, and an MFT to copy a file between them every night at 2 AM:

apiVersion: weik.io/v1alpha1
kind: CoreSystem
metadata:
  name: company_smb
spec:
  category: storage
  type: smb
  title: "Company SMB"
  parameters:
    host: fileserver.company.local
    user: integrations
    pass: '{{azure:smb-password}}'
---
apiVersion: weik.io/v1alpha1
kind: CoreSystem
metadata:
  name: weikio_blob
spec:
  category: storage
  type: azureblob
  title: "Weikio Azure Blob"
  parameters:
    account: weikio
    key: "{{azure:azure-storage-key}}"
---
apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: daily_backup
spec:
  source:
    name: company_smb
    path: data/employees.csv
  destination:
    name: weikio_blob
    path: backups/
  command: copy
  schedule: 0 0 2 * * ?
  requirements:
    Location: OnPremise
  parameters:
    camel.vault.azure.tenantId: '{{env:AZURE_TENANT_ID}}'
    camel.vault.azure.clientId: '{{env:AZURE_CLIENT_ID}}'
    camel.vault.azure.clientSecret: '{{env:AZURE_CLIENT_SECRET}}'
    camel.vault.azure.vaultName: kv-integrations-prod
  additionalPackages: camel-azure-key-vault

Supported storage

Because MFT runs on rclone, it supports over 40 storage providers. The most common ones are:

Cloud

  • AWS S3
  • Azure Blob Storage
  • Google Cloud Storage
  • Backblaze B2

Protocols

  • SFTP
  • FTP/FTPS
  • SMB/CIFS
  • WebDAV

Enterprise drives

  • Box
  • Dropbox
  • OneDrive
  • Google Drive

Check the rclone documentation for the full list and parameter details.

Keeping track of transfers

If you want to see what’s happening with your files:

  • Check the transfer history in the UI.
  • Dig into the agent logs if you need to debug a failure.
  • Set up alerts so you don’t have to watch the logs manually.

Next steps