Weik.io

MFT Setup

Setting up CoreSystems and MFT definitions

To get Managed File Transfer running, you need two things: CoreSystem resources to tell Weik.io how to connect to your storage, and MFT definitions to tell it what to move and when.

Creating CoreSystems

CoreSystems act as your source and destination endpoints. Think of them as reusable connection profiles. Because they use rclone under the hood, the parameters will look familiar if you’ve used rclone before.

Azure Blob Storage

apiVersion: weik.io/v1alpha1
kind: CoreSystem
metadata:
  name: weikio_blob
  description: "Weikio Azure Blob"
spec:
  category: storage
  type: azureblob
  title: "Weikio Azure Blob"
  parameters:
    account: weikio
    key: "{{azure:fabric-azure-weikiostorage}}"

SMB File Share

apiVersion: weik.io/v1alpha1
kind: CoreSystem
metadata:
  name: company_smb
  description: Company SMB Share
spec:
  category: storage
  type: smb
  title: "Company SMB"
  parameters:
    host: fileserver.company.local
    user: integrations
    pass: '{{azure:smb-password}}'
    domain: company.local

SFTP Server

apiVersion: weik.io/v1alpha1
kind: CoreSystem
metadata:
  name: partner_sftp
  description: Partner SFTP Server
spec:
  category: storage
  type: sftp
  title: "Partner SFTP"
  parameters:
    host: sftp.partner.com
    user: integration_user
    pass: '{{azure:sftp-password}}'
    port: 22

AWS S3 Bucket

apiVersion: weik.io/v1alpha1
kind: CoreSystem
metadata:
  name: backup_s3
  description: AWS S3 Backup Bucket
spec:
  category: storage
  type: s3
  title: "Backup S3 Bucket"
  parameters:
    provider: AWS
    access_key_id: '{{azure:aws-access-key}}'
    secret_access_key: '{{azure:aws-secret-key}}'
    region: us-east-1

Using Weik.io Variables

If you’re not using an external vault like Azure Key Vault, you can use built-in Weik.io Variables to handle passwords and keys:

apiVersion: weik.io/v1alpha1
kind: CoreSystem
metadata:
  name: partner_ftp
  description: Partner FTP Server
spec:
  category: storage
  type: ftp
  title: "Partner FTP"
  parameters:
    host: ftp.partner.com
    user: '{{sys:FTP_USERNAME}}'
    pass: '{{sys:FTP_PASSWORD}}'
    port: 21

Creating MFT Definitions

Once your CoreSystems exist, you write MFT definitions to actually move the files. You just point them at the CoreSystems by name.

Basic File Transfer

Here’s how you move a file from an SMB share to Azure Blob storage every day at midnight:

apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: hr_data_to_blob
spec:
  source:
    name: company_smb
    path: integrata/data/employees.csv
  destination:
    name: weikio_blob
    path: company/
  command: copy
  schedule: 0 0 0 * * ?
  requirements:
    Location: OnPremise
  parameters:
    camel.vault.azure.tenantId: '{{env:AZURE_TENANT_ID}}'
    camel.vault.azure.clientId: '{{env:AZURE_CLIENT_ID}}'
    camel.vault.azure.clientSecret: '{{env:AZURE_CLIENT_SECRET}}'
    camel.vault.azure.vaultName: kv-integrations-prod
  additionalPackages: camel-azure-key-vault

What’s happening here:

  • source.name and destination.name point back to the CoreSystems you created earlier.
  • command tells the system what to do (copy, sync, or move).
  • schedule dictates when it runs, using standard cron syntax.
  • requirements makes sure this job runs on an agent in the right location (e.g., one that can actually reach the on-prem SMB share).
  • parameters sets up Azure Key Vault so the job can pull the needed passwords.

Filtered File Transfer

If you only want to grab specific files—say, only the CSVs from a specific folder on an SFTP server—you use a filter:

apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: reports_to_s3
spec:
  source:
    name: partner_sftp
    path: exports/reports/
  destination:
    name: backup_s3
    path: reports/
  command: copy
  filters: "*.csv"
  schedule: 0 0 * * * ?

Note: If your CoreSystems are using Weik.io Variables for their secrets, you don’t need to add all that vault configuration stuff to the MFT definition.

Security practices

Please don’t hardcode passwords in your YAML. It always seems like a quick shortcut until someone commits it to a repo. Use a secret manager.

Azure Key Vault (Best if you’re already in Azure)

You can grab secrets directly from Azure Key Vault by using {{azure:secret-name}}.

parameters:
  pass: '{{azure:smb-password}}'
  key: '{{azure:azure-storage-key}}'
  access_key_id: '{{azure:aws-access-key}}'

To make the connection work, add these vault details to the bottom of the MFT definition:

parameters:
  camel.vault.azure.tenantId: '{{env:AZURE_TENANT_ID}}'
  camel.vault.azure.clientId: '{{env:AZURE_CLIENT_ID}}'
  camel.vault.azure.clientSecret: '{{env:AZURE_CLIENT_SECRET}}'
  camel.vault.azure.vaultName: kv-integrations-prod
additionalPackages: camel-azure-key-vault

Weik.io Variables (Built-in)

The alternative is using Weik.io Variables, referenced via {{sys:VARIABLE_NAME}}.

parameters:
  pass: '{{sys:SMB_PASSWORD}}'
  key: '{{sys:AZURE_STORAGE_KEY}}'
  access_key_id: '{{sys:AWS_ACCESS_KEY}}'

You create the actual variable value using the CLI:

apiVersion: weik.io/v1alpha1
kind: Variable
metadata:
  name: SMB_PASSWORD
spec:
  value: "your-secure-password"
  isSecret: true

Read Using Variables if you need more details.

A few more common sense rules

  • Use read-only accounts when you can. A source system usually just needs to be read.
  • Rotate credentials. If you update a password in Key Vault or a Weik.io Variable, the CoreSystems pick up the change automatically.
  • Watch the logs. Monitor for failed transfers or weird patterns so you catch issues early.

Reusing CoreSystems

The nice thing about separating the storage config from the transfer job is that you can reuse connections. If five different jobs need to talk to the same SMB server, you only define the server once.

# MFT 1: Daily reports
apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: daily_reports
spec:
  source:
    name: company_smb
    path: reports/daily/
  destination:
    name: weikio_blob
    path: reports/
  command: copy
  schedule: 0 0 1 * * ?
---
# MFT 2: Weekly summaries
apiVersion: weik.io/v1alpha1
kind: MFT
metadata:
  name: weekly_summaries
spec:
  source:
    name: company_smb
    path: reports/weekly/
  destination:
    name: weikio_blob
    path: summaries/
  command: copy
  schedule: 0 0 9 * * MON

Both MFTs talk to company_smb and weikio_blob. If the SMB password changes, you update the company_smb CoreSystem, and both jobs keep running without edits.

Next steps