Platform Architecture
Understand Weik.io's platform architecture and core components
Weik.io is a distributed integration platform. It has six components: Backend (orchestration), Agent (execution), UI (web interface), CLI (command-line tool), APIM (API gateway), and NATS (message broker). Components communicate via NATS.
Core Components
Backend
The Backend is the central orchestration component that:
- Provides REST APIs for all platform operations
- Orchestrates integrations and integration agents
- Manages configuration and state
- Coordinates communication between components
- Provides services for UI, CLI, and other components
The Backend exposes APIs for:
- Integration management
- Agent registration and control
- Configuration management
- Monitoring and metrics
- User and access management
Agent
The Integration Agent executes all integrations and processes:
- Runs Apache Camel-based integration flows
- Executes managed file transfers using RCLONE
- Processes events and messages
- Reports status back to the Backend
- Can run on separate machines for scaling
Multiple agents can be deployed for:
- Horizontal scaling - Distribute integration workload across multiple agents
- Environment separation - Run separate agents for development, staging, and production
- Geographic distribution - Deploy agents closer to data sources
- High availability - Eliminate single points of failure
UI
The web-based Admin UI provides:
- Visual interface for managing integrations
- Monitoring dashboards and metrics
- Configuration management
- Event source and subscription management
- API management interface
- Real-time logs and debugging
The UI communicates with the Backend via REST APIs.
CLI
The command-line interface enables:
- Deploying integrations from YAML files
- Managing platform configuration
- Monitoring integration status
- Viewing logs and metrics
- Automating platform operations
- CI/CD pipeline integration
Example CLI commands:
# Deploy an integration
weikio config apply integration.yaml
# Check status
weikio status
# View logs
weikio logs integration-name
APIM (API Management)
The API Management component is built on Microsoft YARP (Yet Another Reverse Proxy) and provides:
- API gateway functionality
- Authentication and authorization
- Rate limiting and quotas
- Request/response transformation
- API analytics and monitoring
- API versioning support
APIM exposes integrations as REST APIs with authentication, authorization, and rate limiting.
NATS
NATS serves as the messaging backbone and data store:
- Message broker for inter-component communication
- Event streaming for event-driven architectures
- Distributed state management
- Pub/sub messaging between components
- JetStream for persistent event storage
NATS features:
- At-least-once delivery guarantees
- Clustering support
- Subject-based routing
- Stream persistence
Configuration
Declarative Configuration
Weik.io uses YAML-based declarative configuration for:
- Integration definitions
- Event sources and subscriptions
- API definitions
- Agent configuration
- Platform settings
Configuration format:
apiVersion: weik.io/v1alpha1
kind: Integration
metadata:
name: example-integration
description: Example integration
spec:
# Integration specification
All configuration can be version-controlled and deployed via CLI or UI.
Integration Technologies
Apache Camel
All integrations are built on Apache Camel, providing:
- 300+ connectors and components
- Enterprise Integration Patterns (EIP)
- Data transformation capabilities
- Routing and mediation
- Error handling and retry logic
Apache Camel enables connections to:
- REST APIs and SOAP services
- Databases (SQL and NoSQL)
- Message queues (Kafka, RabbitMQ, NATS)
- File systems (local, FTP, SFTP)
- Cloud services (AWS, Azure, GCP)
- Enterprise systems (SAP, Salesforce)
Learn more about available components in the Apache Camel Components documentation.
RCLONE
RCLONE handles managed file transfers:
- Support for 40+ cloud storage providers
- Sync and copy operations
- Bandwidth management
- Encryption and compression
- Checksum verification
RCLONE enables file transfers with:
- Local file systems
- Cloud storage (S3, Azure Blob, GCS)
- FTP/SFTP servers
- Object storage
- Network drives (SMB/CIFS)
Architecture Diagram
┌─────────────────────────────────────────────────────────────┐
│ Users │
│ (Browser/CLI/API) │
└──────────────┬──────────────────────┬──────────────────────┘
│ │
▼ ▼
┌─────────┐ ┌──────────┐
│ UI │ │ CLI │
│ (Admin) │ │ (Tool) │
└────┬────┘ └─────┬────┘
│ │
└────────┬───────────────┘
│ REST API
▼
┌──────────────┐
│ Backend │
│ (Orchestrator)│
└───┬──────┬───┘
│ │
┌─────────┘ └──────────┐
│ │
▼ ▼
┌────────┐ ┌────────┐
│ NATS │◄────────────────►│ APIM │
│ (Msg) │ │ (API) │
└───┬────┘ └────────┘
│
│ Events/Messages
│
▼
┌────────────────┐
│ Agent(s) │
│ │
│ ┌──────────┐ │
│ │ Apache │ │
│ │ Camel │ │
│ └──────────┘ │
│ │
│ ┌──────────┐ │
│ │ RCLONE │ │
│ └──────────┘ │
└────────────────┘
Communication Flow
Integration Execution
- User deploys an integration via CLI or UI
- Backend receives the configuration
- Backend stores configuration and notifies Agent(s)
- Agent pulls integration configuration
- Agent creates Apache Camel routes
- Integration starts processing messages
- Agent reports status to Backend via NATS
- UI displays integration status and metrics
Event Processing
- Event Source publishes events to NATS stream
- NATS stores events in JetStream
- Event Subscriptions consume events from streams
- Integration Agent processes events
- Results are sent to destination systems
- Status updates flow back through NATS
API Request
- Client makes API request to APIM
- APIM authenticates and authorizes request
- APIM routes request to appropriate integration
- Integration executes via Agent
- Response flows back through APIM
- APIM applies transformations and returns response
Deployment Patterns
Single-Node Development
For development and testing:
┌────────────────────────────┐
│ Single Machine │
│ │
│ ┌──────┐ ┌────────────┐ │
│ │ NATS │ │ Backend │ │
│ └──────┘ │ + Agent │ │
│ │ + UI │ │
│ └────────────┘ │
└────────────────────────────┘
All components run on a single machine.
Multi-Agent Production
For production workloads:
┌──────────────┐ ┌──────────────┐
│ Backend │ │ Agent 1 │
│ + UI │ │ (Region A) │
│ + NATS │◄────►│ │
└──────────────┘ └──────────────┘
▲
│ ┌──────────────┐
└─────────────►│ Agent 2 │
│ (Region B) │
└──────────────┘
Separate agents for scaling and geographic distribution.
High Availability
For mission-critical deployments:
┌──────────────┐ ┌──────────────┐
│ Backend 1 │◄────►│ Backend 2 │
│ + UI │ │ + UI │
└──────┬───────┘ └──────┬───────┘
│ │
└──────┬──────────────┘
▼
┌───────────────┐
│ NATS Cluster │
│ (3 nodes) │
└───────┬───────┘
│
┌──────┴──────┐
▼ ▼
┌─────────┐ ┌─────────┐
│ Agent 1 │ │ Agent 2 │
└─────────┘ └─────────┘
Clustered NATS and multiple instances of all components.
Scalability
Horizontal Scaling
Add more agents to handle increased load:
- Each agent can process integrations independently
- NATS distributes work across available agents
- No limit to the number of agents
Vertical Scaling
Increase resources for individual components:
- More CPU for complex transformations
- More memory for large message processing
- Faster storage for high-throughput file transfers
Resource Requirements
Minimum requirements per component:
| Component | CPU | Memory | Storage |
|---|---|---|---|
| Backend | 2 cores | 2GB | 10GB |
| Agent | 2 cores | 2GB | 20GB |
| NATS | 1 core | 1GB | 10GB |
| UI | 1 core | 512MB | 1GB |
Production recommendations:
- Backend: 4 cores, 4GB RAM
- Agent: 4 cores, 4GB RAM (per agent)
- NATS: 2 cores, 2GB RAM (3-node cluster)
What’s Next
Now that you understand the architecture:
- Quickstart - Build your first integration
- Integrations Overview - Learn about creating integrations
- APIM Overview - Explore API management features