Skip to main content

Backup and Restore

This section covers backup and restore operations for BK-Lite's critical components, helping operations engineers develop a data protection strategy.

Backup Priority​

Based on data importance, the following backup priority is recommended:

PriorityComponentData TypeRecommended Frequency
P0PostgreSQLCore business dataDaily full + real-time archiving
P0PGVector (metis)AI vector dataDaily full
P1FalkorDBCMDB graph dataDaily full
P2MinIOFiles/model artifactsWeekly full + daily incremental
P2VictoriaMetricsMonitoring metricsAs needed (can be re-collected)
P2VictoriaLogsLog dataAs needed (can be re-collected)
P3Redis (rebuildable cache when RUM is disabled)Cache dataDepends on actual use
P1Conductor (managed Workflow engine)SQLite engine dataDaily and before upgrades; consistent backup
P1RUM (when enabled)Configuration/keys, Redis state, replay, queue, Logs/TracesCoordinated backups based on recovery objectives

Prerequisites​

Deployment Directory​

BK-Lite is deployed by default in the /opt/bk-lite directory. All docker compose commands must be executed from this directory:

# Navigate to the deployment directory
cd /opt/bk-lite

# Check currently running services
docker compose ps

# Load environment variables (for obtaining passwords and other config in scripts)
source .env
Tip

All commands below assume execution from the /opt/bk-lite directory. If you use a custom deployment directory, adjust the paths accordingly.

Create Backup Directory​

# Create backup root directory
mkdir -p /data/backup/bklite/{postgres,pgvector,falkordb,victorialogs,minio}

# Set permissions
chmod 750 /data/backup/bklite

PostgreSQL Backup and Restore​

PostgreSQL stores the system's core business data and is the most important backup target.

Database Details​

DatabasePurposeBackup Necessity
bkliteMain business data (users, configurations, CMDB, etc.)Required
mlflowMLflow experiment tracking dataRecommended

Full Backup​

# Backup all databases
docker compose exec -T postgres pg_dumpall -U postgres > /data/backup/bklite/postgres/full_$(date +%Y%m%d_%H%M%S).sql

# Or backup individual databases (recommended)
docker compose exec -T postgres pg_dump -U postgres -d bklite > /data/backup/bklite/postgres/bklite_$(date +%Y%m%d_%H%M%S).sql
docker compose exec -T postgres pg_dump -U postgres -d mlflow > /data/backup/bklite/postgres/mlflow_$(date +%Y%m%d_%H%M%S).sql
# Backup in custom format (supports parallel restore)
docker compose exec -T postgres pg_dump -U postgres -Fc -d bklite > /data/backup/bklite/postgres/bklite_$(date +%Y%m%d_%H%M%S).dump
docker compose exec -T postgres pg_dump -U postgres -Fc -d mlflow > /data/backup/bklite/postgres/mlflow_$(date +%Y%m%d_%H%M%S).dump

Restore Database​

Warning

The restore operation will overwrite existing data. Make sure the target environment is correct!

# Restore from SQL file
docker compose exec -T postgres psql -U postgres -d bklite < /data/backup/bklite/postgres/bklite_20240101_120000.sql

# Restore from dump file (requires an empty database first)
cat /data/backup/bklite/postgres/bklite_20240101_120000.dump | docker compose exec -T postgres pg_restore -U postgres -d bklite -c

Automated Backup Script​

Create a scheduled backup script at /opt/bk-lite/scripts/backup_postgres.sh:

#!/bin/bash
set -e

BACKUP_DIR="/data/backup/bklite/postgres"
KEEP_DAYS=7
DATE=$(date +%Y%m%d_%H%M%S)
BKLITE_DIR="/opt/bk-lite"

cd "${BKLITE_DIR}"

# Create backup
echo "[$(date)] Starting PostgreSQL backup..."
docker compose exec -T postgres pg_dump -U postgres -Fc -d bklite > "${BACKUP_DIR}/bklite_${DATE}.dump"
docker compose exec -T postgres pg_dump -U postgres -Fc -d mlflow > "${BACKUP_DIR}/mlflow_${DATE}.dump"

# Clean up expired backups
find "${BACKUP_DIR}" -name "*.dump" -mtime +${KEEP_DAYS} -delete
echo "[$(date)] PostgreSQL backup completed."

Add to crontab:

# Execute backup daily at 2:00 AM
0 2 * * * /opt/bk-lite/scripts/backup_postgres.sh >> /var/log/bklite_backup.log 2>&1

PGVector Backup and Restore​

PGVector runs as a PostgreSQL extension, with its data stored in the metis database.

Backup Vector Data​

cd /opt/bk-lite

# PGVector runs in a separate container, connected via the postgres container
# Backup the metis database (contains vector tables)
docker compose exec -T postgres pg_dump -U postgres -Fc -d metis > /data/backup/bklite/pgvector/metis_$(date +%Y%m%d_%H%M%S).dump

Restore Vector Data​

cd /opt/bk-lite

# Ensure the pgvector extension is installed before restoring
docker compose exec -T postgres psql -U postgres -d metis -c "CREATE EXTENSION IF NOT EXISTS vector;"

# Restore data
cat /data/backup/bklite/pgvector/metis_20240101_120000.dump | docker compose exec -T postgres pg_restore -U postgres -d metis -c
Tip

If restoring to a new environment, you need to create the metis database first:

docker compose exec -T postgres psql -U postgres -c "CREATE DATABASE metis;"

FalkorDB Backup and Restore​

FalkorDB is based on the Redis protocol and stores CMDB asset relationship graphs.

Note

FalkorDB maps port 6479:6379 in Docker Compose, meaning the host uses port 6479.

Backup Methods​

FalkorDB supports two backup methods:

cd /opt/bk-lite
source .env

# Trigger RDB persistence
docker compose exec falkordb redis-cli -a ${FALKORDB_PASSWORD} BGSAVE

# Wait for backup to complete (check last save time)
docker compose exec falkordb redis-cli -a ${FALKORDB_PASSWORD} LASTSAVE

# Copy RDB file
docker compose cp falkordb:/var/lib/falkordb/data/dump.rdb /data/backup/bklite/falkordb/dump_$(date +%Y%m%d_%H%M%S).rdb

Method 2: Data Volume Backup​

cd /opt/bk-lite

# Stop service (optional, ensures data consistency)
docker compose stop falkordb

# Get volume name
VOLUME_NAME=$(docker volume ls --filter name=falkordb -q | head -1)

# Backup data volume
docker run --rm -v ${VOLUME_NAME}:/source -v /data/backup/bklite/falkordb:/backup alpine \
  tar czf /backup/falkordb_volume_$(date +%Y%m%d_%H%M%S).tar.gz -C /source .

# Restart service
docker compose start falkordb

Restore Methods​

Restore from RDB File​

cd /opt/bk-lite

# Stop FalkorDB
docker compose stop falkordb

# Copy RDB file to data volume
docker compose cp /data/backup/bklite/falkordb/dump_20240101_120000.rdb falkordb:/var/lib/falkordb/data/dump.rdb

# Start FalkorDB (automatically loads RDB)
docker compose start falkordb

Restore from Data Volume Backup​

cd /opt/bk-lite

# Stop service
docker compose stop falkordb

# Get volume name
VOLUME_NAME=$(docker volume ls --filter name=falkordb -q | head -1)

# Clear and restore data volume
docker run --rm -v ${VOLUME_NAME}:/target -v /data/backup/bklite/falkordb:/backup alpine \
  sh -c "rm -rf /target/* && tar xzf /backup/falkordb_volume_20240101_120000.tar.gz -C /target"

# Start service
docker compose start falkordb

VictoriaLogs Backup and Restore​

VictoriaLogs stores log data on port 9428. Log data can typically be re-collected, but retaining historical logs may be valuable for auditing and analysis.

Note

VictoriaLogs data is stored in the victoria-logs data volume under the /vlogs directory.

Backup Methods​

VictoriaLogs supports online snapshots with no downtime required:

cd /opt/bk-lite

# Create snapshot
docker compose exec victoria-logs wget -qO- 'http://localhost:9428/snapshot/create'

# List snapshots
docker compose exec victoria-logs ls -la /vlogs/snapshots/

# Copy snapshot directory
SNAPSHOT_NAME=$(docker compose exec victoria-logs ls /vlogs/snapshots/ | tail -1 | tr -d '\r')
docker compose cp victoria-logs:/vlogs/snapshots/${SNAPSHOT_NAME} /data/backup/bklite/victorialogs/snapshot_$(date +%Y%m%d_%H%M%S)

Method 2: Data Volume Backup​

cd /opt/bk-lite

# Stop service (ensures data consistency)
docker compose stop victoria-logs

# Get volume name
VOLUME_NAME=$(docker volume ls --filter name=victoria-logs -q | head -1)

# Backup data volume
docker run --rm -v ${VOLUME_NAME}:/source -v /data/backup/bklite/victorialogs:/backup alpine \
  tar czf /backup/victorialogs_volume_$(date +%Y%m%d_%H%M%S).tar.gz -C /source .

# Restart service
docker compose start victoria-logs

Restore Methods​

Restore from Snapshot​

cd /opt/bk-lite

# Stop service
docker compose stop victoria-logs

# Get volume name
VOLUME_NAME=$(docker volume ls --filter name=victoria-logs -q | head -1)

# Clear data directory and restore snapshot
docker run --rm -v ${VOLUME_NAME}:/vlogs -v /data/backup/bklite/victorialogs:/backup alpine \
  sh -c "rm -rf /vlogs/* && cp -r /backup/snapshot_20240101_120000/* /vlogs/"

# Start service
docker compose start victoria-logs

Restore from Data Volume Backup​

cd /opt/bk-lite

# Stop service
docker compose stop victoria-logs

# Get volume name
VOLUME_NAME=$(docker volume ls --filter name=victoria-logs -q | head -1)

# Clear and restore data volume
docker run --rm -v ${VOLUME_NAME}:/target -v /data/backup/bklite/victorialogs:/backup alpine \
  sh -c "rm -rf /target/* && tar xzf /backup/victorialogs_volume_20240101_120000.tar.gz -C /target"

# Start service
docker compose start victoria-logs

RUM and Workflow Backup and Restore​

The generic backup script and disaster recovery steps below do not cover all optional component data. Follow these additional steps when enabled. See the deployment guide for component selection.

Configuration and Versions​

Save common.env, .env, the generated docker-compose.yaml, selected component templates, and current image references/digests from the actual deployment directory. Encrypt backups containing credentials and restrict access. Record actual volume mounts and restore the original volumes or their contents to the correct replacement volumes; starting a service with the same name does not restore its data.

Workflow / Conductor​

The managed engine mounts compose_conductor at /data, containing conductor.db and its WAL/SHM files:

  1. Pause workflow entry points and new triggers, drain running executions or suspend them as required, stop Workflow background processes, then run docker compose stop conductor.
  2. With the engine stopped, back up the entire volume, preserving permissions. Copying only a live conductor.db is insufficient. Also back up workflow definitions/history in platform PostgreSQL, artifacts in MinIO, and connection settings.
  3. Start the engine and verify health, then resume background processes and triggers. Schedule downtime so triggers do not keep submitting work during the backup.
  4. For restoration, keep writes paused, restore matching engine data, platform data, image versions, and CONDUCTOR_BASE_URL, then start services and verify execution, approval, and artifact downloads.

Back up external engines according to their database/storage design. Changing the engine endpoint does not migrate data. Rollback requires the previous image and matching data backup. Never allow two engines to write to the same SQLite volume.

RUM​

RUM spans multiple stores; platform PostgreSQL and the platform victoria-logs volume alone are insufficient:

  • Configuration and keys: Preserve RUM_IDENTITY_HMAC_CURRENT, RUM_REPLAY_SIGNING_SECRET, RUM_REPLAY_INDEX_EPOCH, dedicated NATS credentials, and the compose_rum-secrets volume. Restore the original identities, signing material, and actual index epoch rather than generating replacements.
  • Shared Redis: RUM uses DB0 for admission, DB2 for replay indexes, and DB4 for sessions. Back up according to the configured persistence and verify recovery coverage. Redis must not be treated entirely as disposable cache. Restoring the shared instance affects other modules, so coordinate recovery points.
  • Replay and queue: Back up the MinIO rum-replay bucket, including required object versions when versioning is enabled, and compose_rum-gateway-queue. Undelivered queue events may not yet exist in backend stores. Do not directly merge two nonempty queues during recovery.
  • Experience and trace data: Back up compose_rum-victoria-logs and compose_rum-victoria-traces volumes separately, or use the respective storage backup procedures. These are independent of platform logs and APM stores. Historical browser visits generally cannot be collected again.

For consistent volume backups, pause browser ingestion, RUM writers, and maintenance processes, and coordinate backup times across shared Redis/MinIO and dedicated Logs/Traces. Account for other modules before stopping shared services. Restore configuration, keys, and stores while writes remain paused, then restore RUM services. After signing in, confirm controlAvailable is true at /api/v1/rum/health/, and verify historical queries, replay, and new writes. Disabling a module retains data but is not a backup.

Complete Backup Script​

Create an all-in-one backup script at /opt/bk-lite/scripts/backup_all.sh:

#!/bin/bash
set -e

# Configuration
BACKUP_ROOT="/data/backup/bklite"
BKLITE_DIR="/opt/bk-lite"
KEEP_DAYS=7
DATE=$(date +%Y%m%d_%H%M%S)
LOG_FILE="/var/log/bklite_backup.log"

# Navigate to deployment directory and load environment variables
cd "${BKLITE_DIR}"
source .env

log() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a ${LOG_FILE}
}

log "========== Starting Backup =========="

# Ensure backup directories exist
mkdir -p "${BACKUP_ROOT}"/{postgres,pgvector,falkordb,victorialogs}

# 1. PostgreSQL
log "Backing up PostgreSQL..."
docker compose exec -T postgres pg_dump -U postgres -Fc -d bklite > "${BACKUP_ROOT}/postgres/bklite_${DATE}.dump"
docker compose exec -T postgres pg_dump -U postgres -Fc -d mlflow > "${BACKUP_ROOT}/postgres/mlflow_${DATE}.dump"
log "PostgreSQL backup completed"

# 2. PGVector (metis)
log "Backing up PGVector..."
docker compose exec -T postgres pg_dump -U postgres -Fc -d metis > "${BACKUP_ROOT}/pgvector/metis_${DATE}.dump"
log "PGVector backup completed"

# 3. FalkorDB
log "Backing up FalkorDB..."
docker compose exec falkordb redis-cli -a ${FALKORDB_PASSWORD} BGSAVE 2>/dev/null || true
sleep 5  # Wait for BGSAVE to complete
docker compose cp falkordb:/var/lib/falkordb/data/dump.rdb "${BACKUP_ROOT}/falkordb/dump_${DATE}.rdb"
log "FalkorDB backup completed"

# 4. VictoriaLogs (optional)
log "Backing up VictoriaLogs..."
docker compose exec victoria-logs wget -qO- 'http://localhost:9428/snapshot/create' || true
sleep 2
SNAPSHOT_NAME=$(docker compose exec victoria-logs ls /vlogs/snapshots/ 2>/dev/null | tail -1 | tr -d '\r')
if [ -n "${SNAPSHOT_NAME}" ]; then
    docker compose cp victoria-logs:/vlogs/snapshots/${SNAPSHOT_NAME} "${BACKUP_ROOT}/victorialogs/snapshot_${DATE}"
    log "VictoriaLogs backup completed"
else
    log "VictoriaLogs backup skipped (no snapshot)"
fi

# 5. Clean up expired backups
log "Cleaning up backups older than ${KEEP_DAYS} days..."
find "${BACKUP_ROOT}" -type f \( -name "*.dump" -o -name "*.rdb" -o -name "*.tar.gz" \) -mtime +${KEEP_DAYS} -delete
find "${BACKUP_ROOT}" -type d -name "snapshot_*" -mtime +${KEEP_DAYS} -exec rm -rf {} + 2>/dev/null || true

log "========== Backup Completed =========="

# List backup files
log "Backup files from this run:"
find "${BACKUP_ROOT}" -type f -mmin -10 -exec ls -lh {} \;

Scheduled Task Configuration​

# Add execute permission
chmod +x /opt/bk-lite/scripts/backup_all.sh

# Configure crontab
crontab -e

# Add the following (execute daily at 2:00 AM)
0 2 * * * /opt/bk-lite/scripts/backup_all.sh

Disaster Recovery Procedure​

When you need to restore the entire system in a new environment, follow these steps in order:

1. Deploy Base Environment​

# Install BK-Lite (without data initialization)
curl -sSL https://bklite.cn/install.run | bash -

2. Stop Services​

cd /opt/bk-lite
docker compose stop server web

3. Restore Databases​

cd /opt/bk-lite

# Restore PostgreSQL
docker compose exec -T postgres psql -U postgres -c "DROP DATABASE IF EXISTS bklite;"
docker compose exec -T postgres psql -U postgres -c "CREATE DATABASE bklite;"
cat /path/to/bklite_backup.dump | docker compose exec -T postgres pg_restore -U postgres -d bklite

# Restore MLflow database
docker compose exec -T postgres psql -U postgres -c "DROP DATABASE IF EXISTS mlflow;"
docker compose exec -T postgres psql -U postgres -c "CREATE DATABASE mlflow;"
cat /path/to/mlflow_backup.dump | docker compose exec -T postgres pg_restore -U postgres -d mlflow

# Restore PGVector (metis)
docker compose exec -T postgres psql -U postgres -c "DROP DATABASE IF EXISTS metis;"
docker compose exec -T postgres psql -U postgres -c "CREATE DATABASE metis;"
docker compose exec -T postgres psql -U postgres -d metis -c "CREATE EXTENSION IF NOT EXISTS vector;"
cat /path/to/metis_backup.dump | docker compose exec -T postgres pg_restore -U postgres -d metis

4. Restore FalkorDB​

cd /opt/bk-lite

docker compose stop falkordb
docker compose cp /path/to/dump_backup.rdb falkordb:/var/lib/falkordb/data/dump.rdb
docker compose start falkordb

5. Restore VictoriaLogs (Optional)​

cd /opt/bk-lite

docker compose stop victoria-logs

# Get volume name
VOLUME_NAME=$(docker volume ls --filter name=victoria-logs -q | head -1)

# Restore snapshot
docker run --rm -v ${VOLUME_NAME}:/vlogs -v /path/to/backup:/backup alpine \
  sh -c "rm -rf /vlogs/* && cp -r /backup/snapshot_YYYYMMDD/* /vlogs/"

docker compose start victoria-logs

6. Start Services​

cd /opt/bk-lite

docker compose start server web

# Check service status
docker compose ps

7. Verify Recovery​

  • Log in to the Web interface to verify user data
  • Check CMDB asset data
  • Verify knowledge base search functionality
  • Confirm monitoring alerts are working properly

FAQ​

Q: What if backup files are too large?​

A: You can use compression and splitting:

# Compress backup
gzip /data/backup/bklite/postgres/bklite_20240101.sql

# Split and compress (1GB per volume)
split -b 1G /data/backup/bklite/postgres/bklite.dump bklite.dump.part_

Q: How to verify backup file integrity?​

A: Use checksum verification:

# Generate checksum
sha256sum /data/backup/bklite/postgres/bklite_20240101.dump > /data/backup/bklite/postgres/bklite_20240101.dump.sha256

# Verify checksum
sha256sum -c /data/backup/bklite/postgres/bklite_20240101.dump.sha256

Q: Can backups be performed online?​

A: PostgreSQL, FalkorDB, and VictoriaLogs all support online backups without stopping services. However, for data consistency, it is recommended to perform backups during off-peak hours.

Q: How to back up to remote storage?​

A: You can use rsync or rclone:

# Sync to remote server using rsync
rsync -avz /data/backup/bklite/ user@backup-server:/backup/bklite/

# Sync to S3 using rclone
rclone sync /data/backup/bklite/ s3:my-bucket/bklite-backup/

Next Steps​