Skip to content

Cluster Deployment & Operations Runbook

This document is the authoritative, all-in-one runbook for deploying the Distributed Virtual File System (DVFS) across cluster nodes (such as Raspberry Pi boards, workstations, or Ubuntu Linux servers).

Commands are presented first in sequential execution order, followed by conceptual explanations and operational details in Section 8.


1. Base Setup (Execute on Every Node)

Run these commands on every physical or virtual machine hosting any DVFS service:

./connect.sh # replace id and passwd
# Clone the repository
git clone https://github.com/DVFS-IIT-Gandhinagar/Distributed-Virtual-File-System.git
cd Distributed-Virtual-File-System

# Run base environment preparation
chmod +x ./scripts/rp_115/setup.sh
./scripts/rp_115/setup.sh

# Mask hardware sleep/suspend targets to keep nodes online
chmod +x ./scripts/rp_115/persist.sh
./scripts/rp_115/persist.sh

# Install and start the network re-authentication timer
sudo cp scripts/rp_115/fortinet.service scripts/rp_115/fortinet.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now fortinet.timer
sudo timedatectl set-timezone Asia/Kolkata
sudo systemctl enable --now chrony
sudo tee /etc/chrony/chrony.conf > /dev/null <<'EOF'
pool pool.ntp.org iburst
server time.cloudflare.com iburst
server time.google.com iburst

driftfile /var/lib/chrony/chrony.drift
makestep 1.0 3
rtcsync
EOF
sudo systemctl restart chrony
sudo chronyc makestep

2. Certificate Generation & Distribution (Developer Machine Only)

Execute these commands once on your development workstation:

# Step 1: Mint the air-gapped Root Certificate Authority (10-year validity)
go run scripts/gen-certs/cmd/gen_root_ca/main.go

# Step 2: Mint leaf certificates with DNS SANs for all cluster nodes (dvfs1 through dvfs9, localhost, fs1, mds)
go run scripts/gen-certs/cmd/gen_node_certs/main.go

For each cluster node {i} with IP ${ip}, distribute the minted certificates:

# Create certs directory on remote node
ssh dvfs${i}@${ip} "mkdir -p ~/Distributed-Virtual-File-System/certs"

# Copy leaf cert, leaf private key, and public Root CA cert
scp deploy_certs/dvfs${i}/server.crt dvfs${i}@${ip}:~/Distributed-Virtual-File-System/certs/
scp deploy_certs/dvfs${i}/server.key dvfs${i}@${ip}:~/Distributed-Virtual-File-System/certs/
scp deploy_certs/ca.crt             dvfs${i}@${ip}:~/Distributed-Virtual-File-System/certs/

# Restrict private key permissions
ssh dvfs${i}@${ip} "chmod 600 ~/Distributed-Virtual-File-System/certs/server.key && chmod 644 ~/Distributed-Virtual-File-System/certs/*.crt"

Verify certificate chains:

openssl verify -CAfile deploy_certs/ca.crt deploy_certs/dvfs1/server.crt

3. MetaServer Setup (dvfs1 / Coordinator Node)

MongoDB (required before the MetaServer starts)

The MetaServer and the Admin Console keep the cluster's routing state (nodes, user placement, shares) in MongoDB 7+. Both exit at startup if they cannot reach it, and the systemd units are ordered After=mongod.service.

# Install MongoDB 7 Community (Ubuntu). See https://www.mongodb.com/docs/manual/administration/install-on-linux/
curl -fsSL https://www.mongodb.org/static/pgp/server-7.0.asc | sudo gpg --dearmor -o /usr/share/keyrings/mongodb-server-7.0.gpg
echo "deb [signed-by=/usr/share/keyrings/mongodb-server-7.0.gpg] https://repo.mongodb.org/apt/ubuntu $(lsb_release -cs)/mongodb-org/7.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-7.0.list
sudo apt update && sudo apt install -y mongodb-org
sudo systemctl enable --now mongod

A single mongod on the MetaServer host, bound to loopback (the package default), needs nothing more: the shipped MONGO_URI=mongodb://127.0.0.1:27017/dvfs works as is.

If the Admin Console runs on another host, or you run a replica set across dvfs1–dvfs3, mongod has to listen on the network. Share grants and user placement are authorization data, so never expose an unauthenticated mongod:

# 1. Create the application user (once, on the primary)
mongosh --eval 'db.getSiblingDB("admin").createUser({user:"dvfsadmin",pwd:passwordPrompt(),roles:["userAdminAnyDatabase"]})'
mongosh -u dvfsadmin -p --authenticationDatabase admin \
  --eval 'db.getSiblingDB("dvfs").createUser({user:"dvfs",pwd:passwordPrompt(),roles:[{role:"readWrite",db:"dvfs"}]})'

# 2. /etc/mongod.conf: require auth, bind only the cluster interface, TLS when the network is shared
#    security:
#      authorization: enabled
#    net:
#      bindIp: 127.0.0.1,<this node's cluster IP>
#      tls:
#        mode: requireTLS
#        certificateKeyFile: /etc/ssl/mongod.pem
sudo systemctl restart mongod

# 3. Hand the URI to the services through a root-only file, not the unit file or a command line
sudo install -d -m 0750 /etc/dvfs
sudo sh -c 'umask 077; echo "MONGO_URI=mongodb://dvfs:<password>@dvfs1:27017,dvfs2:27017,dvfs3:27017/dvfs?replicaSet=rs0&authSource=dvfs&tls=true" > /etc/dvfs/mongo.env'
# In dvfs-metaserver.service and dvfs-admin.service, replace the Environment=MONGO_URI line with:
#   EnvironmentFile=/etc/dvfs/mongo.env

MetaServer service

On the machine designated as the MetaServer:

# Install and enable the systemd service
sudo cp scripts/dvfs-metaserver.service /etc/systemd/system/
chmod +x scripts/start-metaserver.sh
sudo systemctl daemon-reload
sudo systemctl enable --now dvfs-metaserver

# Verify status
sudo systemctl status dvfs-metaserver --no-pager

4. Admin UI & Orchestration Server Setup

On the coordinator or management server:

# Step 1: Generate an SSH keypair for cluster management
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N ""

# Step 2: Copy SSH public key to each fileserver node
for ip in <node_ips>; do
    ssh-copy-id -i ~/.ssh/id_ed25519.pub user@${ip}
done

# Step 3: Configure administrative password hash (SHA-256)
# On Linux / macOS / WSL:
echo -n "YourSecretPassword" | sha256sum | awk '{print "ADMIN_PASSWORD_HASH="$1}' > .env
chmod 600 .env

# On Windows PowerShell:
# $hash = [System.BitConverter]::ToString([System.Security.Cryptography.SHA256]::Create().ComputeHash([System.Text.Encoding]::UTF8.GetBytes("YourSecretPassword"))).Replace("-","").ToLower()
# "ADMIN_PASSWORD_HASH=$hash" | Out-File -Encoding ascii .env

# Step 4: Build the React SPA frontend
cd cmd/admin/ui
npm install -D @vitejs/plugin-react@^6.1.1 vite@^8.2.2
npm install
npm run build
cd ../../..

# Step 5: Option A - Run as a systemd service
sudo cp scripts/dvfs-admin.service /etc/systemd/system/
chmod +x scripts/start-admin.sh
sudo systemctl daemon-reload
sudo systemctl enable --now dvfs-admin

# Verify status
sudo systemctl status dvfs-admin --no-pager

# Step 5: Option B - Run binary directly (Development / Testing)
# Note: Password authentication uses ADMIN_PASSWORD_HASH loaded from .env
./bin/admin \
  -port=8080 \
  -mongo_uri=mongodb://127.0.0.1:27017/dvfs \
  -static=./cmd/admin/static \
  -ssh_user=dvfs \
  -ssh_key=~/.ssh/id_ed25519 \
  -repo_path=~/Distributed-Virtual-File-System

Open http://<admin_ip>:8080 (or https:// if TLS certs are supplied) in your web browser.


5. Update Gist Discovery Server Setup

On the machine designated to run the Gist IP updater:

Repeat the above SSH Key copying process for this machine!

# Install system packages
sudo apt update && sudo apt install -y python3 python3-pip python3-requests python3-dotenv

# Prepare execution directory
sudo mkdir -p /opt/dvfs
sudo cp scripts/rp_115/update_gist.py /opt/dvfs/
sudo chmod +x /opt/dvfs/update_gist.py

# Create secret configuration file
sudo tee /opt/dvfs/.env > /dev/null << 'EOF'
TAILSCALE_CLIENT_ID=your_tailscale_oauth_client_id
TAILSCALE_CLIENT_SECRET=your_tailscale_oauth_client_secret
GIST_ID=your_github_gist_id
GITHUB_TOKEN=your_github_personal_access_token
EOF

# Lock down permissions and enable hourly timer
sudo chown -R $USER:$USER /opt/dvfs
sudo chmod 600 /opt/dvfs/.env
sudo cp scripts/rp_115/dvfs-gist.service scripts/rp_115/dvfs-gist.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start dvfs-gist.service
sudo systemctl enable --now dvfs-gist.timer

# Verify timer schedule
systemctl list-timers --all | grep dvfs-gist

6. FileServer Node Setup (Each Storage Node)

On every fileserver node (dvfs1 through dvfs9):

# Step 1: Configure scoped passwordless sudoers rules for remote orchestration
sudo bash -c 'cat <<EOF > /etc/sudoers.d/dvfs
# Scoped dvfs administration privileges without password prompt
$SUDO_USER ALL=(ALL) NOPASSWD: /usr/bin/systemctl restart dvfs-*, /usr/bin/systemctl status dvfs-*, /usr/bin/journalctl, /sbin/reboot, /usr/sbin/reboot, /usr/bin/systemctl reboot, /sbin/shutdown, /usr/bin/apt, /usr/bin/apt-get
EOF'
sudo chmod 0440 /etc/sudoers.d/dvfs

# Step 2: Install and start the FileServer service
# FS_ID is the node's identity in the cluster and must be unique: the MetaServer
# refuses a second live fileserver that claims an id already registered from
# another address. start-fileserver.sh derives it from a dvfsN user or host
# name; on any other machine set it explicitly (Environment=FS_ID=fsN in the
# unit, or FS_ID=fsN in the environment) instead of accepting the fs1 fallback.
sudo cp scripts/dvfs-fileserver.service /etc/systemd/system/
chmod +x scripts/start-fileserver.sh
sudo systemctl daemon-reload
sudo systemctl enable --now dvfs-fileserver

# Step 3: Verify 
sudo systemctl status dvfs-fileserver --no-pager

7. Google OAuth 2.0 & Identity Configuration

DVFS supports user-facing Google OAuth 2.0 authentication with RFC 7636 PKCE (Proof Key for Code Exchange) and high-performance Server Session Tokens (SST).

Environment Variables

Configure the following environment variables in .env (or via systemd service overrides):

Variable Required in Prod? Default Description
GOOGLE_CLIENT_ID Yes "" Google OAuth 2.0 Web Application Client ID from Google Cloud Console.
GOOGLE_CLIENT_SECRET Yes "" Google OAuth 2.0 Client Secret for authorization code exchange.
GOOGLE_REDIRECT_URI No http://localhost:38485/logincallback OAuth redirect URI configured in Google Cloud Console. Must match the client loopback callback address.
DVFS_AUTH_MOCK No (Dev only) false When set to true or 1, allows deterministic mock tokens (mock-jwt.<email>.<exp>) for automated tests and offline development. Fails closed in production.
DVFS_ADMIN_EMAIL No "" Google-authenticated email address granted administrative authorization on FileServers (e.g., executing SetQuota without an admin password hash).
ADMIN_PASSWORD_HASH Yes (for Admin UI) "" Hex-encoded SHA-256 hash of the administrative password for the Admin Web Console and dual-mode administrative gRPC operations.
DVFS_ALLOW_LEGACY_TOKEN_FALLBACK No 0 When set to 1, permits transitional fallback to raw Google ID tokens if an SST is missing. Disabled by default for zero-trust compliance.

Example .env configuration:

GOOGLE_CLIENT_ID=1234567890-abcdefgh.apps.googleusercontent.com
GOOGLE_CLIENT_SECRET=GOCSPX-SampleSecretKey123456
GOOGLE_REDIRECT_URI=http://localhost:38485/logincallback
DVFS_AUTH_MOCK=false
DVFS_ADMIN_EMAIL=admin@example.com
ADMIN_PASSWORD_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Build Configurations

By default, DVFS builds with Google Authentication enabled (USE_GOOGLE_AUTH=1):

# Build all components with Google Auth enabled
make build

# Explicit build targets
make build-google-auth   # Builds with -tags use_google_auth
make build-no-auth       # Builds without Google Auth (legacy testing)

# Run test suite with Google Auth enabled
make test-google-auth    # go test -tags use_google_auth ./...

8. Quick Local Development (Single-Machine)

To test the complete DVFS ecosystem locally on a single machine without systemd (using mock auth mode):

# 1. Generate dev certificates (if not already created)
make certs

# 2. Terminal 1: Start MetaServer (with mock auth enabled). Needs a local mongod,
#    or: docker run -d -p 27017:27017 mongo:7
DVFS_AUTH_MOCK=true go run ./cmd/metaserver/main.go -port=50051 -mongo_uri=mongodb://127.0.0.1:27017/dvfs -tls_cert=certs/server.crt -tls_key=certs/server.key

# 3. Terminal 2: Start FileServer (with mock auth enabled)
DVFS_AUTH_MOCK=true go run ./cmd/fileserver/main.go \
  -id=fs1 \
  -port=50052 \
  -data=./fileserver_data \
  -meta_addr=127.0.0.1:50051 \
  -own_ip=127.0.0.1 \
  -meta_retry_interval=3s \
  -meta_heartbeat_interval=5s \
  -tls_cert=certs/server.crt \
  -tls_key=certs/server.key

# 4. Terminal 3: Start Admin Console
go run ./cmd/admin/main.go \
  -port=8080 \
  -mongo_uri=mongodb://127.0.0.1:27017/dvfs \
  -static=./cmd/admin/static

# 5. Terminal 4: Launch Client (using email as identity)
DVFS_AUTH_MOCK=true go run ./cmd/client/main.go -username=alice@example.com -ip_addr=127.0.0.1 -port=50051 -meta=true

9. Client Usage

Running from Source

# Connect via MetaServer with dynamic Gist discovery (prompts for Google email if omitted)
go run ./cmd/client/main.go -username alice@example.com

# Connect directly to a specific FileServer or MetaServer IP
go run ./cmd/client/main.go -username alice@example.com -ip_addr 10.7.52.85 -port 50051

Running Pre-Compiled Standalone Binaries

Download the matching binary from the repository release artifacts (e.g. dvfs-client-windows-amd64.exe or dvfs-client-linux-amd64):

./dvfs-client-linux-amd64 -username alice@example.com

Interactive Login Flow

When starting the client: 1. If -username is not a valid email, the client prompts: Enter your email:. 2. A temporary loopback HTTP listener starts at http://localhost:38485/logincallback. 3. The client opens your system browser to Google's OAuth 2.0 consent page with RFC 7636 PKCE parameters. 4. After authenticating, Google redirects to the loopback listener. 5. In headless environments, the terminal prints the authorization URL and prompts you to paste the authorization code manually. 6. The client exchanges the authorization code for a Google ID token and performs the RegisterClient handshake with the FileServer, acquiring a 256-bit CSPRNG Server Session Token (SST). 7. All subsequent operations seamlessly authenticate using the SST. On client shutdown, UnregisterClient revokes the active session.



10. Architectural & Operational Explanations

Why Hardware Sleep is Masked (persist.sh)

Linux power management daemons routinely place idle cluster machines into sleep or hybrid-sleep states. The persist.sh script executes systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target to guarantee uninterrupted fileserver availability.

Why the Network Timer Exists (fortinet.service / fortinet.timer)

Campus networks (such as IIT Gandhinagar's Fortinet gateway) frequently terminate outbound internet sessions after 24 hours, requiring web captive portal authentication. The fortinet.timer runs on boot and hourly thereafter, invoking connect.sh to extract CSRF tokens and submit credentials headless to https://fwg.iitgn.ac.in, preventing network dropouts.

Why Certificate Authority Keys Remain Air-Gapped

Traditional TLS setups generate the CA key directly on the server. DVFS decouples this completely: scripts/gen-certs/cmd/gen_root_ca/ writes ca.key strictly to the developer's administrative machine. Only signed end-entity leaf certificates (server.crt) and the public Root certificate (ca.crt) are deployed via scp. If any storage node is physically stolen or compromised, the root certificate authority remains secure.

How Dynamic Discovery Bridges Tailscale and Ethernet IPs

Cluster machines run on DHCP where local IP addresses can change upon router reboot. The update_gist.py script uses a dual-network topology: 1. It queries the Tailscale API to find active devices tagged with tag:dvfsmachines. 2. It uses Tailscale's overlay IPs (100.x.y.z) to SSH into each node and read its physical campus Ethernet interface (eno*|enp*|eth*). 3. It posts the mapping of hostname to campus LAN IP directly to a public GitHub Gist (machines.json). 4. DVFS clients read this Gist on startup, dial the active campus LAN IP directly for high-speed local transfer, and supply the node's hostname (dvfs1..dvfs9) in the TLS SNI header.

Why Sudoers is Scoped

The Admin Console features remote cluster orchestration (restarting services, streaming logs, updating packages, and rebooting nodes). To enable automated execution over SSH without prompting for interactive passwords or granting unrestricted root privileges, /etc/sudoers.d/dvfs restricts passwordless execution specifically to /usr/bin/systemctl, /usr/bin/journalctl, /sbin/reboot, /sbin/shutdown, and /usr/bin/apt.

How Admin Authentication Operates

The Admin Console reads ADMIN_PASSWORD_HASH from .env. When an administrator logs in, the backend computes the SHA-256 hash of the submitted password and compares it in constant time via crypto/subtle.ConstantTimeCompare. A cryptographically secure 32-byte session token is generated and stored with a 12-hour expiration, set via an HttpOnly browser cookie (dvfs_admin_token). Privileged operations such as modifying user storage quotas (SetQuota) can be authenticated either through ADMIN_PASSWORD_HASH (sent by the Admin Web Console backend via x-admin-password-hash) or by a Google-authenticated user matching DVFS_ADMIN_EMAIL.


11. Makefile Targets Reference

The root Makefile automates building, testing, code generation, TLS certificate minting, and cross-platform packaging.

Target Description Underlying Command
make build Builds all 4 binaries (fileserver, client, metaserver, admin) into bin/ (defaults to Google Auth enabled). go build -tags use_google_auth -o bin/<binary> cmd/<component>/main.go
make build-google-auth Builds all 4 binaries explicitly with Google Auth enabled. make build USE_GOOGLE_AUTH=1
make build-no-auth Builds binaries without Google Auth (legacy mode for testing). make build USE_GOOGLE_AUTH=0
make test-google-auth Runs test suite explicitly with -tags use_google_auth. go test -tags use_google_auth ./... -count=1 -v
make proto Recompiles all Protocol Buffer .proto schemas into Go structs and gRPC interfaces. protoc --go_out=. --go-grpc_out=. api/...
make clean Removes build artifacts (bin/, fileserver_data/). Cross-platform (PowerShell / rm). rm -rf bin fileserver_data
make deps Downloads and tidies Go module dependencies. go mod download && go mod tidy
make fmt Formats all Go source files. go fmt ./...
make vet Runs go vet static analysis across the entire project. go vet ./...
make test Runs the full automated test suite with verbose output. go test ./... -count=1 -v
make test-client Runs client-focused test suite (internal/client). go test ./internal/client -count=1 -v
make test-edge Runs edge-case test suite (internal/fileserver, internal/metaserver). go test ./internal/fileserver ./internal/metaserver -count=1 -v
make test-admin Runs admin console test suite (internal/admin). go test ./internal/admin -count=1 -v
make test-integration Runs integration and end-to-end test suite (integration). go test ./integration -count=1 -v
make test-cover Runs test suite and outputs coverage profile and function breakdown. go test ./... -coverprofile=coverage.out && go tool cover -func=coverage.out
make certs Generates local development certificates with SANs for localhost and local LAN IP. go run scripts/gen-certs/main.go $(SERVER)
make certs-force Force regenerates local development certificates even if existing ones are present. go run scripts/gen-certs/main.go -force $(SERVER)
make certs-root-ca Mints air-gapped 10-year RSA 4096-bit Root CA (certs/ca.crt, certs/ca.key). go run scripts/gen-certs/cmd/gen_root_ca/main.go
make certs-nodes Mints verified 2-year leaf certificates for dvfs1–dvfs9, localhost, fs1, mds. go run scripts/gen-certs/cmd/gen_node_certs/main.go
make run-server Builds and runs local FileServer (-id=fs1 -port=50051 -data=./fileserver_data). ./bin/fileserver ...
make run-metaserver Builds and runs local MetaServer (-port=50052 -mongo_uri=...; override with MONGO_URI=...). ./bin/metaserver -port=50052 -mongo_uri=mongodb://127.0.0.1:27017/dvfs
make run-admin Builds and runs Admin Console (-port=8080 -mongo_uri=...). ./bin/admin -port=8080 ...
make run-client Builds and runs interactive client (USER=alice IP_ADDR=127.0.0.1). ./bin/client -username=$(USER) -ip_addr=$(IP_ADDR)
make release Cross-compiles client and node packages for all platforms with SHA256 checksums. go run scripts/build-release/main.go
make release-client Builds standalone client archives for Windows, macOS, and Linux (AMD64 & ARM64). go run scripts/build-release/main.go -client-only
make release-nodes Builds cluster node archives for Linux ARM64 (Raspberry Pis) and Linux AMD64. go run scripts/build-release/main.go -nodes-only
make help Displays available Makefile targets and descriptions. echo ...

12. Systemd Service Units & Template Architecture

DVFS provides two styles of systemd unit files in scripts/:

1. Static Unit Files (Auto User Detection)

These unit files dynamically resolve the primary non-root user (UID 1000, e.g., ubuntu, rpi, jsm) or respect DVFS_USER / DVFS_REPO overrides: - scripts/dvfs-metaserver.service: Runs the MetaServer coordinator daemon. - scripts/dvfs-fileserver.service: Runs the FileServer storage daemon and auto-detects advertised IP via scripts/start-fileserver.sh. - scripts/dvfs-admin.service: Runs the centralized Admin Web Console and orchestration server. - scripts/rp_115/fortinet.service & fortinet.timer: Automated IITGN captive portal re-authentication. - scripts/rp_115/dvfs-gist.service & dvfs-gist.timer: Hourly Tailscale-to-Gist IP synchronization.

2. Multi-User Template Units (@.service)

For multi-user environments or systems where explicit user parameterization is required, template units instantiate daemons scoped to a specific Linux username %i: - scripts/dvfs-metaserver@.service: Usage: sudo systemctl enable --now dvfs-metaserver@<username> - scripts/dvfs-fileserver@.service: Usage: sudo systemctl enable --now dvfs-fileserver@<username> - scripts/dvfs-admin@.service: Usage: sudo systemctl enable --now dvfs-admin@<username> - scripts/rp_115/fortinet@.service: Usage: sudo systemctl enable --now fortinet@<username>

Each template unit sets User=%i, WorkingDirectory=%h/Distributed-Virtual-File-System, and resolves state and binary directories relative to the user's home directory (%h).

State File Path Conventions

  • Cluster metadata lives in MongoDB, not on local disk. Both the MetaServer and the Admin Console take -mongo_uri (or the MONGO_URI environment variable) and an optional -mongo_db, which overrides the database named in the URI (the URI's database is used otherwise, and dvfs if it names none).
  • Under systemd execution via scripts/start-metaserver.sh and scripts/start-admin.sh, MONGO_URI defaults to mongodb://127.0.0.1:27017/dvfs. Point it at the replica set in production, e.g. mongodb://dvfs1:27017,dvfs2:27017,dvfs3:27017/dvfs?replicaSet=rs0. A URI with credentials belongs in an EnvironmentFile (see §3), not in the unit file or on a command line, where systemctl show and ps expose it.
  • Because membership is read from the shared database rather than a local file, the Admin Console no longer has to run on the MetaServer host.
  • There is no import path from the old metaserver_state.json. A cluster starts with an empty database and repopulates itself: fileservers re-register on startup (republishing their users and shares from their own on-disk ACLs), and users are re-assigned a home node on their next login.