feat(compose): run sqlite shards in separate containers

Co-Authored-By: GitHub Copilot <noreply@github.com>

CHANGELOG.md

@@ -6,6 +6,7 @@ Notable changes to this project are documented here.
 
 ### Added
 
+- Docker Compose runs the gateway and each shard in its own container, with one persistent SQLite volume per shard.
 - Public clients are view-only: the UI hides editing controls and the server rejects remote writes; localhost clients retain editing access.
 - Expose the Picard actor console over the `/api/actors/ws` WebSocket endpoint.
 - The world can be split into shards that each own a strip of columns. People and animals that walk across a shard border are transferred with their inventory and stay reachable at the same address.
@@ -28,4 +29,3 @@ Notable changes to this project are documented here.
 - Update Picard to `f88b9cb`.
 - The default world is larger: 192×128 tiles with 180 people, 1200 rabbits, 80 wolves, and twice the food and houses.
 - `DATABASE_URL` is replaced by `SHARD_DATABASE_URL`; existing `sim.db` snapshots are not migrated.
-- Docker Compose creates one Postgres database per shard (`sim_shard_0` to `sim_shard_2`); existing `postgres_data` volumes must be recreated with `docker compose down -v`.

Justfile

@@ -2,10 +2,10 @@ default:
     @just --list
 
 run:
-    docker compose down && docker compose up --build -d
+    docker compose down --remove-orphans && docker compose up --build -d --remove-orphans
 
 stop:
-    docker compose down
+    docker compose down --remove-orphans
 
 dev:
     SHARD_DATABASE_URL="sqlite://sim-shard-{shard}.db?mode=rwc" cargo run

README.md

@@ -6,9 +6,9 @@ Every person, rabbit, wolf, and partnership is a Picard actor that owns its stat
 
 ## Cluster
 
-The world is split into vertical strips of columns called shards. Each shard is a separate process with its own Picard actor system and its own database. `sim` starts as a gateway: it spawns `SIM_SHARDS` (default 3) copies of itself with the `shard` subcommand, steps them in lockstep, and merges their views so the UI sees one world. Edits from the UI are routed to the shard that owns them.
+The world is split into vertical strips of columns called shards. Each shard is a separate process with its own Picard actor system and its own database. `sim` starts as a gateway: by default, it spawns `SIM_SHARDS` (default 3) copies of itself with the `shard` subcommand; when `SIM_SHARD_ADDRS` is set, it connects to externally managed shards instead. The gateway steps shards in lockstep and merges their views so the UI sees one world. Edits from the UI are routed to the shard that owns them.
 
-People, animals and carts that cross a shard border are migrated with their whole state and inventory into the neighbouring shard's Picard actor system. A cluster-wide directory maps each entity address to its current shard and actor path, so world messages continue to reach it after migration without actor-owned forwarding timers or transfer messages. Shards connect directly through Picard's authenticated remoting endpoint (`/remoting`); the gateway continues to use `/control` for simulation coordination. The gateway generates a shared cluster token and passes it to its shard processes for peer authentication.
+People, animals and carts that cross a shard border are migrated with their whole state and inventory into the neighbouring shard's Picard actor system. A cluster-wide directory maps each entity address to its current shard and actor path, so world messages continue to reach it after migration without actor-owned forwarding timers or transfer messages. Shards connect directly through Picard's authenticated remoting endpoint (`/remoting`); the gateway continues to use `/control` for simulation coordination. The gateway generates a shared token when it launches shard processes; external shard deployments supply the same `SIM_CLUSTER_TOKEN` to the gateway and every shard.
 
 ## Public access
 
@@ -22,7 +22,7 @@ Every shard founds markets and workplaces: farms, woodcutters, quarries, hunters
 
 The server enables Picard's console by default and exposes its JSON request/response protocol over `ws://127.0.0.1:7878/api/actors/ws`. The connection reaches shard 0; use `?shard=N` to connect to shard `N`. Every shard registers the others as console peers named `shard-N`: `peers` lists them, `@shard-1 actors` forwards a command, and `send @shard-1 PATH JSON` sends a message to a remote actor through Picard remoting. Send one JSON request per text frame; each receives one response. For example, `{"id":1,"op":"actors"}` lists actors, and `{"id":2,"op":"inspect","path":"/user/person-1"}` inspects one (actors are named `person-<id>`, `rabbit-<id>`, `wolf-<id>`, and `partnership-<id>`). Other operations include `messages`, `pause`, `resume`, `clear`, `stop`, `restart`, and `send`. Console requests are handled by the selected shard's Picard console. `cargo run --no-default-features` disables the endpoint; `--features console` enables it explicitly.
 
-The simulation script is the root [config.rn](./config.rn), a single Rune file. Its `pub const` items hold defaults and tuning values: map dimensions and terrain generation, initial population and resources, calendar and life-stage defaults, hunger, gathering, construction, partnerships and births, animal behavior, and edit limits. Its `pub fn person`, `pub fn rabbit`, and `pub fn wolf` items are the agent behaviours, and they can use the constants directly. Every constant the server reads must be defined; missing or invalid values are reported at startup. `SIM_CONFIG` can point to a different Rune file. Docker Compose mounts the repository's `config.rn` read-only into the app container.
+The simulation script is the root [config.rn](./config.rn), a single Rune file. Its `pub const` items hold defaults and tuning values: map dimensions and terrain generation, initial population and resources, calendar and life-stage defaults, hunger, gathering, construction, partnerships and births, animal behavior, and edit limits. Its `pub fn person`, `pub fn rabbit`, and `pub fn wolf` items are the agent behaviours, and they can use the constants directly. Every constant the server reads must be defined; missing or invalid values are reported at startup. `SIM_CONFIG` can point to a different Rune file. Docker Compose mounts the repository's `config.rn` read-only into the gateway and each shard container.
 
 The in-browser script editor edits the same file contents. Applied edits take effect immediately for behaviours and tuning values, while map dimensions, initial population, and calendar defaults apply when the world is reset. Editor changes are not written back to disk: on restart, `config.rn` from disk replaces them, even when a saved world snapshot is restored.
 
@@ -63,29 +63,29 @@ just
 Useful recipes:
 
 ```sh
-just run  # Stop Compose services, then build and start the app with Postgres
+just run  # Stop Compose services, then build and start the gateway and SQLite shards
-just stop # Stop Compose services; keep the database volume
+just stop # Stop Compose services; keep shard data volumes
 just dev  # Run locally with SQLite
 ```
 
-### Run with Docker Compose and Postgres
+### Run with Docker Compose and SQLite shards
 
-Install Docker with the Compose plugin, then run `just run`. This stops any existing Compose stack before rebuilding and starting the app with Postgres. You can also use the equivalent Docker command directly:
+Install Docker with the Compose plugin, then run `just run`. This stops any existing Compose stack, removes obsolete service containers, then starts one gateway container and three shard containers. Each shard stores its snapshot in its own SQLite database and persistent volume. You can also use the equivalent Docker command directly:
 
 ```sh
-docker compose up --build -d
+docker compose up --build -d --remove-orphans
 ```
 
-Open [http://localhost:7878](http://localhost:7878). Postgres creates one database per shard (`sim_shard_0` to `sim_shard_2`) from [docker/postgres-init.sql](./docker/postgres-init.sql) when the volume is first created; add databases there when raising `SIM_SHARDS`, and recreate the volume after changing the script. Compose waits for Postgres to become healthy before starting the app. Postgres data is stored in the `postgres_data` volume and survives container restarts, `just stop`, and `docker compose down`.
+Open [http://localhost:7878](http://localhost:7878). The `shard_0_data`, `shard_1_data`, and `shard_2_data` volumes survive container restarts, `just stop`, and `docker compose down`.
 
-To remove the database volume and all saved simulation data as well:
+Compose now uses SQLite instead of Postgres. Existing Postgres snapshots are not migrated; the old Postgres volume is left untouched, and the SQLite shards start fresh.
+
+To remove all shard volumes and saved simulation data as well:
 
 ```sh
 docker compose down -v
 ```
 
-The Compose file uses development-only database credentials. Change them before exposing the service beyond a trusted local environment.
-
 ## Database Configuration
 
 Each shard selects its database through `SHARD_DATABASE_URL`. `{shard}` in the URL is replaced with the shard index and is required when there is more than one shard:
@@ -101,9 +101,9 @@ For example, run locally against a reachable Postgres server that has the shard
 SHARD_DATABASE_URL='postgres://USER:PASSWORD@localhost:5432/sim_shard_{shard}' cargo run
 ```
 
-Every shard saves its part of the world as a snapshot about once per second and restores it at startup. SQLite is convenient for local testing; Compose configures the app to use Postgres.
+Every shard saves its part of the world as a snapshot about once per second and restores it at startup. Compose uses a separate SQLite database and volume for each shard; Postgres remains available for deployments that configure it explicitly.
 
-`SIM_SHARDS` sets the number of shards (default 3). `SIM_ADDR` sets the gateway's listen address. It defaults to `127.0.0.1:7878`; containers set it to `0.0.0.0:7878`.
+`SIM_SHARDS` sets the number of shards (default 3). `SIM_SHARD_ADDRS` optionally provides comma-separated shard control addresses in index order; set `SIM_CLUSTER_TOKEN` to the same cluster token on the gateway and every external shard. `SIM_ADDR` sets the gateway's listen address. It defaults to `127.0.0.1:7878`; containers set it to `0.0.0.0:7878`.
 
 ## Project Layout
 
@@ -113,7 +113,7 @@ Every shard saves its part of the world as a snapshot about once per second and
 - `src/storage.rs` provides SQLite and Postgres snapshot storage.
 - `config.rn` is the Rune simulation script: `pub const` defaults and tuning values plus `pub fn` behaviours.
 - `ui/` contains the Leptos frontend, compiled to Wasm by `build.rs`.
-- `Dockerfile`, `compose.yaml`, `docker/postgres-init.sql` and `Justfile` provide container and developer workflows.
+- `Dockerfile`, `compose.yaml`, and `Justfile` provide container and developer workflows.
 
 ## Tests
 

compose.yaml

@@ -6,32 +6,65 @@ services:
       SIM_ADDR: 0.0.0.0:7878
       SIM_CONFIG: /etc/sim/config.rn
       SIM_SHARDS: "3"
+      SIM_SHARD_ADDRS: shard-0:9000,shard-1:9000,shard-2:9000
+      SIM_CLUSTER_TOKEN: ${SIM_CLUSTER_TOKEN:-compose-dev-token}
       SIM_TRUST_PROXY_HEADERS: "1"
-      SHARD_DATABASE_URL: postgres://sim:sim@db:5432/sim_shard_{shard}
     volumes:
       - ./config.rn:/etc/sim/config.rn:ro
     ports:
       - "127.0.0.1:7878:7878"
     depends_on:
-      db:
+      shard-0:
-        condition: service_healthy
+        condition: service_started
+      shard-1:
+        condition: service_started
+      shard-2:
+        condition: service_started
     restart: unless-stopped
 
-  db:
+  shard-0:
-    image: postgres:17-alpine
+    build: .
+    command: ["shard", "--index", "0", "--count", "3", "--seed", "42", "--listen", "0.0.0.0:9000"]
     environment:
-      POSTGRES_DB: sim
+      SIM_CONFIG: /etc/sim/config.rn
-      POSTGRES_USER: sim
+      SIM_CLUSTER_TOKEN: ${SIM_CLUSTER_TOKEN:-compose-dev-token}
-      POSTGRES_PASSWORD: sim
+      SIM_REMOTING_BIND: 0.0.0.0:9001
-    healthcheck:
+      SIM_REMOTING_ADVERTISE: ws://shard-0:9001/remoting
-      test: [ "CMD-SHELL", "pg_isready -U sim -d sim" ]
+      SHARD_DATABASE_URL: sqlite:///data/shard-0.db?mode=rwc
-      interval: 5s
-      timeout: 3s
-      retries: 10
     volumes:
-      - postgres_data:/var/lib/postgresql/data
+      - ./config.rn:/etc/sim/config.rn:ro
-      - ./docker/postgres-init.sql:/docker-entrypoint-initdb.d/shards.sql:ro
+      - shard_0_data:/data
+    restart: unless-stopped
+
+  shard-1:
+    build: .
+    command: ["shard", "--index", "1", "--count", "3", "--seed", "42", "--listen", "0.0.0.0:9000"]
+    environment:
+      SIM_CONFIG: /etc/sim/config.rn
+      SIM_CLUSTER_TOKEN: ${SIM_CLUSTER_TOKEN:-compose-dev-token}
+      SIM_REMOTING_BIND: 0.0.0.0:9001
+      SIM_REMOTING_ADVERTISE: ws://shard-1:9001/remoting
+      SHARD_DATABASE_URL: sqlite:///data/shard-1.db?mode=rwc
+    volumes:
+      - ./config.rn:/etc/sim/config.rn:ro
+      - shard_1_data:/data
+    restart: unless-stopped
+
+  shard-2:
+    build: .
+    command: ["shard", "--index", "2", "--count", "3", "--seed", "42", "--listen", "0.0.0.0:9000"]
+    environment:
+      SIM_CONFIG: /etc/sim/config.rn
+      SIM_CLUSTER_TOKEN: ${SIM_CLUSTER_TOKEN:-compose-dev-token}
+      SIM_REMOTING_BIND: 0.0.0.0:9001
+      SIM_REMOTING_ADVERTISE: ws://shard-2:9001/remoting
+      SHARD_DATABASE_URL: sqlite:///data/shard-2.db?mode=rwc
+    volumes:
+      - ./config.rn:/etc/sim/config.rn:ro
+      - shard_2_data:/data
     restart: unless-stopped
 
 volumes:
-  postgres_data:
+  shard_0_data:
+  shard_1_data:
+  shard_2_data:

src/cluster/gateway.rs

@@ -1,4 +1,4 @@
-//! The gateway: starts the shard nodes, steps them in lockstep and presents their parts of the
+//! The gateway: connects to shard nodes, steps them in lockstep and presents their parts of the
 //! world to the UI as one world.
 
 use super::protocol::{Control, ControlReply, Peer};
@@ -39,6 +39,10 @@ pub struct GatewayOptions {
     pub seed: u64,
     /// Database URL of each shard; `{shard}` is replaced by the shard's index.
     pub database_template: String,
+    /// Control addresses of externally managed shard processes, in index order.
+    pub external_shards: Option<Vec<String>>,
+    /// Shared cluster token; generated when shards are managed by this gateway.
+    pub cluster_token: Option<String>,
 }
 
 /// Where the gateway publishes the unified world for the HTTP handlers.
@@ -129,8 +133,20 @@ struct ShardLink {
 impl ShardLink {
     fn connect(index: usize, address: &str) -> Result<Self, String> {
         let url = format!("ws://{address}/control");
-        let (socket, _) = tungstenite::connect(&url)
+        let started = Instant::now();
-            .map_err(|e| format!("failed to connect to shard {index} at {url}: {e}"))?;
+        let (socket, _) = loop {
+            match tungstenite::connect(&url) {
+                Ok(connection) => break connection,
+                Err(_) if started.elapsed() < SHARD_TIMEOUT => {
+                    std::thread::sleep(Duration::from_millis(250));
+                }
+                Err(error) => {
+                    return Err(format!(
+                        "failed to connect to shard {index} at {url}: {error}"
+                    ));
+                }
+            }
+        };
         if let MaybeTlsStream::Plain(stream) = socket.get_ref() {
             let _ = stream.set_nodelay(true);
             let _ = stream.set_read_timeout(Some(SHARD_TIMEOUT));
@@ -220,7 +236,25 @@ struct Cluster {
 
 impl Cluster {
     fn start(options: &GatewayOptions) -> Result<Self, String> {
-        let cluster_token = Uuid::new_v4().to_string();
+        let cluster_token = options
+            .cluster_token
+            .clone()
+            .unwrap_or_else(|| Uuid::new_v4().to_string());
+        let (processes, links) = if let Some(addresses) = &options.external_shards {
+            if addresses.len() != options.shards {
+                return Err(format!(
+                    "SIM_SHARD_ADDRS lists {} shards, but SIM_SHARDS is {}",
+                    addresses.len(),
+                    options.shards
+                ));
+            }
+            let links = addresses
+                .iter()
+                .enumerate()
+                .map(|(index, address)| ShardLink::connect(index, address))
+                .collect::<Result<Vec<_>, _>>()?;
+            (Vec::new(), links)
+        } else {
             let processes = (0..options.shards)
                 .map(|index| ShardProcess::start(index, options, &cluster_token))
                 .collect::<Result<Vec<_>, _>>()?;
@@ -229,6 +263,8 @@ impl Cluster {
                 .enumerate()
                 .map(|(index, process)| ShardLink::connect(index, &process.address))
                 .collect::<Result<Vec<_>, _>>()?;
+            (processes, links)
+        };
         let mut cluster = Self {
             links,
             _processes: processes,

src/main.rs

@@ -278,8 +278,8 @@ fn shard_options(args: &[String], script: SimScript) -> ShardOptions {
     let mut index = None;
     let mut count = None;
     let mut seed = 42;
-    let mut cluster_token = None;
+    let mut cluster_token = std::env::var("SIM_CLUSTER_TOKEN").ok();
-    let mut database_url = None;
+    let mut database_url = std::env::var("SHARD_DATABASE_URL").ok();
     let mut listen = "127.0.0.1:0".to_owned();
     let mut attached = false;
     let mut args = args.iter();
@@ -359,6 +359,28 @@ async fn main() {
         shards == 1 || database_template.contains("{shard}"),
         "SHARD_DATABASE_URL must contain {{shard}} so that every shard gets its own database"
     );
+    let external_shards = std::env::var("SIM_SHARD_ADDRS").ok().map(|addresses| {
+        addresses
+            .split(',')
+            .map(str::trim)
+            .map(str::to_owned)
+            .collect::<Vec<_>>()
+    });
+    if let Some(addresses) = &external_shards {
+        assert_eq!(
+            addresses.len(),
+            shards,
+            "SIM_SHARD_ADDRS must list one control address per shard"
+        );
+        assert!(
+            addresses.iter().all(|address| !address.is_empty()),
+            "SIM_SHARD_ADDRS contains an empty address"
+        );
+        assert!(
+            std::env::var("SIM_CLUSTER_TOKEN").is_ok(),
+            "SIM_CLUSTER_TOKEN is required with SIM_SHARD_ADDRS"
+        );
+    }
     let addr = std::env::var("SIM_ADDR").unwrap_or_else(|_| "127.0.0.1:7878".into());
     let listener = tokio::net::TcpListener::bind(&addr)
         .await
@@ -378,6 +400,8 @@ async fn main() {
             shards,
             seed,
             database_template,
+            external_shards,
+            cluster_token: std::env::var("SIM_CLUSTER_TOKEN").ok(),
         };
         std::thread::Builder::new()
             .name("gateway".into())

src/world.rs

@@ -3514,7 +3514,8 @@ impl World {
                 .filter(|index| *index != self.shard.index)
                 .map(|index| (format!("shard-{index}"), token.to_owned())),
         );
-        let bind = "127.0.0.1:0"
+        let bind = std::env::var("SIM_REMOTING_BIND")
+            .unwrap_or_else(|_| "127.0.0.1:0".into())
             .parse()
             .map_err(|error: std::net::AddrParseError| error.to_string())?;
         let listener = self
@@ -3529,7 +3530,10 @@ impl World {
         let address = listener
             .local_addr()
             .ok_or_else(|| "Picard remoting listener did not report its address".to_owned())?;
-        self.remote_address = Some(format!("ws://{address}/remoting"));
+        self.remote_address = Some(
+            std::env::var("SIM_REMOTING_ADVERTISE")
+                .unwrap_or_else(|_| format!("ws://{address}/remoting")),
+        );
         self.remote_transport = Some(transport);
         self.remote_listener = Some(listener);
         self.router.register_migration_handler();