IAmHardliner
/
sim
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat(cluster): route entity messages by actor path 

Co-Authored-By: GitHub Copilot <noreply@github.com>
This commit is contained in:
Steve Biedermann 2026-10-13 16:33:23 +02:00
parent 0ffb5dadb1
commit a9055b8218
12 changed files with 464 additions and 819 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
Cargo.lock generated
View File

@ -1939,7 +1939,7 @@ checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
[[package]]
name = "picard"
version = "0.4.2"
source = "git+https://github.com/picard-rs/picard?rev=f88b9cbf078512f726c4c47dbce842fda878388e#f88b9cbf078512f726c4c47dbce842fda878388e"
source = "git+https://github.com/picard-rs/picard?rev=85b51c3da6bde1a3bfa3d3e9043578ce90f4a6c3#85b51c3da6bde1a3bfa3d3e9043578ce90f4a6c3"
dependencies = [
"better_default",
"chrono",
@ -1968,7 +1968,7 @@ dependencies = [
[[package]]
name = "picard-macros"
version = "0.2.0"
source = "git+https://github.com/picard-rs/picard?rev=f88b9cbf078512f726c4c47dbce842fda878388e#f88b9cbf078512f726c4c47dbce842fda878388e"
source = "git+https://github.com/picard-rs/picard?rev=85b51c3da6bde1a3bfa3d3e9043578ce90f4a6c3#85b51c3da6bde1a3bfa3d3e9043578ce90f4a6c3"
dependencies = [
"proc-macro2",
"quote",

2
Cargo.toml
View File

@ -25,7 +25,7 @@ serde = { version = "1", features = ["derive"] }
serde_json = "1"
sqlx = { version = "0.8", default-features = false, features = ["runtime-tokio-rustls", "any", "sqlite", "postgres"] }
rand = "0.8"
picard = { git = "https://github.com/picard-rs/picard", rev = "f88b9cbf078512f726c4c47dbce842fda878388e", features = [
picard = { git = "https://github.com/picard-rs/picard", rev = "85b51c3da6bde1a3bfa3d3e9043578ce90f4a6c3", features = [
"remoting",
"stats",
"tracing",

4
README.md
View File

@ -8,7 +8,7 @@ Every person, rabbit, wolf, and partnership is a Picard actor that owns its stat
The world is split into vertical strips of columns called shards. Each shard is a separate process with its own Picard actor system and its own database. `sim` starts as a gateway: it spawns `SIM_SHARDS` (default 3) copies of itself with the `shard` subcommand, steps them in lockstep, and merges their views so the UI sees one world. Edits from the UI are routed to the shard that owns them.
People, animals and carts that cross a shard border are transferred with their whole state and inventory to the neighbouring shard's actor system. They keep their address, and messages to them are forwarded transparently, so actors never know which shard they run on. Shards talk to each other directly over WebSockets (`/remote` for actor messages and transfers, `/control` for the gateway protocol).
People, animals and carts that cross a shard border are migrated with their whole state and inventory into the neighbouring shard's Picard actor system. A cluster-wide directory maps each entity address to its current shard and actor path, so world messages continue to reach it after migration without actor-owned forwarding timers or transfer messages. Shards connect directly through Picard's authenticated remoting endpoint (`/remoting`); the gateway continues to use `/control` for simulation coordination. The gateway generates a shared cluster token and passes it to its shard processes for peer authentication.
## Economy
@ -16,7 +16,7 @@ Every shard founds markets and workplaces: farms, woodcutters, quarries, hunters
## Actor Console
The server enables Picard's console by default and exposes its JSON request/response protocol over `ws://127.0.0.1:7878/api/actors/ws`. The connection reaches shard 0; use `?shard=N` to connect to shard `N`. Every shard registers the others as console peers named `shard-N`: `peers` lists them, `@shard-1 actors` forwards a command, and `send @shard-1 PATH JSON` sends a message to a remote actor. Send one JSON request per text frame; each receives one response. For example, `{"id":1,"op":"actors"}` lists actors, and `{"id":2,"op":"inspect","path":"/user/person-1"}` inspects one (actors are named `person-<id>`, `rabbit-<id>`, `wolf-<id>`, and `partnership-<id>`). Other operations include `messages`, `pause`, `resume`, `clear`, `stop`, `restart`, and `send`. Console requests are answered by the shard's actor system without waiting for the simulation thread. `cargo run --no-default-features` disables the endpoint; `--features console` enables it explicitly.
The server enables Picard's console by default and exposes its JSON request/response protocol over `ws://127.0.0.1:7878/api/actors/ws`. The connection reaches shard 0; use `?shard=N` to connect to shard `N`. Every shard registers the others as console peers named `shard-N`: `peers` lists them, `@shard-1 actors` forwards a command, and `send @shard-1 PATH JSON` sends a message to a remote actor through Picard remoting. Send one JSON request per text frame; each receives one response. For example, `{"id":1,"op":"actors"}` lists actors, and `{"id":2,"op":"inspect","path":"/user/person-1"}` inspects one (actors are named `person-<id>`, `rabbit-<id>`, `wolf-<id>`, and `partnership-<id>`). Other operations include `messages`, `pause`, `resume`, `clear`, `stop`, `restart`, and `send`. Console requests are handled by the selected shard's Picard console. `cargo run --no-default-features` disables the endpoint; `--features console` enables it explicitly.
The simulation script is the root [config.rn](./config.rn), a single Rune file. Its `pub const` items hold defaults and tuning values: map dimensions and terrain generation, initial population and resources, calendar and life-stage defaults, hunger, gathering, construction, partnerships and births, animal behavior, and edit limits. Its `pub fn person`, `pub fn rabbit`, and `pub fn wolf` items are the agent behaviours, and they can use the constants directly. Every constant the server reads must be defined; missing or invalid values are reported at startup. `SIM_CONFIG` can point to a different Rune file. Docker Compose mounts the repository's `config.rn` read-only into the app container.

45
src/cluster/gateway.rs
View File

@ -71,7 +71,7 @@ struct ShardProcess {
}
impl ShardProcess {
fn start(index: usize, options: &GatewayOptions) -> Result<Self, String> {
fn start(index: usize, options: &GatewayOptions, cluster_token: &str) -> Result<Self, String> {
let executable = std::env::current_exe().map_err(|e| e.to_string())?;
let database = options
.database_template
@ -81,6 +81,7 @@ impl ShardProcess {
.args(["--index", &index.to_string()])
.args(["--count", &options.shards.to_string()])
.args(["--seed", &options.seed.to_string()])
.args(["--cluster-token", cluster_token])
.args(["--database", &database])
.stdin(Stdio::piped())
.stdout(Stdio::piped())
@ -174,6 +175,7 @@ impl ShardLink {
struct Ready {
system_id: Uuid,
remote_address: String,
seed: u64,
tick: u64,
restored: bool,
@ -183,11 +185,13 @@ fn ready(reply: ControlReply) -> Result<Ready, String> {
match reply {
ControlReply::Ready {
system_id,
remote_address,
seed,
tick,
restored,
} => Ok(Ready {
system_id,
remote_address,
seed,
tick,
restored,
@ -202,10 +206,11 @@ fn unexpected(reply: &ControlReply) -> String {
struct Cluster {
links: Vec<ShardLink>,
processes: Vec<ShardProcess>,
_processes: Vec<ShardProcess>,
seed: u64,
cluster_token: String,
/// Directory changes to publish with the next step.
directory: Vec<(EntityKey, Option<usize>)>,
directory: Vec<(EntityKey, Option<(usize, String)>)>,
board: Vec<Quote>,
paused: bool,
speed: u64,
@ -215,8 +220,9 @@ struct Cluster {
impl Cluster {
fn start(options: &GatewayOptions) -> Result<Self, String> {
let cluster_token = Uuid::new_v4().to_string();
let processes = (0..options.shards)
.map(|index| ShardProcess::start(index, options))
.map(|index| ShardProcess::start(index, options, &cluster_token))
.collect::<Result<Vec<_>, _>>()?;
let links = processes
.iter()
@ -225,8 +231,9 @@ impl Cluster {
.collect::<Result<Vec<_>, _>>()?;
let mut cluster = Self {
links,
processes,
_processes: processes,
seed: options.seed,
cluster_token,
directory: Vec::new(),
board: Vec::new(),
paused: false,
@ -288,9 +295,9 @@ impl Cluster {
fn connect_peers(&mut self, readies: &[Ready]) -> Result<(), String> {
let peers: Vec<Peer> = readies
.iter()
.zip(&self.processes)
.map(|(ready, process)| Peer {
address: process.address.clone(),
.map(|ready| Peer {
address: ready.remote_address.clone(),
token: self.cluster_token.clone(),
system_id: ready.system_id,
})
.collect();
@ -300,8 +307,10 @@ impl Cluster {
let ControlReply::Roster(keys) = reply else {
return Err(unexpected(&reply));
};
self.directory
.extend(keys.into_iter().map(|key| (key, Some(index))));
self.directory.extend(
keys.into_iter()
.map(|(key, path)| (key, Some((index, path)))),
);
}
self.board.clear();
Ok(())
@ -320,10 +329,18 @@ impl Cluster {
return Err(unexpected(&reply));
};
self.board.extend(quotes);
self.directory
.extend(report.moved.into_iter().map(|(key, to)| (key, Some(to))));
self.directory
.extend(report.arrived.into_iter().map(|key| (key, Some(index))));
self.directory.extend(
report
.moved
.into_iter()
.map(|(key, to, path)| (key, Some((to, path)))),
);
self.directory.extend(
report
.arrived
.into_iter()
.map(|(key, path)| (key, Some((index, path)))),
);
gone.extend(report.left.into_iter().map(|key| (key, None)));
}
// Departures last, so that an entity that moved and died in the same step stays gone.

13
src/cluster/protocol.rs
View File

@ -4,12 +4,14 @@ use crate::world::{EntityKey, Quote, StepReport};
use serde::{Deserialize, Serialize};
use uuid::Uuid;
/// Where a shard node accepts entity messages from the other shards.
/// How another shard can authenticate and connect to this shard's Picard remoting listener.
#[derive(Clone, Debug, Deserialize, Serialize, PartialEq, Eq)]
pub struct Peer {
/// The `host:port` of the shard node, which serves `/remote` and `/console`.
/// Picard remote endpoint.
pub address: String,
/// The current actor system of the shard, which remote requests are addressed to.
/// Shared authentication token for this cluster.
pub token: String,
/// The current actor system of the shard.
pub system_id: Uuid,
}
@ -21,7 +23,7 @@ pub enum Control {
Peers(Vec<Peer>),
/// Runs one step after applying the gateway's directory changes and market board.
Step {
directory: Vec<(EntityKey, Option<usize>)>,
directory: Vec<(EntityKey, Option<(usize, String)>)>,
board: Vec<Quote>,
},
/// Asks for the shard's part of the world, and for the terrain when `terrain` is set.
@ -44,6 +46,7 @@ pub enum Control {
pub enum ControlReply {
Ready {
system_id: Uuid,
remote_address: String,
seed: u64,
tick: u64,
restored: bool,
@ -57,7 +60,7 @@ pub enum ControlReply {
state: String,
terrain: Option<String>,
},
Roster(Vec<EntityKey>),
Roster(Vec<(EntityKey, String)>),
Script(Result<(), String>),
Console(String),
Failed(String),

137
src/cluster/shard.rs
View File

@ -1,17 +1,14 @@
//! A shard node: simulates one strip of the world, answers the gateway on `/control` and takes
//! entity messages from the other shards on `/remote`.
//! A shard node: simulates one strip of the world, answers the gateway on `/control`, and accepts
//! actor messages and migrations from other shards through Picard remoting.
use super::protocol::{Control, ControlReply, Peer};
use super::transport::{self, ClusterTransport};
use crate::storage::Storage;
use crate::world::{Router, Shard, SimScript, World};
use crate::world::{Shard, SimScript, World};
use axum::extract::State;
use axum::extract::ws::{Message, WebSocket, WebSocketUpgrade};
use axum::response::Response;
use axum::routing::get;
use std::io::Write;
use std::sync::{Arc, RwLock};
use std::time::Duration;
use tokio::sync::{mpsc, oneshot, watch};
/// The line a shard prints to stdout once it accepts connections, followed by its address.
@ -20,6 +17,7 @@ pub const LISTENING: &str = "SHARD_LISTENING";
pub struct ShardOptions {
pub shard: Shard,
pub seed: u64,
pub cluster_token: String,
pub database_url: String,
pub listen: String,
pub script: SimScript,
@ -28,38 +26,17 @@ pub struct ShardOptions {
}
type Command = (Control, oneshot::Sender<ControlReply>);
type CurrentRouter = Arc<RwLock<Option<Arc<Router>>>>;
#[cfg(feature = "console")]
type CurrentConsole = Arc<RwLock<Option<picard::console::Console>>>;
/// What the request handlers need from the current world, which changes on restart.
#[derive(Clone, Default)]
struct Current {
router: CurrentRouter,
#[cfg(feature = "console")]
console: CurrentConsole,
}
impl Current {
fn publish(&self, world: &World) {
*self.router.write().expect("router lock") = Some(Arc::clone(world.router()));
#[cfg(feature = "console")]
{
*self.console.write().expect("console lock") = Some(world.console());
}
}
}
#[derive(Clone)]
struct Node {
commands: mpsc::UnboundedSender<Command>,
current: Current,
}
pub async fn run(options: ShardOptions) {
let ShardOptions {
shard,
seed,
cluster_token,
database_url,
listen,
script,
@ -78,16 +55,16 @@ pub async fn run(options: ShardOptions) {
let (saves, snapshots) = watch::channel(None::<String>);
tokio::spawn(save_snapshots(storage, snapshots));
let current = Current::default();
let (commands, receiver) = mpsc::unbounded_channel();
{
let current = current.clone();
std::thread::Builder::new()
.name(format!("shard-{}", shard.index))
.spawn(move || {
let (world, restored) = open_world(saved.as_deref(), seed, script, shard);
current.publish(&world);
run_world(world, restored, receiver, &current, &saves);
let (mut world, restored) = open_world(saved.as_deref(), seed, script, shard);
world
.start_remoting(&cluster_token)
.unwrap_or_else(|error| panic!("failed to start shard remoting: {error}"));
run_world(world, restored, receiver, &saves, &cluster_token);
})
.expect("failed to start the simulation thread");
}
@ -98,12 +75,8 @@ pub async fn run(options: ShardOptions) {
});
}
let app = axum::Router::new()
.route("/control", get(control))
.route("/remote", get(remote));
#[cfg(feature = "console")]
let app = app.route("/console", get(console_socket));
let app = app.with_state(Node { commands, current });
let app = axum::Router::new().route("/control", get(control));
let app = app.with_state(Node { commands });
let listener = tokio::net::TcpListener::bind(&listen)
.await
.unwrap_or_else(|e| panic!("shard {} failed to bind {listen}: {e}", shard.index));
@ -149,8 +122,8 @@ fn run_world(
mut world: World,
mut restored: bool,
mut commands: mpsc::UnboundedReceiver<Command>,
current: &Current,
saves: &watch::Sender<Option<String>>,
cluster_token: &str,
) {
while let Some((control, reply)) = commands.blocking_recv() {
let answer = match control {
@ -191,8 +164,10 @@ fn run_world(
Control::Script(source) => ControlReply::Script(world.set_script(&source)),
Control::Restart { seed } => {
world = world.restarted(seed);
world
.start_remoting(cluster_token)
.unwrap_or_else(|error| panic!("failed to restart shard remoting: {error}"));
restored = false;
current.publish(&world);
ready(&world, restored)
}
Control::Console(request) => console(&world, &request),
@ -204,6 +179,9 @@ fn run_world(
fn ready(world: &World, restored: bool) -> ControlReply {
ControlReply::Ready {
system_id: world.router().system_id(),
remote_address: world
.remote_address()
.expect("the shard remote listener is started"),
seed: world.seed(),
tick: world.tick(),
restored,
@ -211,25 +189,9 @@ fn ready(world: &World, restored: bool) -> ControlReply {
}
fn connect(world: &World, peers: Vec<Peer>) {
let this = world.shard().index;
let transport = ClusterTransport::new(this, peers.clone());
world.router().set_remote(Arc::clone(&transport) as _);
#[cfg(feature = "console")]
for (index, peer) in peers
.into_iter()
.enumerate()
.filter(|&(index, _)| index != this)
{
let transport = Arc::clone(&transport);
world.register_console_peer(
&format!("shard-{index}"),
Arc::new(move |path, message| {
crate::world::RemoteSender::send(&*transport, index, path, message);
Ok(())
}),
Arc::new(move |request| transport::console_query(&peer.address, request)),
);
}
world
.connect_peers(peers)
.unwrap_or_else(|error| panic!("failed to connect shard peers: {error}"));
}
#[cfg(feature = "console")]
@ -273,58 +235,3 @@ async fn serve_control(mut socket: WebSocket, node: Node) {
}
}
}
async fn remote(upgrade: WebSocketUpgrade, State(node): State<Node>) -> Response {
upgrade
.max_message_size(1 << 30)
.on_upgrade(move |socket| serve_remote(socket, node))
}
async fn serve_remote(mut socket: WebSocket, node: Node) {
while let Some(Ok(message)) = socket.recv().await {
let Message::Text(text) = message else {
continue;
};
let router = node.current.router.read().expect("router lock").clone();
let Some(router) = router else {
return;
};
let answer = tokio::task::spawn_blocking(move || {
transport::receive(&router, "shard", text.as_str())
});
let Ok(Ok(frame)) = tokio::time::timeout(Duration::from_secs(30), answer).await else {
return;
};
if socket.send(Message::Text(frame.into())).await.is_err() {
return;
}
}
}
#[cfg(feature = "console")]
async fn console_socket(upgrade: WebSocketUpgrade, State(node): State<Node>) -> Response {
upgrade.on_upgrade(move |socket| serve_console(socket, node))
}
/// Answers console requests forwarded by another shard's console; it never waits for the
/// simulation thread, which may itself be waiting for this answer.
#[cfg(feature = "console")]
async fn serve_console(mut socket: WebSocket, node: Node) {
while let Some(Ok(message)) = socket.recv().await {
let Message::Text(text) = message else {
continue;
};
let console = node.current.console.read().expect("console lock").clone();
let Some(console) = console else {
return;
};
let Ok(response) =
tokio::task::spawn_blocking(move || console.handle_text(text.as_str())).await
else {
return;
};
if socket.send(Message::Text(response.into())).await.is_err() {
return;
}
}
}

304
src/cluster/transport.rs
View File

@ -1,255 +1,85 @@
//! Delivers entity messages between shard nodes over persistent WebSockets.
//!
//! Every peer gets one connection and one worker thread. Messages are queued in order and sent
//! in batches; the receiving shard answers every batch with one response per message, so
//! [`RemoteSender::flush`] can wait until everything sent so far has reached its mailbox.
//! Adapts Picard's authenticated cluster transport to the world's location router.
use super::protocol::Peer;
use crate::world::{RemoteSender, Router};
use picard::remoting::{ActorAddress, RemoteRequest, RemoteResponse};
use std::collections::VecDeque;
use std::net::TcpStream;
use std::sync::{Arc, Condvar, Mutex};
use std::thread::JoinHandle;
use std::time::{Duration, Instant};
use tungstenite::stream::MaybeTlsStream;
use tungstenite::{Message, WebSocket};
use crate::world::RemoteSender;
use picard::actors::ActorSystem;
use picard::remoting::{
ActorAddress, ClusterTransport as PicardClusterTransport, RemoteConnection, RemotePeer,
};
use serde_json::Value;
use std::collections::HashMap;
use std::sync::Arc;
use uuid::Uuid;
const MAX_BATCH: usize = 512;
const FLUSH_TIMEOUT: Duration = Duration::from_secs(10);
const CONNECT_ATTEMPTS: u32 = 50;
#[derive(Default)]
struct Queue {
waiting: VecDeque<RemoteRequest>,
/// Messages queued or sent but not yet answered.
unanswered: usize,
closed: bool,
#[derive(Clone)]
pub(crate) struct PeerConnection {
pub system_id: Uuid,
pub endpoint: String,
pub connection: RemoteConnection,
}
struct Link {
queue: Mutex<Queue>,
changed: Condvar,
}
impl Link {
fn lock(&self) -> std::sync::MutexGuard<'_, Queue> {
self.queue.lock().expect("transport queue lock")
}
}
struct Connection {
peer: Peer,
link: Arc<Link>,
worker: Option<JoinHandle<()>>,
}
pub struct ClusterTransport {
connections: Vec<Option<Connection>>,
pub(crate) struct ClusterTransport {
peers: HashMap<usize, PeerConnection>,
}
impl ClusterTransport {
/// Connects lazily to every peer except `this` shard.
pub fn new(this: usize, peers: Vec<Peer>) -> Arc<Self> {
let connections = peers
.into_iter()
.enumerate()
.map(|(index, peer)| {
(index != this).then(|| {
let link = Arc::new(Link {
queue: Mutex::default(),
changed: Condvar::new(),
});
let worker = {
let (link, url) =
(Arc::clone(&link), format!("ws://{}/remote", peer.address));
std::thread::Builder::new()
.name(format!("shard-link-{index}"))
.spawn(move || run_link(&url, &link))
.expect("failed to start a shard link")
};
Connection {
peer,
link,
worker: Some(worker),
}
})
})
.collect();
Arc::new(Self { connections })
pub fn connect(
system: &ActorSystem,
this: usize,
peers: Vec<Peer>,
transport: Arc<PicardClusterTransport>,
) -> Result<Arc<Self>, String> {
let mut connections = HashMap::new();
for (index, peer) in peers.into_iter().enumerate() {
if index == this {
continue;
}
let name = format!("shard-{index}");
let remote = RemotePeer::new(name.clone(), peer.address.clone(), peer.token)
.map_err(|error| error.to_string())?;
let connection = system.connect_remote_with(remote, transport.clone());
#[cfg(feature = "console")]
system.register_console_remote_peer(name, connection.clone());
connections.insert(
index,
PeerConnection {
system_id: peer.system_id,
endpoint: peer.address,
connection,
},
);
}
Ok(Arc::new(Self { peers: connections }))
}
pub fn peers(&self) -> HashMap<usize, PeerConnection> {
self.peers.clone()
}
}
impl RemoteSender for ClusterTransport {
fn send(&self, shard: usize, path: &str, payload: serde_json::Value) {
let Some(Some(connection)) = self.connections.get(shard) else {
fn send(&self, shard: usize, path: &str, payload: Value) {
let Some(peer) = self.peers.get(&shard) else {
tracing::warn!(shard, path, "dropped a message for an unknown shard");
return;
};
let destination = ActorAddress::local(connection.peer.system_id, path);
let mut queue = connection.link.lock();
queue
.waiting
.push_back(RemoteRequest::new(destination, None, payload));
queue.unanswered += 1;
connection.link.changed.notify_all();
}
fn flush(&self) {
let deadline = Instant::now() + FLUSH_TIMEOUT;
for (shard, connection) in self.connections.iter().enumerate() {
let Some(connection) = connection else {
continue;
};
let mut queue = connection.link.lock();
while queue.unanswered > 0 && !queue.closed {
let left = deadline.saturating_duration_since(Instant::now());
if left.is_zero() {
tracing::warn!(
shard,
unanswered = queue.unanswered,
"shard flush timed out"
);
break;
}
queue = connection
.link
.changed
.wait_timeout(queue, left)
.expect("transport queue lock")
.0;
}
}
}
}
impl Drop for ClusterTransport {
fn drop(&mut self) {
for connection in self.connections.iter_mut().flatten() {
connection.link.lock().closed = true;
connection.link.changed.notify_all();
if let Some(worker) = connection.worker.take() {
let _ = worker.join();
}
}
}
}
type Socket = WebSocket<MaybeTlsStream<TcpStream>>;
fn connect(url: &str) -> Option<Socket> {
for attempt in 0..CONNECT_ATTEMPTS {
match tungstenite::connect(url) {
Ok((socket, _)) => {
if let MaybeTlsStream::Plain(stream) = socket.get_ref() {
let _ = stream.set_nodelay(true);
}
return Some(socket);
}
Err(error) if attempt + 1 == CONNECT_ATTEMPTS => {
tracing::error!(url, %error, "failed to connect to a shard");
}
Err(_) => std::thread::sleep(Duration::from_millis(100)),
}
}
None
}
/// Sends queued messages to one peer until the transport is dropped.
fn run_link(url: &str, link: &Link) {
let mut socket: Option<Socket> = None;
loop {
let batch: Vec<RemoteRequest> = {
let mut queue = link.lock();
while queue.waiting.is_empty() && !queue.closed {
queue = link.changed.wait(queue).expect("transport queue lock");
}
if queue.closed {
if let Some(mut socket) = socket {
let _ = socket.close(None);
}
return;
}
let count = queue.waiting.len().min(MAX_BATCH);
queue.waiting.drain(..count).collect()
};
let sent = batch.len();
if socket.is_none() {
socket = connect(url);
}
let delivered = socket
.as_mut()
.is_some_and(|socket| exchange(socket, &batch));
if !delivered {
tracing::warn!(url, dropped = sent, "lost messages to a shard");
socket = None;
}
let mut queue = link.lock();
queue.unanswered = queue.unanswered.saturating_sub(sent);
link.changed.notify_all();
}
}
fn exchange(socket: &mut Socket, batch: &[RemoteRequest]) -> bool {
let Ok(frame) = serde_json::to_string(batch) else {
return false;
};
if socket.send(Message::Text(frame.into())).is_err() {
return false;
}
loop {
match socket.read() {
Ok(Message::Text(text)) => {
let Ok(responses) = serde_json::from_str::<Vec<RemoteResponse>>(&text) else {
return false;
};
for response in responses.iter().filter(|response| !response.accepted) {
tracing::debug!(error = ?response.error, "a shard rejected a message");
}
return true;
}
Ok(Message::Ping(_) | Message::Pong(_)) => {}
_ => return false,
}
}
}
/// Answers one batch of messages from another shard.
pub fn receive(router: &Router, peer: &str, frame: &str) -> String {
let responses: Vec<RemoteResponse> = match serde_json::from_str::<Vec<RemoteRequest>>(frame) {
Ok(requests) => requests
.into_iter()
.map(|request| router.handle(peer, request))
.collect(),
Err(error) => {
tracing::warn!(%error, "received an unreadable batch from a shard");
Vec::new()
}
};
serde_json::to_string(&responses).unwrap_or_else(|_| "[]".into())
}
/// Runs one Picard console request on the shard at `address` and returns its JSON response.
#[cfg(feature = "console")]
pub fn console_query(address: &str, request: &str) -> Result<String, String> {
let url = format!("ws://{address}/console");
let (mut socket, _) =
tungstenite::connect(&url).map_err(|e| format!("cannot reach {url}: {e}"))?;
if let MaybeTlsStream::Plain(stream) = socket.get_ref() {
let _ = stream.set_read_timeout(Some(FLUSH_TIMEOUT));
}
socket
.send(Message::Text(request.to_owned().into()))
.map_err(|e| e.to_string())?;
loop {
match socket.read() {
Ok(Message::Text(text)) => {
let _ = socket.close(None);
return Ok(text.as_str().to_owned());
}
Ok(Message::Ping(_) | Message::Pong(_)) => {}
Ok(_) => return Err("the shard closed the console".into()),
Err(error) => return Err(error.to_string()),
let destination = ActorAddress::remote(peer.system_id, peer.endpoint.clone(), path);
match picard::remoting::block_on(peer.connection.send(destination, None, payload)) {
Ok(response) if response.accepted => {}
Ok(response) => tracing::warn!(
shard,
path,
error = ?response.error,
"Picard rejected a remote actor message"
),
Err(error) => tracing::warn!(
shard,
path,
%error,
"failed to send a remote actor message"
),
}
}
fn flush(&self) {}
}

3
src/main.rs
View File

@ -229,6 +229,7 @@ fn shard_options(args: &[String], script: SimScript) -> ShardOptions {
let mut index = None;
let mut count = None;
let mut seed = 42;
let mut cluster_token = None;
let mut database_url = None;
let mut listen = "127.0.0.1:0".to_owned();
let mut attached = false;
@ -254,6 +255,7 @@ fn shard_options(args: &[String], script: SimScript) -> ShardOptions {
.parse()
.unwrap_or_else(|_| panic!("invalid seed: {value}"))
}
"--cluster-token" => cluster_token = Some(value.clone()),
"--database" => database_url = Some(value.clone()),
"--listen" => listen.clone_from(value),
_ => panic!("unknown shard option {flag}"),
@ -267,6 +269,7 @@ fn shard_options(args: &[String], script: SimScript) -> ShardOptions {
ShardOptions {
shard,
seed,
cluster_token: cluster_token.unwrap_or_else(|| uuid::Uuid::new_v4().to_string()),
database_url: database_url.expect("--database is required"),
listen,
script,

318
src/world.rs
View File

@ -9,8 +9,8 @@ use actors::{
};
pub use economy::Quote;
use economy::{Cart, CartMsg, Job, Market, MarketMsg, Workplace, WorkplaceMsg};
use picard::actors::{ActorRef, ActorSystem, Sender, Tell};
use router::{Arrival, LocalActor};
use picard::actors::{ActorRef, ActorReference, ActorSystem, Sender, Tell};
use router::{Admission, Arrival, LocalActor};
pub use router::{EntityKey, Kind, RemoteSender, Router, Shard};
use rune::runtime::RuntimeContext;
use rune::termcolor::Buffer;
@ -1134,24 +1134,13 @@ impl Species {
}
}
/// How long a departed entity's actor keeps forwarding messages to the shard it moved to.
const FORWARDING_TICKS: u64 = 100;
/// An entity that moved to another shard and whose old actor still forwards its messages.
struct Departure {
key: EntityKey,
shard: usize,
tick: u64,
actor: LocalActor,
}
/// What changed in a shard during its steps, for the gateway to maintain the cluster directory.
#[derive(Clone, Debug, Default, Deserialize, Serialize)]
pub struct StepReport {
/// Entities handed to another shard.
pub moved: Vec<(EntityKey, usize)>,
pub moved: Vec<(EntityKey, usize, String)>,
/// Entities that were born on, spawned on or moved into this shard.
pub arrived: Vec<EntityKey>,
pub arrived: Vec<(EntityKey, String)>,
/// Entities that died or were removed.
pub left: Vec<EntityKey>,
}
@ -1211,9 +1200,11 @@ pub struct World {
next_partnership_id: u32,
mailbox: Mailbox,
actor_system: ActorSystem,
remote_transport: Option<Arc<picard::remoting::ClusterTransport>>,
remote_listener: Option<Box<dyn picard::remoting::RemoteListener>>,
remote_address: Option<String>,
shard: Shard,
router: Arc<Router>,
departed: Vec<Departure>,
report: StepReport,
paths: PathScratch,
houses: Vec<(usize, usize)>,
@ -1493,8 +1484,10 @@ impl World {
mailbox: Mailbox::new(&actor_system),
router: Router::new(shard, &actor_system),
actor_system,
remote_transport: None,
remote_listener: None,
remote_address: None,
shard,
departed: Vec::new(),
report: StepReport::default(),
paths: PathScratch::default(),
houses: vec![],
@ -1593,8 +1586,10 @@ impl World {
/// Records an entity appearing on this shard, unless it is the only one.
fn report_arrival(&mut self, key: EntityKey) {
if self.shard.count > 1 {
self.report.arrived.push(key);
if self.shard.count > 1
&& let Some(path) = self.router.actor_path(key)
{
self.report.arrived.push((key, path));
}
}
@ -1716,46 +1711,31 @@ impl World {
EntityKey::new(Kind::Cart, id)
}
};
self.departed.retain(|departure| departure.key != key);
self.report_arrival(key);
}
}
/// Stops the actors of entities that left long enough ago for every shard to know where
/// they went; the router keeps forwarding their messages from then on.
fn retire_departures(&mut self) {
let tick = self.tick;
let (expired, waiting): (Vec<_>, Vec<_>) = self
.departed
.drain(..)
.partition(|departure| tick.saturating_sub(departure.tick) >= FORWARDING_TICKS);
self.departed = waiting;
for departure in expired {
if self.router.retire_forwarder(departure.key, departure.shard) {
departure.actor.stop();
}
}
}
/// Hands every entity standing outside this shard's columns to the shard that owns its
/// position. The entities' actors pass their whole state on themselves.
/// Migrates every entity standing outside this shard's columns to its owning shard.
fn transfer_out(&mut self, census: &Census) {
if self.shard.count == 1 {
return;
}
let request = self.mailbox.request();
let mut leaving = Vec::new();
for person in &census.people {
if !self.owns(person.position())
&& let Some(actor) = self.person_actors.remove(&person.id)
{
if !self.owns(person.position()) && self.person_actors.contains_key(&person.id) {
let shard = self.shard.owner(person.x, self.width);
actor.tell(PersonMsg::Depart(request, shard), self.mailbox.reply_to());
leaving.push((
EntityKey::person(person.id),
shard,
LocalActor::Person(actor),
));
let key = EntityKey::person(person.id);
match self
.router
.migrate(key, shard, Admission::Person(Box::new(person.clone())))
{
Ok(path) => {
self.person_actors.remove(&person.id);
self.report.moved.push((key, shard, path));
}
Err(error) => {
tracing::error!(entity = %key.name(), shard, %error, "failed to migrate entity");
}
}
}
}
for species in [Species::Rabbit, Species::Wolf] {
@ -1764,35 +1744,43 @@ impl World {
Species::Wolf => &census.wolves,
};
for (id, animal) in animals {
if !self.owns(animal.position())
&& let Some(actor) = self.animal_actors_mut(species).remove(id)
{
if !self.owns(animal.position()) && self.animal_actors(species).contains_key(id) {
let shard = self.shard.owner(animal.x, self.width);
actor.tell(AnimalMsg::Depart(request, shard), self.mailbox.reply_to());
let key = EntityKey::new(species.kind(), *id);
leaving.push((key, shard, LocalActor::Animal(actor)));
match self.router.migrate(
key,
shard,
Admission::Animal(species.kind(), *id, animal.clone()),
) {
Ok(path) => {
self.animal_actors_mut(species).remove(id);
self.report.moved.push((key, shard, path));
}
Err(error) => {
tracing::error!(entity = %key.name(), shard, %error, "failed to migrate entity");
}
}
}
}
}
for cart in &census.carts {
if !self.owns(cart.position())
&& let Some(actor) = self.cart_actors.remove(&cart.id)
{
if !self.owns(cart.position()) && self.cart_actors.contains_key(&cart.id) {
let shard = self.shard.owner(cart.x, self.width);
actor.tell(CartMsg::Depart(request, shard), self.mailbox.reply_to());
leaving.push((cart.key(), shard, LocalActor::Cart(actor)));
let key = cart.key();
match self
.router
.migrate(key, shard, Admission::Cart(cart.clone()))
{
Ok(path) => {
self.cart_actors.remove(&cart.id);
self.report.moved.push((key, shard, path));
}
Err(error) => {
tracing::error!(entity = %key.name(), shard, %error, "failed to migrate entity");
}
}
}
}
self.mailbox.collect(request, leaving.len());
for (key, shard, actor) in leaving {
self.report.moved.push((key, shard));
self.departed.push(Departure {
key,
shard,
tick: self.tick,
actor,
});
}
}
/// Sends one message to every actor in `actors` and waits for all replies.
@ -2124,7 +2112,6 @@ impl World {
pub fn step(&mut self) {
self.tick += 1;
self.receive_arrivals();
self.retire_departures();
self.regrow();
let new_day = self.tick.is_multiple_of(self.calendar.ticks_per_day);
if new_day {
@ -3503,18 +3490,6 @@ impl World {
pub fn console(&self) -> picard::console::Console {
self.actor_system.console()
}
/// Lets the Picard console reach the actors of another shard as `@name`.
#[cfg(feature = "console")]
pub fn register_console_peer(
&self,
name: &str,
sender: picard::console::ConsolePeerSender,
query: picard::console::ConsolePeerQuery,
) {
self.actor_system.register_console_peer(name, sender);
self.actor_system.register_console_peer_query(name, query);
}
}
/// The interface the cluster uses to drive a shard.
@ -3528,14 +3503,60 @@ impl World {
self.receive_arrivals();
}
pub fn shard(&self) -> Shard {
self.shard
}
pub fn router(&self) -> &Arc<Router> {
&self.router
}
pub fn start_remoting(&mut self, token: &str) -> Result<(), String> {
let transport = Arc::new(picard::remoting::ClusterTransport::new());
let authenticator = picard::remoting::PeerAuthenticator::new(
(0..self.shard.count)
.filter(|index| *index != self.shard.index)
.map(|index| (format!("shard-{index}"), token.to_owned())),
);
let bind = "127.0.0.1:0"
.parse()
.map_err(|error: std::net::AddrParseError| error.to_string())?;
let listener = self
.actor_system
.listen_remote_with(
bind,
authenticator,
self.actor_system.remote_handler(),
transport.clone(),
)
.map_err(|error| error.to_string())?;
let address = listener
.local_addr()
.ok_or_else(|| "Picard remoting listener did not report its address".to_owned())?;
self.remote_address = Some(format!("ws://{address}/remoting"));
self.remote_transport = Some(transport);
self.remote_listener = Some(listener);
self.router.register_migration_handler();
Ok(())
}
pub fn connect_peers(&self, peers: Vec<crate::cluster::protocol::Peer>) -> Result<(), String> {
let transport = self
.remote_transport
.as_ref()
.ok_or_else(|| "Picard remoting has not been started".to_owned())?
.clone();
let peers = crate::cluster::transport::ClusterTransport::connect(
&self.actor_system,
self.shard.index,
peers,
transport,
)?;
self.router.set_remote(peers.clone());
self.router.set_peers(peers.peers());
Ok(())
}
pub fn remote_address(&self) -> Option<String> {
self.remote_address.clone()
}
/// Prices of this shard's markets, for the gateway to publish to every shard.
pub fn quotes(&self) -> Vec<Quote> {
let local: HashSet<EntityKey> = self
@ -3567,29 +3588,34 @@ impl World {
std::mem::take(&mut self.report)
}
/// Every person and animal currently simulated by this shard.
pub fn roster(&self) -> Vec<EntityKey> {
let people = self.person_actors.keys().map(|&id| EntityKey::person(id));
/// Every entity currently simulated by this shard and its current actor path.
pub fn roster(&self) -> Vec<(EntityKey, String)> {
let people = self
.person_actors
.iter()
.map(|(&id, actor)| (EntityKey::person(id), actor.path().to_string()));
let rabbits = self
.rabbit_actors
.keys()
.map(|&id| EntityKey::new(Kind::Rabbit, id));
.iter()
.map(|(&id, actor)| (EntityKey::new(Kind::Rabbit, id), actor.path().to_string()));
let wolves = self
.wolf_actors
.keys()
.map(|&id| EntityKey::new(Kind::Wolf, id));
let workplaces = self
.workplace_actors
.keys()
.map(|&id| EntityKey::new(Kind::Workplace, id));
.iter()
.map(|(&id, actor)| (EntityKey::new(Kind::Wolf, id), actor.path().to_string()));
let workplaces = self.workplace_actors.iter().map(|(&id, actor)| {
(
EntityKey::new(Kind::Workplace, id),
actor.path().to_string(),
)
});
let markets = self
.market_actors
.keys()
.map(|&id| EntityKey::new(Kind::Market, id));
.iter()
.map(|(&id, actor)| (EntityKey::new(Kind::Market, id), actor.path().to_string()));
let carts = self
.cart_actors
.keys()
.map(|&id| EntityKey::new(Kind::Cart, id));
.iter()
.map(|(&id, actor)| (EntityKey::new(Kind::Cart, id), actor.path().to_string()));
people
.chain(rabbits)
.chain(wolves)
@ -3602,6 +3628,9 @@ impl World {
impl Drop for World {
fn drop(&mut self) {
if let Some(listener) = &self.remote_listener {
listener.shutdown();
}
// Shutdown starts eagerly; Drop must not block waiting for completion.
drop(self.actor_system.shutdown());
}
@ -3613,47 +3642,32 @@ mod tests {
use super::economy::{Cart, Good, Job, MarketMsg, Phase, Trade, Workplace};
use super::router::Kind;
use super::{
Animal, CROP, DEFAULT_SCRIPT, EntityKey, FORWARDING_TICKS, H, LifeStage, Person, Router,
Shard, SimScript, W, World, cheb, default_config,
Animal, CROP, DEFAULT_SCRIPT, EntityKey, H, LifeStage, Person, Shard, SimScript, W, World,
cheb, default_config,
};
use crate::world::router::RemoteSender;
use picard::remoting::{ActorAddress, RemoteRequest};
use std::sync::{Arc, Mutex, Weak};
use std::time::{Duration, Instant};
/// Delivers remote messages by calling the receiving shard's router directly.
#[derive(Default)]
struct Loopback {
routers: Mutex<Vec<Weak<Router>>>,
}
impl RemoteSender for Loopback {
fn send(&self, shard: usize, path: &str, payload: serde_json::Value) {
let router = self.routers.lock().unwrap()[shard].upgrade().unwrap();
let destination = ActorAddress::local(router.system_id(), path);
let response =
router.handle("loopback", RemoteRequest::new(destination, None, payload));
assert!(response.accepted, "{path} rejected: {:?}", response.error);
}
fn flush(&self) {}
}
fn connected_shards(count: usize) -> Vec<World> {
let loopback = Arc::new(Loopback::default());
let shards: Vec<World> = (0..count)
let mut shards: Vec<World> = (0..count)
.map(|index| {
let shard = Shard::new(index, count).unwrap();
World::new_shard(42, SimScript::load(DEFAULT_SCRIPT).unwrap(), shard)
})
.collect();
let token = uuid::Uuid::new_v4().to_string();
for world in &mut shards {
world.start_remoting(&token).unwrap();
}
let peers = shards
.iter()
.map(|world| crate::cluster::protocol::Peer {
address: world.remote_address().unwrap(),
token: token.clone(),
system_id: world.router().system_id(),
})
.collect::<Vec<_>>();
for world in &shards {
loopback
.routers
.lock()
.unwrap()
.push(Arc::downgrade(world.router()));
world.router().set_remote(loopback.clone());
world.connect_peers(peers.clone()).unwrap();
}
shards
}
@ -3663,8 +3677,18 @@ mod tests {
let mut changes = Vec::new();
for (index, world) in shards.iter_mut().enumerate() {
let report = world.take_report();
changes.extend(report.moved.into_iter().map(|(key, to)| (key, Some(to))));
changes.extend(report.arrived.into_iter().map(|key| (key, Some(index))));
changes.extend(
report
.moved
.into_iter()
.map(|(key, to, path)| (key, Some((to, path)))),
);
changes.extend(
report
.arrived
.into_iter()
.map(|(key, path)| (key, Some((index, path)))),
);
changes.extend(report.left.into_iter().map(|key| (key, None)));
}
for world in shards {
@ -4095,7 +4119,11 @@ mod tests {
assert!(position == expected.home || expected.res[index] == *res);
}
}
let mut ids: Vec<EntityKey> = shards.iter().flat_map(World::roster).collect();
let mut ids: Vec<EntityKey> = shards
.iter()
.flat_map(World::roster)
.map(|(key, _)| key)
.collect();
let total = ids.len();
ids.sort_unstable();
ids.dedup();
@ -4118,7 +4146,10 @@ mod tests {
shards[0].step();
assert!(!shards[0].person_actors.contains_key(&id));
let report = shards[0].take_report();
assert_eq!(report.moved, vec![(EntityKey::person(id), 1)]);
assert_eq!(report.moved.len(), 1);
assert_eq!(report.moved[0].0, EntityKey::person(id));
assert_eq!(report.moved[0].1, 1);
assert!(report.moved[0].2.starts_with("/user/person-"));
shards[1].receive_arrivals();
assert!(shards[1].person_actors.contains_key(&id));
@ -4130,21 +4161,14 @@ mod tests {
assert_eq!(arrived.wood, 7);
assert!(shards[1].owns(arrived.position()));
// The old actor forwards messages sent to the person on the shard they left.
// Picard's migration forwarder keeps the old address usable before the directory update.
shards[0].tell_person(id, PersonMsg::Edit(PersonEdit::Stone(3)));
wait_for_person(&shards[1], id, |person| person.stone == 3);
// Once it stops, the router forwards them itself until the directory catches up.
shards[0].tick += FORWARDING_TICKS;
shards[0].retire_departures();
assert!(shards[0].departed.is_empty());
publish_directory(&mut shards);
shards[0].tell_person(id, PersonMsg::Edit(PersonEdit::Stone(4)));
wait_for_person(&shards[1], id, |person| person.stone == 4);
publish_directory(&mut shards);
shards[0].tell_person(id, PersonMsg::Edit(PersonEdit::Stone(5)));
wait_for_person(&shards[1], id, |person| person.stone == 5);
// Walking back resumes life on the first shard.
let back = shards[0].home;
shards[1].tell_person(id, PersonMsg::MoveTo(back));
@ -4153,8 +4177,8 @@ mod tests {
assert!(shards[0].person_actors.contains_key(&id));
assert!(!shards[1].person_actors.contains_key(&id));
publish_directory(&mut shards);
shards[1].tell_person(id, PersonMsg::Edit(PersonEdit::Stone(6)));
wait_for_person(&shards[0], id, |person| person.stone == 6);
shards[1].tell_person(id, PersonMsg::Edit(PersonEdit::Stone(5)));
wait_for_person(&shards[0], id, |person| person.stone == 5);
}
#[test]

72
src/world/actors.rs
View File

@ -5,7 +5,7 @@
//! lets them apply entity-local rules (ageing, hunger, eating, energy, dying) themselves.
use super::economy::{Cart, Job, Market, Traded, Workplace};
use super::router::{Admission, EntityKey, Router};
use super::router::EntityKey;
use super::{Animal, CalendarSettings, LifeStage, Partnership, Person, Rng, sample_stage_duration};
use picard::actors::{
Actor, ActorFactoryArgs, ActorRef, ActorRefFactory, ActorSystem, Context as ActorContext,
@ -14,7 +14,6 @@ use picard::actors::{
use serde::{Deserialize, Serialize};
use std::cell::{Cell, RefCell};
use std::collections::HashMap;
use std::sync::Arc;
use std::sync::mpsc::{Receiver, Sender as MpscSender, channel};
use std::time::Duration;
@ -125,11 +124,6 @@ pub(super) enum PersonMsg {
},
SetJob(Option<Job>),
SetCart(Option<u64>),
/// Hands the person to `shard`, whose region they walked into; this actor then forwards
/// every message to them until it is stopped.
Depart(RequestId, usize),
/// The person walked back into this shard while this actor was still forwarding.
Readmit(Person),
}
/// One animal tick: the world supplies movement and opportunities, the animal its metabolism.
@ -159,8 +153,6 @@ pub(super) enum AnimalMsg {
/// A successful attack; only the first one kills the animal.
Hunted(RequestId),
SetEnergyWithin(Area, f32),
Depart(RequestId, usize),
Readmit(Animal),
}
#[derive(Clone, Debug, Deserialize, Serialize)]
@ -188,7 +180,6 @@ pub(super) enum ReplyBody {
Hunted(AnimalId, bool),
Partnership(Partnership),
BirthDue(Partnership, bool),
Departed,
Market(Market),
Workplace(Workplace),
Cart(Cart),
@ -215,22 +206,15 @@ pub(super) fn reply(sender: Sender, request: RequestId, body: ReplyBody) {
pub(super) struct PersonActor {
person: Person,
router: Arc<Router>,
/// The shard this person moved to; set while this actor only forwards messages.
forward_to: Option<usize>,
}
impl ActorFactoryArgs<(Person, Arc<Router>)> for PersonActor {
fn create_args((mut person, router): (Person, Arc<Router>)) -> Self {
impl ActorFactoryArgs<Person> for PersonActor {
fn create_args(mut person: Person) -> Self {
if person.rng == 0 {
// Snapshots from before people had their own random stream.
person.rng = u64::from(person.id).wrapping_mul(0x9E37_79B9_7F4A_7C15) | 1;
}
Self {
person,
router,
forward_to: None,
}
Self { person }
}
}
@ -320,16 +304,6 @@ impl Actor for PersonActor {
type Msg = PersonMsg;
fn recv(&mut self, _ctx: &ActorContext<Self::Msg>, msg: Self::Msg, sender: Sender) {
if let Some(shard) = self.forward_to {
if let PersonMsg::Readmit(person) = msg {
self.person = person;
self.forward_to = None;
} else {
let key = EntityKey::person(self.person.id);
self.router.forward(shard, key, &msg);
}
return;
}
match msg {
PersonMsg::Observe(request) => {
reply(sender, request, ReplyBody::Person(self.person.clone()));
@ -367,34 +341,18 @@ impl Actor for PersonActor {
self.person.unpaid_days = 0;
}
PersonMsg::SetCart(cart) => self.person.cart = cart,
PersonMsg::Depart(request, shard) => {
self.router
.admit_at(shard, Admission::Person(Box::new(self.person.clone())));
self.forward_to = Some(shard);
reply(sender, request, ReplyBody::Departed);
}
PersonMsg::Readmit(person) => self.person = person,
}
}
}
pub(super) struct AnimalActor {
key: EntityKey,
id: AnimalId,
animal: Animal,
router: Arc<Router>,
forward_to: Option<usize>,
}
impl ActorFactoryArgs<(EntityKey, Animal, Arc<Router>)> for AnimalActor {
fn create_args((key, animal, router): (EntityKey, Animal, Arc<Router>)) -> Self {
Self {
key,
id: key.id,
animal,
router,
forward_to: None,
}
impl ActorFactoryArgs<(EntityKey, Animal)> for AnimalActor {
fn create_args((key, animal): (EntityKey, Animal)) -> Self {
Self { id: key.id, animal }
}
}
@ -436,15 +394,6 @@ impl Actor for AnimalActor {
type Msg = AnimalMsg;
fn recv(&mut self, _ctx: &ActorContext<Self::Msg>, msg: Self::Msg, sender: Sender) {
if let Some(shard) = self.forward_to {
if let AnimalMsg::Readmit(animal) = msg {
self.animal = animal;
self.forward_to = None;
} else {
self.router.forward(shard, self.key, &msg);
}
return;
}
match msg {
AnimalMsg::Observe(request) => {
reply(
@ -475,13 +424,6 @@ impl Actor for AnimalActor {
self.animal.energy = energy;
}
}
AnimalMsg::Depart(request, shard) => {
let admission = Admission::Animal(self.key.kind, self.id, self.animal.clone());
self.router.admit_at(shard, admission);
self.forward_to = Some(shard);
reply(sender, request, ReplyBody::Departed);
}
AnimalMsg::Readmit(animal) => self.animal = animal,
}
}
}

31
src/world/economy.rs
View File

@ -6,7 +6,7 @@
//! between people, workplaces and markets.
use super::actors::{PersonMsg, ReplyBody, RequestId, reply};
use super::router::{Admission, EntityKey, Kind, Router};
use super::router::{EntityKey, Kind, Router};
use picard::actors::{Actor, ActorFactoryArgs, Context as ActorContext, Sender};
use serde::{Deserialize, Serialize};
use std::sync::Arc;
@ -722,23 +722,15 @@ pub(super) enum CartMsg {
},
SetOwner(Option<u32>),
MoveTo((usize, usize)),
Depart(RequestId, usize),
Readmit(Cart),
}
pub(super) struct CartActor {
cart: Cart,
router: Arc<Router>,
forward_to: Option<usize>,
}
impl ActorFactoryArgs<(Cart, Arc<Router>)> for CartActor {
fn create_args((cart, router): (Cart, Arc<Router>)) -> Self {
Self {
cart,
router,
forward_to: None,
}
impl ActorFactoryArgs<Cart> for CartActor {
fn create_args(cart: Cart) -> Self {
Self { cart }
}
}
@ -746,15 +738,6 @@ impl Actor for CartActor {
type Msg = CartMsg;
fn recv(&mut self, _ctx: &ActorContext<Self::Msg>, msg: Self::Msg, sender: Sender) {
if let Some(shard) = self.forward_to {
if let CartMsg::Readmit(cart) = msg {
self.cart = cart;
self.forward_to = None;
} else {
self.router.forward(shard, self.cart.key(), &msg);
}
return;
}
let cart = &mut self.cart;
match msg {
CartMsg::Observe(request) => reply(sender, request, ReplyBody::Cart(cart.clone())),
@ -773,12 +756,6 @@ impl Actor for CartActor {
}
CartMsg::SetOwner(owner) => cart.owner = owner,
CartMsg::MoveTo(position) => (cart.x, cart.y) = position,
CartMsg::Depart(request, shard) => {
self.router.admit_at(shard, Admission::Cart(cart.clone()));
self.forward_to = Some(shard);
reply(sender, request, ReplyBody::Departed);
}
CartMsg::Readmit(readmitted) => *cart = readmitted,
}
}
}

350
src/world/router.rs
View File

@ -15,17 +15,14 @@ use super::economy::{
WorkplaceMsg,
};
use super::{Animal, Partnership, Person};
use crate::cluster::transport::PeerConnection;
use picard::actors::{ActorRef, ActorRefFactory, ActorReference, ActorSystem, CreateError, Tell};
use picard::remoting::{RemoteError, RemoteHandler, RemoteRequest, RemoteResponse};
use serde::{Deserialize, Serialize};
use std::collections::HashMap;
use std::ops::Range;
use std::sync::{Arc, Mutex, RwLock, Weak};
use std::time::Duration;
/// Remote path that accepts entities moving into this shard's region.
pub const ADMIT_PATH: &str = "/shard/admit";
#[derive(Clone, Copy, Debug, Deserialize, Serialize, PartialEq, Eq, Hash, PartialOrd, Ord)]
#[serde(rename_all = "snake_case")]
pub enum Kind {
@ -39,16 +36,6 @@ pub enum Kind {
}
impl Kind {
const ALL: [Kind; 7] = [
Kind::Person,
Kind::Rabbit,
Kind::Wolf,
Kind::Partnership,
Kind::Workplace,
Kind::Market,
Kind::Cart,
];
fn name(self) -> &'static str {
match self {
Kind::Person => "person",
@ -82,16 +69,6 @@ impl EntityKey {
pub fn name(self) -> String {
format!("{}-{}", self.kind.name(), self.id)
}
pub fn path(self) -> String {
format!("/user/{}", self.name())
}
pub fn from_path(path: &str) -> Option<Self> {
let (kind, id) = path.strip_prefix("/user/")?.rsplit_once('-')?;
let kind = Kind::ALL.into_iter().find(|k| k.name() == kind)?;
Some(Self::new(kind, id.parse().ok()?))
}
}
/// This process's part of the world: shard `index` of `count` owns a vertical strip of columns.
@ -142,11 +119,10 @@ impl Shard {
/// Delivers serialised messages to other shards; implemented by the cluster transport.
pub trait RemoteSender: Send + Sync {
/// Queues `payload` for the actor at `path` on shard `shard`. Messages to one shard are
/// delivered in the order they were queued.
/// Sends `payload` to the actor at `path` on shard `shard`.
fn send(&self, shard: usize, path: &str, payload: serde_json::Value);
/// Waits until every queued message has been accepted by its receiving shard.
/// Waits until all sent messages have been accepted by their receiving actors.
fn flush(&self);
}
@ -172,6 +148,17 @@ impl LocalActor {
LocalActor::Cart(actor) => actor.stop(),
}
}
pub(super) fn path(&self) -> String {
match self {
LocalActor::Person(actor) => actor.path().to_string(),
LocalActor::Animal(actor) => actor.path().to_string(),
LocalActor::Partnership(actor) => actor.path().to_string(),
LocalActor::Workplace(actor) => actor.path().to_string(),
LocalActor::Market(actor) => actor.path().to_string(),
LocalActor::Cart(actor) => actor.path().to_string(),
}
}
}
/// A message type that can be routed to the entities whose actors accept it.
@ -241,16 +228,6 @@ pub(super) enum Admission {
Cart(Cart),
}
impl Admission {
fn key(&self) -> EntityKey {
match self {
Admission::Person(person) => EntityKey::person(person.id),
Admission::Animal(kind, id, _) => EntityKey::new(*kind, *id),
Admission::Cart(cart) => cart.key(),
}
}
}
/// An admitted entity, waiting for the world to pick it up at the start of its next step.
pub(super) enum Arrival {
Person(Box<Person>, ActorRef<PersonMsg>),
@ -258,26 +235,14 @@ pub(super) enum Arrival {
Cart(u64, ActorRef<CartMsg>),
}
impl Arrival {
fn key(&self) -> EntityKey {
match self {
Arrival::Person(person, _) => EntityKey::person(person.id),
Arrival::Animal(kind, id, ..) => EntityKey::new(*kind, *id),
Arrival::Cart(id, _) => EntityKey::new(Kind::Cart, *id),
}
}
}
pub struct Router {
shard: Shard,
system: ActorSystem,
this: Weak<Router>,
dispatch: RemoteHandler,
local: RwLock<HashMap<EntityKey, LocalActor>>,
/// Shard of every entity elsewhere in the cluster, as last published by the gateway.
directory: RwLock<HashMap<EntityKey, usize>>,
/// Where entities that left this shard went, until the directory catches up.
forwarding: RwLock<HashMap<EntityKey, usize>>,
/// Shard and current actor path of every entity elsewhere in the cluster.
directory: RwLock<HashMap<EntityKey, (usize, String)>>,
peers: RwLock<HashMap<usize, PeerConnection>>,
remote: RwLock<Option<Arc<dyn RemoteSender>>>,
arrivals: Mutex<Vec<Arrival>>,
}
@ -288,10 +253,9 @@ impl Router {
shard,
system: system.clone(),
this: this.clone(),
dispatch: system.remote_handler(),
local: RwLock::default(),
directory: RwLock::default(),
forwarding: RwLock::default(),
peers: RwLock::default(),
remote: RwLock::default(),
arrivals: Mutex::default(),
})
@ -306,12 +270,15 @@ impl Router {
*self.remote.write().expect("router lock") = Some(remote);
}
pub(super) fn set_peers(&self, peers: HashMap<usize, PeerConnection>) {
*self.peers.write().expect("router peer lock") = peers;
}
fn arc(&self) -> Arc<Router> {
self.this.upgrade().expect("the router outlives its actors")
}
pub(super) fn register(&self, key: EntityKey, actor: LocalActor) {
self.forwarding.write().expect("router lock").remove(&key);
self.local.write().expect("router lock").insert(key, actor);
}
@ -319,26 +286,16 @@ impl Router {
self.local.write().expect("router lock").remove(&key);
}
/// Remembers that `key` moved to `shard` once its forwarding actor is gone.
pub(super) fn set_forwarding(&self, key: EntityKey, shard: usize) {
self.forwarding
.write()
.expect("router lock")
.insert(key, shard);
}
/// Applies directory changes published by the gateway; `None` means the entity is gone.
pub fn apply_directory(&self, changes: &[(EntityKey, Option<usize>)]) {
pub fn apply_directory(&self, changes: &[(EntityKey, Option<(usize, String)>)]) {
let mut directory = self.directory.write().expect("router lock");
let mut forwarding = self.forwarding.write().expect("router lock");
for &(key, shard) in changes {
forwarding.remove(&key);
match shard {
Some(shard) if shard != self.shard.index => {
directory.insert(key, shard);
for (key, location) in changes {
match location {
Some((shard, path)) if *shard != self.shard.index => {
directory.insert(*key, (*shard, path.clone()));
}
_ => {
directory.remove(&key);
directory.remove(key);
}
}
}
@ -350,22 +307,21 @@ impl Router {
|| self.remote_location(key).is_some()
}
fn remote_location(&self, key: EntityKey) -> Option<usize> {
let forwarded = self
.forwarding
pub(super) fn actor_path(&self, key: EntityKey) -> Option<String> {
self.local
.read()
.expect("router lock")
.get(&key)
.copied();
forwarded
.or_else(|| {
self.directory
.read()
.expect("router lock")
.get(&key)
.copied()
})
.filter(|&shard| shard != self.shard.index)
.map(LocalActor::path)
}
fn remote_location(&self, key: EntityKey) -> Option<(usize, String)> {
self.directory
.read()
.expect("router lock")
.get(&key)
.filter(|(shard, _)| *shard != self.shard.index)
.cloned()
}
/// Delivers `message` to the entity, wherever in the cluster it is.
@ -379,17 +335,17 @@ impl Router {
.cloned();
if let Some(actor) = local {
actor.tell(message, None);
} else if let Some(shard) = self.remote_location(key) {
self.forward(shard, key, &message);
} else if let Some((shard, path)) = self.remote_location(key) {
self.forward(shard, &path, &message);
} else {
tracing::debug!(entity = %key.name(), "dropped a message for an unknown entity");
}
}
/// Sends `message` to the entity on `shard`, bypassing the local registry.
pub(super) fn forward<M: Routed>(&self, shard: usize, key: EntityKey, message: &M) {
pub(super) fn forward<M: Routed>(&self, shard: usize, path: &str, message: &M) {
match serde_json::to_value(message) {
Ok(payload) => self.send_remote(shard, &key.path(), payload),
Ok(payload) => self.send_remote(shard, path, payload),
Err(error) => tracing::warn!(%error, "failed to serialise a routed message"),
}
}
@ -402,72 +358,37 @@ impl Router {
}
}
/// Hands an entity to the shard owning its new position.
pub(super) fn admit_at(&self, shard: usize, admission: Admission) {
if shard == self.shard.index {
self.admit(admission);
return;
}
match serde_json::to_value(&admission) {
Ok(payload) => self.send_remote(shard, ADMIT_PATH, payload),
Err(error) => tracing::warn!(%error, "failed to serialise an admission"),
}
}
/// Takes in an entity: an actor still forwarding for it resumes, otherwise a new one starts.
pub(super) fn admit(&self, admission: Admission) {
// Holding the arrivals lock keeps `retire_forwarder` from stopping a resumed actor.
/// Recreates an entity actor at the destination path of a Picard migration.
pub(super) fn admit(&self, admission: Admission, path: &str) -> Result<(), String> {
let mut arrivals = self.arrivals.lock().expect("router lock");
let key = admission.key();
let existing = self.local.read().expect("router lock").get(&key).cloned();
let arrival = match (admission, existing) {
(Admission::Person(person), Some(LocalActor::Person(actor))) => {
actor.tell(PersonMsg::Readmit((*person).clone()), None);
let name = path
.rsplit('/')
.next()
.filter(|name| !name.is_empty())
.ok_or_else(|| format!("invalid migrated actor path: {path}"))?;
let arrival = match admission {
Admission::Person(person) => {
let actor = self
.spawn_person_named((*person).clone(), name)
.ok_or_else(|| format!("failed to recreate person actor at {path}"))?;
Arrival::Person(person, actor)
}
(Admission::Person(person), _) => {
let Some(actor) = self.spawn_person((*person).clone()) else {
return;
};
Arrival::Person(person, actor)
}
(Admission::Animal(kind, id, animal), Some(LocalActor::Animal(actor))) => {
actor.tell(AnimalMsg::Readmit(animal), None);
Admission::Animal(kind, id, animal) => {
let actor = self
.spawn_animal_named(kind, id, animal, name)
.ok_or_else(|| format!("failed to recreate animal actor at {path}"))?;
Arrival::Animal(kind, id, actor)
}
(Admission::Animal(kind, id, animal), _) => {
let Some(actor) = self.spawn_animal(kind, id, animal) else {
return;
};
Arrival::Animal(kind, id, actor)
}
(Admission::Cart(cart), Some(LocalActor::Cart(actor))) => {
Admission::Cart(cart) => {
let id = cart.id;
actor.tell(CartMsg::Readmit(cart), None);
Arrival::Cart(id, actor)
}
(Admission::Cart(cart), _) => {
let id = cart.id;
let Some(actor) = self.spawn_cart(cart) else {
return;
};
let actor = self
.spawn_cart_named(cart, name)
.ok_or_else(|| format!("failed to recreate cart actor at {path}"))?;
Arrival::Cart(id, actor)
}
};
self.forwarding.write().expect("router lock").remove(&key);
arrivals.push(arrival);
}
/// Replaces the forwarding actor of an entity that moved to `shard` by a forwarding entry;
/// returns `false` when the entity has come back in the meantime and must keep its actor.
pub(super) fn retire_forwarder(&self, key: EntityKey, shard: usize) -> bool {
let arrivals = self.arrivals.lock().expect("router lock");
let returned = arrivals.iter().any(|arrival| arrival.key() == key);
if !returned {
self.unregister(key);
self.set_forwarding(key, shard);
}
!returned
Ok(())
}
pub(super) fn take_arrivals(&self) -> Vec<Arrival> {
@ -477,13 +398,12 @@ impl Router {
/// Starts an actor, retrying while a stopped actor with the same name is still shutting down.
fn start<M: picard::LocalMessage>(
&self,
key: EntityKey,
name: &str,
create: impl Fn(&str) -> Result<ActorRef<M>, CreateError>,
) -> Option<ActorRef<M>> {
let name = key.name();
let mut attempts = 0;
loop {
match create(&name) {
match create(name) {
Ok(actor) => return Some(actor),
Err(CreateError::AlreadyExists(_)) if attempts < 200 => {
attempts += 1;
@ -499,9 +419,14 @@ impl Router {
pub(super) fn spawn_person(&self, person: Person) -> Option<ActorRef<PersonMsg>> {
let key = EntityKey::person(person.id);
let actor = self.start(key, |name| {
self.spawn_person_named(person, &key.name())
}
fn spawn_person_named(&self, person: Person, name: &str) -> Option<ActorRef<PersonMsg>> {
let key = EntityKey::person(person.id);
let actor = self.start(name, |name| {
self.system
.actor_of_args::<PersonActor, _>(name, (person.clone(), self.arc()))
.actor_of_args::<PersonActor, _>(name, person.clone())
})?;
self.register(key, LocalActor::Person(actor.clone()));
Some(actor)
@ -514,9 +439,20 @@ impl Router {
animal: Animal,
) -> Option<ActorRef<AnimalMsg>> {
let key = EntityKey::new(kind, id);
let actor = self.start(key, |name| {
self.spawn_animal_named(kind, id, animal, &key.name())
}
fn spawn_animal_named(
&self,
kind: Kind,
id: AnimalId,
animal: Animal,
name: &str,
) -> Option<ActorRef<AnimalMsg>> {
let key = EntityKey::new(kind, id);
let actor = self.start(name, |name| {
self.system
.actor_of_args::<AnimalActor, _>(name, (key, animal.clone(), self.arc()))
.actor_of_args::<AnimalActor, _>(name, (key, animal.clone()))
})?;
self.register(key, LocalActor::Animal(actor.clone()));
Some(actor)
@ -527,7 +463,7 @@ impl Router {
partnership: Partnership,
) -> Option<ActorRef<PartnershipMsg>> {
let key = EntityKey::new(Kind::Partnership, u64::from(partnership.id));
let actor = self.start(key, |name| {
let actor = self.start(&key.name(), |name| {
self.system
.actor_of_args::<PartnershipActor, _>(name, partnership.clone())
})?;
@ -537,7 +473,7 @@ impl Router {
pub(super) fn spawn_workplace(&self, workplace: Workplace) -> Option<ActorRef<WorkplaceMsg>> {
let key = workplace.key();
let actor = self.start(key, |name| {
let actor = self.start(&key.name(), |name| {
self.system
.actor_of_args::<WorkplaceActor, _>(name, (workplace.clone(), self.arc()))
})?;
@ -547,7 +483,7 @@ impl Router {
pub(super) fn spawn_market(&self, market: Market) -> Option<ActorRef<MarketMsg>> {
let key = market.key();
let actor = self.start(key, |name| {
let actor = self.start(&key.name(), |name| {
self.system
.actor_of_args::<MarketActor, _>(name, (market.clone(), self.arc()))
})?;
@ -557,53 +493,19 @@ impl Router {
pub(super) fn spawn_cart(&self, cart: Cart) -> Option<ActorRef<CartMsg>> {
let key = cart.key();
let actor = self.start(key, |name| {
self.spawn_cart_named(cart, &key.name())
}
fn spawn_cart_named(&self, cart: Cart, name: &str) -> Option<ActorRef<CartMsg>> {
let key = cart.key();
let actor = self.start(name, |name| {
self.system
.actor_of_args::<CartActor, _>(name, (cart.clone(), self.arc()))
.actor_of_args::<CartActor, _>(name, cart.clone())
})?;
self.register(key, LocalActor::Cart(actor.clone()));
Some(actor)
}
/// Answers a remoting request from another shard: admissions are taken in directly, entity
/// messages go to Picard's remote dispatch, and messages for entities that are no longer
/// here are forwarded to where they went.
pub fn handle(&self, peer: &str, request: RemoteRequest) -> RemoteResponse {
let id = request.id;
if request.destination.system_id != self.system.id() {
return RemoteResponse::rejected(
id,
RemoteError::Rejected("destination system ID does not match this shard".into()),
);
}
if request.destination.path == ADMIT_PATH {
return match serde_json::from_value::<Admission>(request.payload) {
Ok(admission) => {
self.admit(admission);
RemoteResponse::accepted(id)
}
Err(error) => {
RemoteResponse::rejected(id, RemoteError::InvalidMessage(error.to_string()))
}
};
}
let path = request.destination.path.clone();
let payload = request.payload.clone();
let response = (self.dispatch)(peer, request);
if !response.accepted
&& matches!(
response.error,
Some(RemoteError::ActorNotFound(_) | RemoteError::Rejected(_))
)
&& let Some(key) = EntityKey::from_path(&path)
&& let Some(shard) = self.remote_location(key)
{
self.send_remote(shard, &path, payload);
return RemoteResponse::accepted(id);
}
response
}
/// Waits until every message queued for other shards has been accepted.
pub fn flush(&self) {
let remote = self.remote.read().expect("router lock").clone();
@ -611,11 +513,59 @@ impl Router {
remote.flush();
}
}
pub(super) fn migrate(
&self,
key: EntityKey,
shard: usize,
admission: Admission,
) -> Result<String, String> {
let peer = self
.peers
.read()
.expect("router peer lock")
.get(&shard)
.cloned()
.ok_or_else(|| format!("shard {shard} is not connected"))?;
let local = self
.local
.read()
.expect("router lock")
.get(&key)
.cloned()
.ok_or_else(|| format!("actor {} is not local", key.name()))?;
let destination_path = format!("/user/{}-m-{}", key.name(), uuid::Uuid::new_v4());
let state = serde_json::to_value(admission).map_err(|error| error.to_string())?;
self.system
.migrate_actor(
&local.path(),
peer.system_id,
Some(peer.endpoint),
Some(&destination_path),
"sim-entity",
state,
&peer.connection,
)
.map_err(|error| error.to_string())?;
Ok(destination_path)
}
pub(super) fn register_migration_handler(&self) {
let router = self.arc();
self.system.register_migration_handler(
"sim-entity",
Arc::new(move |_system, payload| {
let admission: Admission = serde_json::from_value(payload.state.clone())
.map_err(|error| format!("invalid migrated entity: {error}"))?;
router.admit(admission, &payload.destination_path)
}),
);
}
}
#[cfg(test)]
mod tests {
use super::{EntityKey, Kind, Shard};
use super::Shard;
#[test]
fn shards_partition_the_columns_and_hand_out_distinct_ids() {
@ -643,12 +593,4 @@ mod tests {
assert_eq!(Shard::new(1, 3).unwrap().share(10), 3);
assert_eq!(Shard::new(0, 3).unwrap().share(10), 4);
}
#[test]
fn entity_keys_round_trip_through_actor_paths() {
let key = EntityKey::new(Kind::Partnership, 42);
assert_eq!(key.path(), "/user/partnership-42");
assert_eq!(EntityKey::from_path(&key.path()), Some(key));
assert_eq!(EntityKey::from_path("/user/world-inbox"), None);
}
}