Metsuke: reading the telemetry archive

This server collects signed telemetry from the pools in the MusashiNet rewards program and keeps every submission exactly as it arrived. This page is how you get that archive onto a machine you can query. It is for whoever is analysing the data; the pool operators sending it start at the quickstart instead.

An archive object is one submission: a batch of metrics scrapes, or a batch of the node trace lines the program selects. Both are JSON Lines, compressed, and every line carries the pool and agent it came from. Nothing is aggregated and nothing is rewritten, so what you download is what a pool signed.

1. Get an account

The archive is not public. Reading it needs a developer account on this server, a username and a password held in the metsuke archive realm, which the operators of this deployment issue. Ask them, and put the password in a file of its own:

mkdir -p ~/.config/metsuke
install -m 0600 /dev/null ~/.config/metsuke/password
$EDITOR ~/.config/metsuke/password

The empty file is created with its mode before anything is written to it, rather than fixed with chmod afterwards: between writing and the chmod your editor has already saved the password at whatever your umask allows, and on a shared machine that window is long enough. The first command truncates the file, so skip it when you come back to change a password rather than set one.

The password goes in a file rather than in a flag or a variable holding the secret itself: an argument is visible in the process table, and an environment variable reaches every child process and anything that may read /proc. The tool below takes the path and reads it itself.

2. Get the tool

metsuke-fetch is what pulls the archive down. It is one static binary with nothing to link against, the same as the agent:

ARCH=x86_64-linux            # or aarch64-linux

curl -fO https://metsuke-leios.play.dev.cardano.org/files/metsuke-fetch-static-$ARCH
curl -fO https://metsuke-leios.play.dev.cardano.org/files/metsuke-fetch-static-$ARCH.sha256
sha256sum -c metsuke-fetch-static-$ARCH.sha256
mv metsuke-fetch-static-$ARCH metsuke-fetch
chmod +x metsuke-fetch

The block above fetches the build for your machine. Either architecture can be built rather than downloaded, with nix build github:input-output-hk/metsuke#metsuke-fetch-static-$ARCH, and nix run github:input-output-hk/metsuke#metsuke-fetch runs it without installing anything. ./metsuke-fetch --help is the complete flag list; what follows is the part of it you need first.

3. Sync the archive

Point it at this server, say who you are, and name two paths: a cursor file and a directory to download into.

export METSUKE_FETCH_SERVER=https://metsuke-leios.play.dev.cardano.org/
export METSUKE_FETCH_USER=<your account>
export METSUKE_FETCH_PASSWORD_FILE=~/.config/metsuke/password
export METSUKE_FETCH_TIMEOUT_MS=30000

./metsuke-fetch sync --state ~/.local/state/metsuke-fetch/cursor.json --into ~/archive

Each object lands under its own key, so the directory is safe to sync into repeatedly. The run prints what it did:

3 keys under v1/; 3 selected, 0 outside the selection, 0 this build cannot name
3 objects into ~/archive, 3145728 bytes; 1 cold-signed, 2 Leios-signed, 0 unattested, 0 not written

The cursor is what makes the next run a delta rather than a re-download: it records the last key seen, and a later run resumes past it. Deleting the file is the only way to start over, and the run after that downloads everything again, because nothing checks whether an object is already on disk.

The cold-signed, Leios-signed and unattested counts on the second line are how much of the archive can prove itself. A cold-signed object was signed by a key that derives the pool it is filed under, so it stays checkable from the object alone, forever. A Leios-signed object had its signature checked here too, but which pool it belongs to is this server's word, taken from a roster; its bytes are just as authentic, and only the attribution rests on us. An unattested object came with nothing to check. Most of the archive is Leios-signed by design, and reading the archive has the flags that keep only what you are willing to trust.

Start with an unbounded sync as above. --from and --to bound a run by when this server received a submission, which is not when its rows were measured; an agent that spooled through an outage can put those days far apart. Slice by time in SQL, where the row's own timestamp is, rather than in the flags.

4. Read it

An object is a zstd stream whose first frame is skippable, so duckdb reads the downloaded tree where it landed with no unpacking step. Neither kind of object reads usefully through select * though: a scrape holds its metrics as a nested list and a trace line holds its payload as a map, so both want flattening first. Two init files do that, and this server hands them out:

curl -fO https://metsuke-leios.play.dev.cardano.org/files/analytics.sql
curl -fO https://metsuke-leios.play.dev.cardano.org/files/archive.sql

analytics.sql is the one to start with. It defines the views a consumer actually groups over: coverage per agent, the block producer's scoreboard, and the lifecycle of every endorsement block. Most first questions are answered without you writing a join:

METSUKE_ARCHIVE=~/archive duckdb -init analytics.sql

archive.sql is the same flattening with none of the views, as three tables over whatever directory you point it at. Load it when your question is not one the views already answer. It globs recursively rather than assuming the archive's layout, and it keeps a row for every submission including the failed ones:

METSUKE_ARCHIVE=~/archive duckdb -init archive.sql archive.duckdb

Naming a database file as above makes the tables persist. A later session then opens duckdb archive.duckdb and asks, rather than re-reading every object. Both files carry their own instructions in their opening comments.

Metsuke Analysis

1. Get an account

2. Get the tool

3. Sync the archive

4. Read it

Further reading