Metsuke: telemetry for the MusashiNet rewards program

Metsuke is a small agent you run beside the Haskell cardano-node on the Musashi network. It reads your node's Prometheus endpoint over loopback, collects the trace lines it has been configured for, and sends this server a signed submission. It never opens your node socket, and it reads only the signing key you point it at.

These instructions assume a node running on the Musashi network, which the getting started guide covers, and a pool already in the rewards program, with your application code in your pool registration's metadata. Submissions are refused until that code is on chain, and the details page says how to put it there.

Try it

Nothing is installed, and it all stops when you press Ctrl-C.

# x86_64-linux. On aarch64-linux, step 1 installs either build.
curl -fo metsuke https://metsuke-leios.play.dev.cardano.org/files/metsuke-static-x86_64-linux
echo 'd20d3b3ab1dcc2406b00c209195b217d012792d3f51a0ac7d192f567c29ce56e  metsuke' | sha256sum -c
chmod +x metsuke
curl -fO https://metsuke-leios.play.dev.cardano.org/files/config.pipe.toml

# Set your pool id, and a spool_path you can write
$EDITOR config.pipe.toml

cardano-node run ... | ./metsuke --config config.pipe.toml --signing-key bls.skey > /dev/null

# Or, where your node already runs under systemd and you can read its
# journal. These are the flags the journald setup runs itself, so the agent
# reads the same bytes either way
journalctl -f -n 0 -o cat -u <your-node> | ./metsuke --config config.pipe.toml --signing-key bls.skey > /dev/null

# Or, where your node writes to a file. -F and not -f, because only -F
# reopens by name after a rotation
tail -n 0 -F /var/log/node.log | ./metsuke --config config.pipe.toml --signing-key bls.skey > /dev/null

That is the whole agent. It writes your node's output straight back out, and the redirect to /dev/null drops that copy: without it a node's trace stream buries what the agent has to say. What is left on your terminal is the agent's own output. The first five lines arrive at once, and the scrape submission about ten seconds later, once the node has had time to bind its metrics endpoint, or straight away against a node already running. The trace-line submission waits for the next upload, one hour by default:

metsuke 0.2.1 starting
config: {"agent_id":"relay-1","compression_level":0, …}
metsuke on relay-1 scraping http://127.0.0.1:12798/metrics for pool13vscgf9dwn0jt56u965wp99ychz6avktk3pyrye326f3xctz4nm, signing with its cold key
collecting trace lines from the node's stdout: namespaces Consensus.LeiosKernel, Consensus.LeiosPeer, ChainDB.AddBlockEvent.AddedToCurrentChain, Forge.Loop.AdoptedBlock
the first scrape waits for the node to bind its metrics endpoint
submission 1 payload 2ac13de45f3cd39f accepted: 1 scrape, 1902 bytes
submission 2 payload e79180946b65545b accepted: 282 trace lines, 8302 bytes

The second line, cut short above, is every setting the agent is running with, including the defaults for the ones you did not set. Yours runs to about nine hundred characters. Send that line if you ask anyone for help. It answers most of what anyone helping you would otherwise have to ask for, and it holds no key material: the signing key appears as the path it was read from, never its contents.

Without the > /dev/null your node's output comes back, interleaved with the agent's own lines. That pass-through is what the setups below depend on: whatever read your node's output before still reads it.

The config already points at this server, and its metrics URL is the port most nodes already serve Prometheus on. If your node is configured differently, the metrics URL is the second line to change in the config file.

spool_path is the line a run from a shell usually has to change. It ships as a file in the state directory the systemd setups below create, which a shell often cannot write to, and an agent that cannot write its spool says so and stops rather than starting with nowhere to put a scrape. Once the agent has run under a unit of its own, that directory is a symlink only root can follow, so set your own path even though it looks present.

How you'll run it for real

Each setup below costs something different. Everything after the table is the same four steps whichever row you take; only the filenames change.

Setup	Trace line source	What it costs in privilege	Restarts	Metsuke config	Systemd service	Systemd drop-in
Its own unit the usual one	the node's journal	its own user, confined; reads every unit's journal	on its own	config.journald.toml	metsuke-journald.service	none
Your node's unit a drop-in	the node's stdout	your node's user and sandbox; your node can read your signing key	with the node	config.pipe.toml	your node's own	node-pipe.conf
A container no systemd	the node's stdout	whatever your entrypoint runs as	with the node	config.pipe.toml	your own entrypoint	none
Metrics only fallback	nothing	its own user, confined; no group at all	on its own	config.minimal.toml	metsuke.service	none
A shell debugging	the node's stdout	yours, unconfined	not at all	config.pipe.toml	none	none

Its own unit is what most pools want. The agent runs under a unit of its own, so it restarts without touching your node and keeps the confinement that unit sets, and the cost is one group: systemd-journal can read every unit's journal on the host, not only your node's.

Your node's unit costs more than it appears to, in a way worth reading before you take it. The drop-in runs the agent inside your node's unit, so it is your node's user that runs it and your node's sandbox that confines it, not the one metsuke.service sets. The drop-in also loads your signing key as a credential of that unit, which makes it readable by every process the unit runs, cardano-node included. On a block producer signing with its Leios key that costs nothing new, since your node already holds that key; on a relay it is a key the node could not otherwise read. It also ties the agent's lifecycle to your node's, so restarting one restarts the other. The details page covers it in full.

If you take that row, use your Leios key and not a cold key. Both work, but a cold key cannot be replaced: your pool id is its hash, so a compromised one cannot be rotated; it is the end of that pool.

Both of those need a node that already runs as a systemd unit, and Its own unit also needs it logging to the journal, which is where the agent reads its trace lines from. cardano-node.service is an example to adapt where yours has no unit yet. It suits either row: the journald setup reads the journal it writes to, and the drop-in setup replaces the command it runs.

A container is the row for anything that starts your node without systemd: a container entrypoint, but equally a wrapper script, a runit or s6 run file, or a supervisord program. The pipeline goes wherever that command lives. The details page has the two things that setup has to get right itself, which the systemd rows get for free: rotating a log file the agent now holds open, and restarting the pipeline.

The last two are fallbacks rather than places to stay. Metrics only collects no trace lines, and the rewards program wants those, so it is where to drop back to if collection is giving you trouble. A shell is the try-it above: nothing starts it at boot and nothing restarts it, so it is primarily for quick debugging.

1. Install the agent

ARCH=x86_64-linux            # or aarch64-linux

# Download the build for your architecture, and its checksum
curl -fO https://metsuke-leios.play.dev.cardano.org/files/metsuke-static-$ARCH
curl -fO https://metsuke-leios.play.dev.cardano.org/files/metsuke-static-$ARCH.sha256

# Check it is the build this page describes
sha256sum -c metsuke-static-$ARCH.sha256

# Install it where the unit will look for it
sudo install -D -m 0755 metsuke-static-$ARCH /usr/local/bin/metsuke

One static binary, nothing to link against, and the block above fetches the one for your machine. Either architecture can be built rather than downloaded, with nix build github:input-output-hk/metsuke#metsuke-static-$ARCH. There is no install script and no self-update. On NixOS, import the module instead and skip the rest of this page.

2. Configure it

Take the config your setup names in the table above, set your pool id, and put your signing key where the agent will look for it:

# Both the config and signing-key files go here, and nothing has made this
# directory yet
sudo mkdir -p /etc/metsuke

# Download one config file, and only one, from the preferred setup above

# Its own unit
sudo curl -fo /etc/metsuke/config.toml https://metsuke-leios.play.dev.cardano.org/files/config.journald.toml

# Your node's unit, A container, or A shell
sudo curl -fo /etc/metsuke/config.toml https://metsuke-leios.play.dev.cardano.org/files/config.pipe.toml

# Metrics only
sudo curl -fo /etc/metsuke/config.toml https://metsuke-leios.play.dev.cardano.org/files/config.minimal.toml

# Set your pool id, and check the metrics URL. Two setups have more to set:
#   Its own unit  your node's unit, and journalctl's absolute path
#   A shell       a spool_path you can write
sudo $EDITOR /etc/metsuke/config.toml

# Install your signing key, bls.skey or pool.skey, readable by root alone
sudo install -m 0400 -o root -g root bls.skey /etc/metsuke/signing-key

The journald config needs two things the agent cannot guess: your node's unit name, and where journalctl lives on your distribution. They fail differently. A journalctl that is not at the path you gave stops the agent at startup and says so. A unit name that matches nothing does not stop it: journalctl follows it and prints nothing, so the agent starts, ships its scrapes, and never ships a trace line.

agent_id is what a submission calls the machine it came from, and it needs no setting: left unset, it is that host's hostname, which the startup line prints. Set it where the hostname is not the name you want, and in a container, where the hostname is the runtime's rather than yours.

Those permissions are for the three systemd setups, where the unit hands the key over as a credential and the service never reads the file itself. If you took A container or A shell, you point --signing-key at your own copy instead, and it needs to be readable by whoever runs the agent.

Both your Leios key and your cold key work; use your Leios key, for the reason the details page gives.

There is nothing to configure for the trace lines themselves. The config you just took already selects the traces the program asks for, and the published MusashiNet node configuration already emits them. If you have changed your node's tracing, the details page offers further guidance.

3. Run it

The page splits three ways from here, and the row you took decides which one you follow. If you took Its own unit or Metrics only, the agent runs under a unit of its own, and you download one of the two:

# Its own unit
sudo curl -fo /etc/systemd/system/metsuke.service https://metsuke-leios.play.dev.cardano.org/files/metsuke-journald.service

# Metrics only
sudo curl -fo /etc/systemd/system/metsuke.service https://metsuke-leios.play.dev.cardano.org/files/metsuke.service

# Start it now, and on every boot
sudo systemctl daemon-reload
sudo systemctl enable --now metsuke

If you took Your node's unit, the agent gets no unit of its own. A drop-in replaces the command your node's unit already runs with that same command piped through the agent, for the reasons the details page gives.

# The directory is yours to make. The zzzz- name sorts the drop-in last, so
# nothing host-wide can clear its credential after it
sudo mkdir -p /etc/systemd/system/<your-node>.service.d
sudo curl -fo /etc/systemd/system/<your-node>.service.d/zzzz-metsuke.conf https://metsuke-leios.play.dev.cardano.org/files/node-pipe.conf

# Replace YOUR-NODE-COMMAND with what your node already runs, which
# systemctl cat <your-node>.service prints
sudo $EDITOR /etc/systemd/system/<your-node>.service.d/zzzz-metsuke.conf

# Restarting the node is what starts the agent
sudo systemctl daemon-reload
sudo systemctl restart <your-node>

If you took A container or A shell, nothing above applies and nothing starts the agent for you. In a container the pipeline goes in your entrypoint, which the details page shows in full against the MusashiNet node image. In a shell you run the try-it command at the top of this page.

4. Check it

Where the agent's own lines come out depends on your setup:

# Its own unit, or Metrics only
journalctl -u metsuke -f

# Your node's unit: the agent runs inside it, so its lines are the node's
journalctl -u <your-node> -f

# A container
docker logs -f <container>

# A shell: they are already on your terminal

Whichever command you ran, expect the same lines the try-it printed above, though not all at once. The first submission goes out on the first scrape rather than on a cadence, so the startup lines and the scrape line appear within seconds of each other and you do not wait out an interval to find out something is wrong. Trace lines have not accumulated by then, so their submission goes out on the next upload, one hour by default.

A refused submission logs this server's reason in place of one of those accepted lines, and the scrapes stay spooled, so nothing is lost while you fix it.

If you took Its own unit or Metrics only, a failure naming cannot read signing key under /run/credentials is usually the unit's credential being cleared rather than a permission problem. systemctl cat metsuke prints the unit and every drop-in that applies to it, including the host-wide ones under /usr/lib/systemd/system/service.d/ that the LXC and LXD images ship. A bare LoadCredential= in one of those resets the list after your unit's own line. Restore it in a drop-in that sorts after theirs.

sudo mkdir -p /etc/systemd/system/metsuke.service.d
sudo tee /etc/systemd/system/metsuke.service.d/zzzz-credential.conf >/dev/null <<'EOF'
[Service]
LoadCredential=signing-key:/etc/metsuke/signing-key
EOF

sudo systemctl daemon-reload
sudo systemctl restart metsuke

Your node's unit is already out of this: the drop-in you installed is named to sort after anything host-wide, which is what the zzzz- is for.

Staying up to date

This server was built against agent 0.2.1 and reports that version to every agent that uploads to it. Your agent logs a warning when its own version is older. To update, repeat step 1, then restart whatever runs it:

# Its own unit, or Metrics only
sudo systemctl restart metsuke

# Your node's unit: the agent restarts with the node
sudo systemctl restart <your-node>

# A container
docker restart <container>

# A shell: run the try-it command again

The spool is on disk, so queued scrapes survive the restart. Nothing is scraped while the agent is down.