Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Getting Started with EJ

Welcome to EJ - a modular and scalable framework for automated testing on physical embedded boards!

What is EJ?

EJ enables dispatching tests to real hardware, collecting logs, test results, and detect hardware-specific issues. The framework is designed to support diverse board architectures and simplify distributed hardware testing for embedded projects.

Key Components

EJ consists of two main applications and several supporting libraries:

Core Applications:

  • EJB (EJ Builder) - Manages build processes and board communication
  • EJD (EJ Dispatcher) - Handles job queuing, distribution, and result collection
  • EJCli (EJ CLI) - Helper cli tool to interface with EJD

Libraries:

  • ej-builder-sdk - Interface library for creating custom builder applications
  • ej-dispatcher-sdk - Interface library for interfacing with dispatchers
  • ej-auth - Authentication utilities (JWT management, password hashing)
  • ej-config - Shared configuration structures and utilities
  • ej-io - Program management utilities
  • ej-models - Database models for EJ
  • ej-requests - HTTP request handling utilities
  • ej-web - Internal web utilities for the dispatcher

Architecture Overview

EJ follows a tree-like distributed architecture:

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Git Repo      │    │   CI/CD         │    │   Developer     │
│                 │    │                 │    │                 │
└─────────┬───────┘    └─────────┬───────┘    └─────────┬───────┘
          │                      │                      │
          │                      │                      │
          └──────────────────────┼──────────────────────┘
                                 │
                    ┌────────────▼────────────┐
                    │   EJD (Dispatcher)      │
                    │                         │
                    │  - Job Queuing          │
                    │  - Result Storage       │
                    │  - Authentication       │
                    └────────────┬────────────┘
                                 │
                 ┌───────────────┼───────────────┐
                 │               │               │
    ┌────────────▼────────────┐  │  ┌────────────▼────────────┐
    │   EJB (Builder 1)       │  │  │   EJB (Builder 2)       │
    │                         │  │  │                         │
    │  - Build Management     │  │  │  - Build Management     │
    │  - Board Communication  │  │  │  - Board Communication  │
    └────────────┬────────────┘  │  └────────────┬────────────┘
                 │               │               │
         ┌───────┼─────────┐     │       ┌───────┼────────┐
         │       │         │     │       │       │        │
    ┌────▼──┐┌───▼───┐ ┌───▼───┐ │  ┌────▼──┐┌───▼───┐┌───▼───┐
    │Board 1││Board 2│ │Board 3│ │  │Board 4││Board 5││Board 6│
    │(RPi4) ││(ESP32)│ │(PC)   │ │  │(RPi3) ││(STM32)││(x86)  │
    └───────┘└───────┘ └───────┘ │  └───────┘└───────┘└───────┘

Design Philosophy

EJ doesn't make assumptions about how to build, run, or manage your test results. This flexibility is achieved through:

  • Builder SDK - Create custom build and run scripts with seamless builder communication and job cancellation support
  • Dispatcher SDK - Interface with the dispatcher to dispatch jobs and retrieve results

This gives us complete control over how tests are built and deployed, how results are parsed, and board-specific configurations and requirements.

Guide Structure

This guide series will walk us through setting up and using EJ from basic to advanced scenarios:

01 - Builder

Learn how to set up our first EJ Builder with a basic configuration. We'll deploy a simple application to a Raspberry Pi as a practical example, covering:

  • Installing and configuring EJB
  • Creating your first board configuration
  • Writing build and run scripts
  • Deploying and testing a simple application

02 - Builder SDK

Discover why the Builder SDK exists and how it solves common deployment issues. We'll explore:

  • The problems that can arise if we aren't careful with automatic deployments
  • How the Builder SDK provides better control and monitoring
  • Migrating from basic scripts to SDK-based solutions

03 - Dispatcher

Set up a centralized job management system with EJD. This guide covers:

  • Installing and configuring the EJ Dispatcher
  • Connecting builders to the dispatcher
  • Managing jobs, queues, and results

04 - Dispatcher SDK

Create custom tools to interface with your dispatcher. Learn how to:

  • Build a custom CLI tool using the Dispatcher SDK
  • Submit jobs programmatically
  • Parse and analyze results

Prerequisites

Before starting with the guides, ensure you have:

  • Rust toolchain (latest stable version)
  • Target hardware (Raspberry Pi recommended for examples)
  • SSH access to your target boards
  • Basic command line familiarity
  • Git for version control

Next Steps

Ready to get started? Head over to Guide 01 - Builder to set up your first EJ Builder and deploy your first application!

Getting Help

  • Issues: Report bugs and request features on our GitHub repository
  • Documentation: Check the README.md and our crates in crates.io for API references
  • Examples: Explore the examples/ directory for configuration templates

EJ was originally designed and built for a bachelor thesis to provide LVGL with automatic performance measurement in CI. The architecture supports both small-scale local testing and large-scale distributed testing scenarios.

Guide 01: Setting Up Our First EJ Builder

This guide will walk us through setting up our first EJ Builder (EJB) and deploying a simple application to a Raspberry Pi. By the end of this guide, we'll have a working EJ Builder that can build and run applications on physical hardware.

Overview

In this guide, we'll:

  1. Install and configure EJB
  2. Set up a Raspberry Pi target board
  3. Configure board settings and deployment scripts
  4. Test the complete build and deployment workflow

Prerequisites

Before starting, ensure you have:

  • A Raspberry Pi with Raspberry Pi OS 64 bit installed
  • SSH access to your Raspberry Pi
  • Cargo installed on your host machine
  • The AArch64 GNU/Linux toolchain in your PATH.
  • Basic familiarity with SSH and shell scripting is a bonus

Application example: K-mer Algorithm Performance Benchmark

For this guide, we'll use a k-mer counting application that:

  • Processes the digits of PI to count k-mer occurrences
  • Measures execution time and memory usage
  • Outputs performance metrics to stdout
  • Demonstrates computational differences between platforms

Note: K-mer algorithms are typically used in bioinformatics for analyzing DNA sequences (Wikipedia: K-mer). For this guide, we use the digits of PI as our input sequence since it provides a deterministic, easily reproducible dataset that still demonstrates the algorithm's computational characteristics.

This example showcases:

  • Cross-platform deployment (development machine to Raspberry Pi)
  • Performance measurement (timing and resource usage)
  • Result collection (stdout capture)
  • Real computational workload (pattern counting algorithm)

Step 1: Clone the kmer application

Application Code

We provide multiple versions of a kmer application

  • An unoptimized version
  • A single-threaded optimized version
  • A multi-threaded optimized version
mkdir -p ~/ej-workspace
cd ~/ej-workspace
git clone https://github.com/embj-org/kmer.git
cd kmer

Cross compile the application

This is to ensure everything is working properly

cmake -B build-pi \
      -DCMAKE_TOOLCHAIN_FILE=aarch64_toolchain.cmake
cmake --build build-pi -j$(nproc)

Test the application

PI_USERNAME=<your_pi_username>
PI_ADDRESS=<your_pi_ip_address>

scp -r build-pi/k-mer-omp inputs ${PI_USERNAME}@${PI_ADDRESS}:~
ssh ${PI_USERNAME}@${PI_ADDRESS} "./k-mer-omp inputs/pi_dec_1k.txt 3"

If any of these steps fail, ensure you have the correct toolchain installed and available in your PATH.

Now that we've ensured everything is working, it's time to use EJ.

Step 2: Install EJB

EJB is available in crates.io so you can install it like this:

cargo install ejb

Step 3: Create a build and run script

Inside ~/ej-workspace, create the following scripts:

Build Script (build.sh)

This script is responsible for building the application. We already did this previously so we can simply copy the same steps as before.

#!/bin/bash
set -e

SCRIPT=$(readlink -f $0)
SCRIPTPATH=$(dirname $SCRIPT)

cmake -B ${SCRIPTPATH}/kmer/build-pi \
      -S ${SCRIPTPATH}/kmer \
      -DCMAKE_TOOLCHAIN_FILE=${SCRIPTPATH}/kmer/aarch64_toolchain.cmake

cmake --build ${SCRIPTPATH}/kmer/build-pi -j$(nproc)

Run Script (run.sh)

Same thing for the run script but right now what we'll be doing is only testing the original implementation. Additionally, we need to output the program results to a file so they can be used later. Finally, besides the results we'll actually time the application:

#!/bin/bash
set -e

PI_USERNAME=<your_pi_username>
PI_ADDRESS=<your_pi_ip_address>

SCRIPT=$(readlink -f $0)
SCRIPTPATH=$(dirname $SCRIPT)

scp -r ${SCRIPTPATH}/kmer/build-pi/k-mer-original \
    ${SCRIPTPATH}/kmer/inputs ${PI_USERNAME}@${PI_ADDRESS}:~

ssh ${PI_USERNAME}@${PI_ADDRESS} \
    "time ./k-mer-original inputs/input.txt 3" 2>&1 | tee ${SCRIPTPATH}/results.txt

Making Scripts Executable

cd ~/ej-workspace
chmod +x build.sh run.sh

Step 4: Configuring EJB

Board Configuration

Inside ~/ej-workspace

Create your config.toml. This file is responsible for describing every board and every board configuration EJB should handle.

We'll start with a very simple config file that describes our single board and a single config.

NOTE: Replace <user> with your username.

[global]
version = "1.0.0"

[[boards]]
name = "Raspberry Pi"
description = "Raspberry Pi with Raspberry OS 64 bits"

[[boards.configs]]
name = "k-mer-original"
tags = ["arm64", "kmer unoptimized"]
build_script = "/home/<user>/ej-workspace/build.sh"
run_script = "/home/<user>/ej-workspace/run.sh"
results_path = "/home/<user>/ej-workspace/results_k-mer-original.txt"
library_path = "/home/<user>/ej-workspace/kmer"

You may notice the board config name, the results path and the executable all have the same name. This is NOT mandatory but will make it easier to build upon later on.

Configuration Explanation

  • Board Definition: Describes your Raspberry Pi hardware
  • Config Section: Defines how to build and run the k-mer benchmark
  • Scripts: Point to your build and run scripts
  • Results Path: Where EJB will look for captured stdout output
  • Tags: Help categorize and filter boards

Step 5: Testing the config

Parse the config

EJB can be used to parse the file to make sure the config is correct

ejb --config config.toml parse

You should see the following output:

Configuration parsed successfully
Global version: 1.0.0
Number of boards: 1

Board 1: Raspberry Pi
  Description: Raspberry Pi with Raspberry OS 64 bits
  Configurations: 1
    Config 1: k-mer-original
      Tags: ["arm64", "kmer unoptimized"]
      Build script: "/home/andre/ej-workspace/build.sh"
      Run script: "/home/andre/ej-workspace/run.sh"
      Results path: "/home/andre/ej-workspace/results_k-mer-original.txt"
      Library path: "/home/andre/ej-workspace/kmer"

Test the config

You can now use EJB to run the described tests for you:

ejb --config config.toml validate

You should see this output followed by the test results

Validating configuration file: "config.toml"
2025-07-10T12:30:08.401673Z  INFO ejb::build: Board 1/1: Raspberry Pi
2025-07-10T12:30:08.401682Z  INFO ejb::build: Config 1: k-mer-original
2025-07-10T12:30:08.401882Z  INFO ejb::build: Raspberry Pi - k-mer-original Build started
2025-07-10T12:30:08.512101Z  INFO ejb::build: Raspberry Pi - k-mer-original Build ended successfully
2025-07-10T12:30:08.512427Z  INFO ejb::run: k-mer-original - Run started
2025-07-10T12:30:10.054405Z  INFO ejb::run: k-mer-original - Run ended successfully
2025-07-10T12:30:10.054516Z  INFO ejb::run: Found results for Raspberry Pi - k-mer-original
========================
Log outputs for Raspberry Pi k-mer-original
========================
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /home/andre/ej-workspace/kmer/build-pi
[ 50%] Built target k-mer
[ 66%] Built target k-mer-original
[100%] Built target k-mer-omp
Results:
ABC: 2
BCD: 1
CDA: 1
DAB: 1

real	0m0.005s
user	0m0.000s
sys	0m0.005s

========================
Result for Raspberry Pi k-mer-original
========================
Results:
ABC: 2
BCD: 1
CDA: 1
DAB: 1

real	0m0.005s
user	0m0.000s
sys	0m0.005s

Step 6: Adding more configs

We showcased a pretty simple example to see how to setup one board with one config. In reality, EJ is equipped to handle multiple boards with multiple configs each.

The kmer repository contains 3 versions of the same software, so let's use EJ to actually test the three versions but before, let's look into how how EJB launches our application.

Script Arguments

The scripts we created were pretty basic and only do one thing. We can easily imagine this becoming cumbersome very quickly.

EJ solves this problem by having EJB launch your build and run scripts with some arguments:

  • argv[0]: Your script name
  • argv[1]: Action (build or run)
  • argv[2]: Config file path
  • argv[3]: Board name
  • argv[4]: Board config name
  • argv[5]: Socket path for EJB communication. We'll be discussing this one further in a following guide.

With these arguments, we can actually create a more sophisticated script to handle every config for us.

Let's modify our run.sh script to handle this for us:

#!/bin/bash
set -e
PI_USERNAME=<your_pi_username>
PI_ADDRESS=<your_pi_ip_address>

SCRIPT=$(readlink -f $0)
SCRIPTPATH=$(dirname $SCRIPT)

BOARD_CONFIG_NAME=$4

scp -r ${SCRIPTPATH}/kmer/build-pi/${BOARD_CONFIG_NAME} \
    ${SCRIPTPATH}/kmer/inputs ${PI_USERNAME}@${PI_ADDRESS}:~

ssh ${PI_USERNAME}@${PI_ADDRESS} \
    "time ./${BOARD_CONFIG_NAME} inputs/pi_dec_1k.txt 3" 2>&1 | tee ${SCRIPTPATH}/results_${BOARD_CONFIG_NAME}.txt

Here, by using the board config name that is automatically passed to us by EJB (argv[4]), we can now use the same script for every board config.

This is the reason we matched the application name, the results path and the board config name in our config.toml earlier.

Add these new config entries at the bottom of your ~/ej-workspace/config.toml:

NOTE: Replace <user> with your username.

[[boards.configs]]
name = "k-mer"
tags = ["arm64", "kmer optimized"]
build_script = "/home/<user>/ej-workspace/build.sh"
run_script = "/home/<user>/ej-workspace/run.sh"
results_path = "/home/<user>/ej-workspace/results_k-mer.txt"
library_path = "/home/<user>/ej-workspace/kmer"

[[boards.configs]]
name = "k-mer-omp"
tags = ["arm64", "kmer multi-threaded optimized"]
build_script = "/home/<user>/ej-workspace/build.sh"
run_script = "/home/<user>/ej-workspace/run.sh"
results_path = "/home/<user>/ej-workspace/results_k-mer-omp.txt"
library_path = "/home/<user>/ej-workspace/kmer"

Don't hesitate to use ejb to parse your config and make sure it can be parsed written.

With this new config we can now run ejb again and we'll see that it runs all three configs:

ejb --config config.toml validate

ejb --config config.toml validate
Validating configuration file: "config.toml"
2025-07-10T12:33:02.582019Z  INFO ejb::build: Board 1/1: Raspberry Pi
2025-07-10T12:33:02.582045Z  INFO ejb::build: Config 1: k-mer-original
2025-07-10T12:33:02.582278Z  INFO ejb::build: Raspberry Pi - k-mer-original Build started
2025-07-10T12:33:02.692504Z  INFO ejb::build: Raspberry Pi - k-mer-original Build ended successfully
2025-07-10T12:33:02.692524Z  INFO ejb::build: Config 2: k-mer
2025-07-10T12:33:02.692779Z  INFO ejb::build: Raspberry Pi - k-mer Build started
2025-07-10T12:33:02.802979Z  INFO ejb::build: Raspberry Pi - k-mer Build ended successfully
2025-07-10T12:33:02.803001Z  INFO ejb::build: Config 3: k-mer-omp
2025-07-10T12:33:02.803285Z  INFO ejb::build: Raspberry Pi - k-mer-omp Build started
2025-07-10T12:33:02.913480Z  INFO ejb::build: Raspberry Pi - k-mer-omp Build ended successfully
2025-07-10T12:33:02.913827Z  INFO ejb::run: k-mer-original - Run started
2025-07-10T12:33:04.675982Z  INFO ejb::run: k-mer-original - Run ended successfully
2025-07-10T12:33:04.676299Z  INFO ejb::run: k-mer - Run started
2025-07-10T12:33:06.328239Z  INFO ejb::run: k-mer - Run ended successfully
2025-07-10T12:33:06.328546Z  INFO ejb::run: k-mer-omp - Run started
2025-07-10T12:33:07.870387Z  INFO ejb::run: k-mer-omp - Run ended successfully
2025-07-10T12:33:07.870464Z  INFO ejb::run: Found results for Raspberry Pi - k-mer-omp
2025-07-10T12:33:07.870479Z  INFO ejb::run: Found results for Raspberry Pi - k-mer-original
2025-07-10T12:33:07.870484Z  INFO ejb::run: Found results for Raspberry Pi - k-mer
========================
Log outputs for Raspberry Pi k-mer-original
========================
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /home/andre/ej-workspace/kmer/build-pi
[ 33%] Built target k-mer-original
[100%] Built target k-mer-omp
[100%] Built target k-mer
Results:
ABC: 2
BCD: 1
CDA: 1
DAB: 1

real	0m0.005s
user	0m0.000s
sys	0m0.005s

========================
Result for Raspberry Pi k-mer-original
========================
Results:
ABC: 2
BCD: 1
CDA: 1
DAB: 1

real	0m0.005s
user	0m0.000s
sys	0m0.005s

========================
Log outputs for Raspberry Pi k-mer
========================
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /home/andre/ej-workspace/kmer/build-pi
[ 33%] Built target k-mer-omp
[100%] Built target k-mer-original
[100%] Built target k-mer
Results:
ABC: 2
BCD: 1
CDA: 1
DAB: 1

real	0m0.005s
user	0m0.004s
sys	0m0.001s

========================
Result for Raspberry Pi k-mer
========================
Results:
ABC: 2
BCD: 1
CDA: 1
DAB: 1

real	0m0.005s
user	0m0.004s
sys	0m0.001s

========================
Log outputs for Raspberry Pi k-mer-omp
========================
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /home/andre/ej-workspace/kmer/build-pi
[ 33%] Built target k-mer-original
[ 66%] Built target k-mer
[100%] Built target k-mer-omp
Results:
ABC: 2
BCD: 1
CDA: 1
DAB: 1

real	0m0.008s
user	0m0.005s
sys	0m0.006s

========================
Result for Raspberry Pi k-mer-omp
========================
Results:
ABC: 2
BCD: 1
CDA: 1
DAB: 1

real	0m0.008s
user	0m0.005s
sys	0m0.006s

Understanding what just happened

When we run a job, EJB follows this process:

1. Build phase

EJB executes each build script sequentially

Build scripts are run sequentially. This allows us to use every available core to speed up our build process for each individual config.

Don't share build folders for multiple configs as EJB won't run our config before every other config has finished building.

2. Execution phase

EJB executes each run script

Run scripts are run in parallel across different boards and sequentially for each board config.

We must not share the same results path between multiple boards to avoid race conditions.

3. Result collection

EJB collects the results from the results_path once the run finishes. This phase happens at the same time as phase 2, the results are collected once the corresponding run script finishes.

We can use whatever we want as a way to represent our test results, EJ will simply collect what's inside the results_path at the moment the run_script ends.

Next Steps

Congratulations! We now have a very simple but working EJ Builder setup which can already be used to automate our testing environment. The same way we created our Raspberry PI board, we could've just as easily added more board descriptions, there's no limit to how many boards and configs EJB can manage.

The simple shell script approach, although easy to setup, has some limitations, even for this simple example. If you can't think of any, don't worry, we'll dive into those in Guide 02 - Builder SDK which will present these issues and how the Builder SDK solves them.

Guide 02: Understanding and Using the Builder SDK

You can find the SDK's documentation in crates.io

Overview

In the previous guide, we successfully set up a basic EJ Builder using shell scripts to deploy and test applications on embedded hardware. While this approach works well for simple scenarios, we may have encountered some limitations - particularly around handling long-running processes or cleaning up after interrupted tests.

This guide explores those limitations and demonstrates how the EJ Builder SDK provides robust solutions for production deployments. We'll learn how to convert our shell scripts into a proper Rust application that can handle job cancellation, manage resources properly, and integrate seamlessly with advanced EJ features.

By the end of this guide, we'll have a production-ready builder setup that can handle complex deployment scenarios with confidence.

Prerequisites

Before starting this guide, ensure you have:

  • Completed Guide 01: This guide builds directly on the EJ Builder setup from the previous guide
  • Rust toolchain installed: We'll need cargo and the Rust compiler
    • Install via rustup.rs if we haven't already
    • No prior Rust experience required - the guide explains all concepts as we go
  • Our working EJ Builder setup: From the previous guide, including:
    • The kmer project configured and working
    • SSH access to your target device (Raspberry Pi)
    • The config.toml file with your board configurations
  • Understanding of the shell script approach: You should have successfully run the previous guide's shell scripts

The Problem with Basic Script Deployment

In the previous guide, we set up a basic EJ Builder using shell scripts. While this approach works for simple scenarios, you may have noticed some limitations.

Let's revisit what happens when we deploy applications using basic SSH and shell scripts, particularly with our Raspberry Pi example from Guide 01.

For this, let's add this new config to our ~/ej-workspace/config.toml:

NOTE: Replace <user> with your username.

[[boards.configs]]
name = "infinite-loop"
tags = ["arm64", "infinite-loop"]
build_script = "/home/<user>/ej-workspace/build.sh"
run_script = "/home/<user>/ej-workspace/run.sh"
results_path = "/home/<user>/ej-workspace/results_infinite-loop.txt"
library_path = "/home/<user>/ej-workspace/kmer"

The application we will run enters an infinite loop, meaning the application will never exit.

ejb --config config.toml validate

It won't take long to see the problem:

Validating configuration file: "config.toml"
2025-07-10T13:19:55.029647Z  INFO ejb::build: Board 1/1: Raspberry Pi
2025-07-10T13:19:55.029655Z  INFO ejb::build: Config 1: k-mer-original
2025-07-10T13:19:55.029879Z  INFO ejb::build: Raspberry Pi - k-mer-original Build started
2025-07-10T13:19:55.140067Z  INFO ejb::build: Raspberry Pi - k-mer-original Build ended successfully
2025-07-10T13:19:55.140084Z  INFO ejb::build: Config 2: k-mer
2025-07-10T13:19:55.140310Z  INFO ejb::build: Raspberry Pi - k-mer Build started
2025-07-10T13:19:55.250487Z  INFO ejb::build: Raspberry Pi - k-mer Build ended successfully
2025-07-10T13:19:55.250504Z  INFO ejb::build: Config 3: k-mer-omp
2025-07-10T13:19:55.250722Z  INFO ejb::build: Raspberry Pi - k-mer-omp Build started
2025-07-10T13:19:55.360908Z  INFO ejb::build: Raspberry Pi - k-mer-omp Build ended successfully
2025-07-10T13:19:55.360933Z  INFO ejb::build: Config 4: infinite-loop
2025-07-10T13:19:55.361238Z  INFO ejb::build: Raspberry Pi - infinite-loop Build started
2025-07-10T13:19:55.471432Z  INFO ejb::build: Raspberry Pi - infinite-loop Build ended successfully
2025-07-10T13:19:55.471698Z  INFO ejb::run: k-mer-original - Run started
2025-07-10T13:19:56.903312Z  INFO ejb::run: k-mer-original - Run ended successfully
2025-07-10T13:19:56.903571Z  INFO ejb::run: k-mer - Run started
2025-07-10T13:19:58.114931Z  INFO ejb::run: k-mer - Run ended successfully
2025-07-10T13:19:58.115205Z  INFO ejb::run: k-mer-omp - Run started
2025-07-10T13:19:59.326567Z  INFO ejb::run: k-mer-omp - Run ended successfully
2025-07-10T13:19:59.326831Z  INFO ejb::run: infinite-loop - Run started

The underlying application entered an infinite loop and thus both EJB and our run.sh script are stuck forever waiting for it to end.

To quit it, we can press CTRL+C essentially killing EJB and the run.sh script process that is holding the ssh connection.

Now if we run the validation again:

ejb --config config.toml validate

Something we may not expect happens - EJB (or rather the underlying run.sh script) fails!

Taking a look at the logs, we can see that scp failed because the infinite-loop file is locked as it's still being executed inside our target device.

scp: dest open "./infinite-loop": Failure
scp: failed to upload file /home/andre/ej-workspace/kmer/build-pi/infinite-loop to ~

To actually stop it we need to connect to our Raspberry Pi and kill it:

ssh ${PI_USERNAME}@${PI_ADDRESS} "killall infinite-loop"

This poses a real problem when deploying this to a production environment as we'd like to make sure that if one job fails, we want to be able to simply consider this job as a failure and run a new job later without having to manually connect to our target board to clean up failed job leftovers.

EJ solves this problem by providing an SDK - called EJ Builder SDK - that handles communicating with EJB through an exposed Unix Socket. The Builder SDK abstracts all of this for us with some pretty simple boilerplate code. Let's create a script to see what it looks like.

Step 1: Setup an application with the EJB SDK

cd ~/ej-workspace
cargo init --bin ejkmer-builder
cd ejkmer-builder
cargo add ej-builder-sdk # EJB SDK
cargo add tokio -F macros -F rt-multi-thread -F process # Async runtime
cargo add num_cpus # (Optional) Used to be able to write -j$(nprocs) during the build phase

Now let's add this boilerplate code to our src/main.rs file:

use ej_builder_sdk::{Action, BuilderEvent, BuilderSdk, prelude::*};

#[tokio::main]
async fn main() -> Result<()> {
    let sdk = BuilderSdk::init(|sdk, event| async move {
        match event {
            BuilderEvent::Exit => todo!("Handle exit command"),
        }
    })
    .await?;

    match sdk.action() {
        Action::Build => todo!("Handle build command"),
        Action::Run => todo!("Handle run command"),
    }
}

Let's go through each line of code to understand what it's going on:

#![allow(unused)]
fn main() {
use ej_builder_sdk::{Action, BuilderEvent, BuilderSdk, prelude::*};
}

This line is including stuff we need from the ej_builder_sdk that we added to our project when we ran the cargo add ej-builder-sdk command.

  • Action: is a Rust Enum used to describe the action this script should take (either Build or Run). This lets us use the same script as our build and run script - although this isn't mandatory.

  • BuilderEvent: is a Rust Enum that describe an Event received by EJB. For now, the only event we can expect is Exit but there may be others in the future as EJ evolves.

  • BuilderSDK: is the main BuilderSDK data structure, it will contain every information passed by EJB, this includes:

    • The action to take (Build or Run)
    • The path to the config.toml file
    • The current board name
    • The current board config name

    These informations allow us to use a single script to handle building and testing our application throughout multiple boards and configs.

  • prelude::*: is the BuilderSDK crate prelude that imports a Result and a common Error type that can be used by your script.

#[tokio::main]
async fn main() -> Result<()> {

These two lines allow us to describe our main function as Asynchronous - Wikipedia.

BuilderSDK uses async tasks under the hood to manage the connection with EJB in a transparent way so this will allow us to call these functions.

The return type of our main function is the Result type. This Result type is pulled from the BuilderSDK prelude and uses its internal Error type as the Result error type. This allows us to use the ? operator to easily handle errors in our application.

#![allow(unused)]
fn main() {
    let sdk = BuilderSdk::init(|sdk, event| async move {
        match event {
            BuilderEvent::Exit => todo!("Handle exit command"),
        }
    })
    .await?;
}

This portion of code initializes the BuilderSDK. The return type will be a BuilderSDK or an error if something went wrong during the initialization process.

The BuilderSDK::init function takes in an async function callback that will be called when it receives a new event from EJB.

This lets us handle these events the way we see fit (e.g., by killing the process in our target board when we receive an exit request).

The .await is necessary because the init function is async, this essentially tells the program to wait for the execution of this call instead of deferring it for later.

The ? operator will return from the main function (and thus the application) if the BuilderSDK::init function returns an error. In this case, the exit code of the application will be non-zero.

#![allow(unused)]
fn main() {
    match sdk.action() {
        Action::Build => todo!("Handle build command"),
        Action::Run => todo!("Handle run command"),
    }
}

Now that we've initialized everything, the sdk variable holds every information passed by EJB. We can use it to query the action to take, the path to the config file, the board name and the board config name allowing us to create generic scripts that handle our build and deployment needs.

Here we are matching on the action to take (either Action::Build or Action::Run).

Let's now write what the application should do when asked to build and run our application.

Step 2: Convert our build shell script to Rust code

As a reminder, our current build script looks like this:

cmake -B ${SCRIPTPATH}/kmer/build-pi \
      -S ${SCRIPTPATH}/kmer \
      -DCMAKE_TOOLCHAIN_FILE=${SCRIPTPATH}/kmer/aarch64_toolchain.cmake

cmake --build ${SCRIPTPATH}/kmer/build-pi -j$(nproc)

First off let's write some utility functions to manage the paths we need for this. As a reminder, EJB provides us with the absolute path to our config.toml, following the directory structure we set up for our project we can find the workspace folder as the parent of our config.toml file:

#![allow(unused)]
fn main() {
use std::path::{Path, PathBuf}

fn workspace_folder(config_path: &Path) -> PathBuf {
    config_path
        .parent()
        .expect(&format!(
            "Failed to get folder containing `config.toml` - Config path is: {}",
            config_path.display()
        ))
        .to_path_buf()
}
}

The source folder sits inside our workspace folder:

#![allow(unused)]
fn main() {
fn source_folder(config_path: &Path) -> PathBuf {
    workspace_folder(config_path).join("kmer")
}
}

And our build folder and toolchain file sit inside the source folder:

#![allow(unused)]
fn main() {
fn build_folder(config_path: &Path) -> PathBuf {
    source_folder(config_path).join("build-pi")
}

fn toolchain_file(config_path: &Path) -> PathBuf {
    source_folder(config_path).join("aarch64_toolchain.cmake")
}
}

Once we have these helper functions we can write a very elegant build function:

#![allow(unused)]
fn main() {
use tokio::process::Command;

async fn build_application(sdk: &BuilderSdk) -> Result<()> {
    let config_path = &sdk.config_path();
    let status = Command::new("cmake")
        .arg("-B")
        .arg(build_folder(config_path))
        .arg("-S")
        .arg(source_folder(config_path))
        .arg(&format!(
            "-DCMAKE_TOOLCHAIN_FILE={}",
            toolchain_file(config_path).display()
        ))
        .spawn()?
        .wait()
        .await?;

    assert!(status.success(), "CMake execution failed");

    Command::new("cmake")
        .arg("--build")
        .arg(build_folder(config_path))
        .arg("-j")
        .arg(num_cpus::get().to_string())
        .spawn()?
        .wait()
        .await?;

    assert!(status.success(), "Build failed");
    Ok(())
}
}

NOTE: We use the tokio::process module instead of std::process to keep our code async. Be careful calling sync functions from async code. We recommend this tokio guide that explains how to bridge the two if you're interested.

Step 3: Convert our run shell script to Rust code

Following the same process with the helper functions we can write similar to our original shell script:

scp -r ${SCRIPTPATH}/kmer/build-pi/${BOARD_CONFIG_NAME} \
    ${SCRIPTPATH}/kmer/inputs ${PI_USERNAME}@${PI_ADDRESS}:~

ssh ${PI_USERNAME}@${PI_ADDRESS} \
    "time ./${BOARD_CONFIG_NAME} inputs/input.txt 3" 2>&1 | tee ${SCRIPTPATH}/results_${BOARD_CONFIG_NAME}.txt

#![allow(unused)]
fn main() {
const PI_USERNAME: &str = "";
const PI_ADDRESS: &str = "";

fn application_path(config_path: &Path, application_name: &str) -> PathBuf {
    build_folder(config_path).join(application_name)
}

fn inputs_path(config_path: &Path) -> PathBuf {
    source_folder(config_path).join("inputs")
}

fn results_path(config_path: &Path, application_name: &str) -> PathBuf {
    workspace_folder(config_path).join(format!("results_{}", application_name))
}
async fn run_application(sdk: &BuilderSdk) -> Result<()> {
    let config_path = &sdk.config_path();
    let app_name = &sdk.board_config_name();

    let result = Command::new("scp")
        .arg("-r")
        .arg(application_path(config_path, app_name))
        .arg(inputs_path(config_path))
        .arg(&format!("{PI_USERNAME}@{PI_ADDRESS}:~"))
        .spawn()?
        .wait()
        .await?;

    assert!(result.success(), "SCP execution failed");

    let result = Command::new("ssh")
        .arg(&format!("{}@{}", PI_USERNAME, PI_ADDRESS))
        .arg(&format!("time ./{} inputs/input.txt 3", app_name))
        .spawn()?
        .wait_with_output()
        .await?;

    let stdout = String::from_utf8_lossy(&result.stdout);
    let stderr = String::from_utf8_lossy(&result.stderr);

    assert!(result.status.success(), "SSH execution failed");

    std::fs::write(
        results_path(config_path, app_name),
        format!("{}\n{}", stdout, stderr),
    )?;

    Ok(())
}
}

Step 4: Handling cancellation using the EJ Builder SDK

Finally, the reason we started the journey of writing a Rust program instead of a shell script was to be able to handle cancelling our job correctly to not leave a process running forever in our Raspberry Pi.

Here we can open a new SSH connection to kill the process running on our target board, the same way we did manually before:

#![allow(unused)]
fn main() {
async fn kill_application_in_rpi(sdk: &BuilderSdk) -> Result<()> {
    let result = Command::new("ssh")
        .arg(format!("{PI_USERNAME}@{PI_ADDRESS}"))
        .arg(format!("killall {}", sdk.board_config_name()))
        .spawn()?
        .wait()
        .await?;
    assert!(result.success(), "Failed to kill process in RPI");
    Ok(())
}
}

Step 5: Putting it all together

Using our new functions, we can finish off writing our main application:

NOTE: Replace PI_USERNAME and PI_ADDRESS with their corresponding values.

use std::path::{Path, PathBuf};
use tokio::process::Command;

use ej_builder_sdk::{Action, BuilderEvent, BuilderSdk, prelude::*};

const PI_USERNAME: &str = "";
const PI_ADDRESS: &str = "";

async fn kill_application_in_rpi(sdk: &BuilderSdk) -> Result<()> {
    let result = Command::new("ssh")
        .arg(format!("{PI_USERNAME}@{PI_ADDRESS}"))
        .arg(format!("killall {}", sdk.board_config_name()))
        .spawn()?
        .wait()
        .await?;
    assert!(result.success(), "Failed to kill process in RPI");
    Ok(())
}
fn workspace_folder(config_path: &Path) -> PathBuf {
    config_path
        .parent()
        .expect(&format!(
            "Failed to get folder containing `config.toml` - Config path is: {}",
            config_path.display()
        ))
        .to_path_buf()
}

fn source_folder(config_path: &Path) -> PathBuf {
    workspace_folder(config_path).join("kmer")
}

fn build_folder(config_path: &Path) -> PathBuf {
    source_folder(config_path).join("build-pi")
}
fn toolchain_file(config_path: &Path) -> PathBuf {
    source_folder(config_path).join("aarch64_toolchain.cmake")
}
fn application_path(config_path: &Path, application_name: &str) -> PathBuf {
    build_folder(config_path).join(application_name)
}

fn inputs_path(config_path: &Path) -> PathBuf {
    source_folder(config_path).join("inputs")
}

fn results_path(config_path: &Path, application_name: &str) -> PathBuf {
    workspace_folder(config_path).join(format!("results_{}", application_name))
}
async fn run_application(sdk: &BuilderSdk) -> Result<()> {
    let config_path = &sdk.config_path();
    let app_name = &sdk.board_config_name();

    let result = Command::new("scp")
        .arg("-r")
        .arg(application_path(config_path, app_name))
        .arg(inputs_path(config_path))
        .arg(&format!("{PI_USERNAME}@{PI_ADDRESS}:~"))
        .spawn()?
        .wait()
        .await?;

    assert!(result.success(), "SCP execution failed");

    let result = Command::new("ssh")
        .arg(&format!("{}@{}", PI_USERNAME, PI_ADDRESS))
        .arg(&format!("time ./{} inputs/input.txt 3", app_name))
        .spawn()?
        .wait_with_output()
        .await?;

    let stdout = String::from_utf8_lossy(&result.stdout);
    let stderr = String::from_utf8_lossy(&result.stderr);

    assert!(result.status.success(), "SSH execution failed");

    std::fs::write(
        results_path(config_path, app_name),
        format!("{}\n{}", stdout, stderr),
    )?;

    Ok(())
}
async fn build_application(sdk: &BuilderSdk) -> Result<()> {
    let config_path = &sdk.config_path();

    let status = Command::new("cmake")
        .arg("-B")
        .arg(build_folder(config_path))
        .arg("-S")
        .arg(source_folder(config_path))
        .arg(&format!(
            "-DCMAKE_TOOLCHAIN_FILE={}",
            toolchain_file(config_path).display()
        ))
        .spawn()?
        .wait()
        .await?;

    assert!(status.success(), "CMake execution failed");

    Command::new("cmake")
        .arg("--build")
        .arg(build_folder(config_path))
        .arg("-j")
        .arg(num_cpus::get().to_string())
        .spawn()?
        .wait()
        .await?;

    assert!(status.success(), "Build failed");
    Ok(())
}

#[tokio::main]
async fn main() -> Result<()> {
    let sdk = BuilderSdk::init(|sdk, event| async move {
        match event {
            BuilderEvent::Exit => kill_application_in_rpi(&sdk).await,
        }
    })
    .await?;

    match sdk.action() {
        Action::Build => build_application(&sdk).await,
        Action::Run => run_application(&sdk).await,
    }
}

Now, whenever a job is cancelled by either EJB or EJD (Guide 03) the script will receive the Exit event and will clean the necessary resources.

Step 6: Build your application

cd ~/ej-workspace/ejkmer-builder
cargo build --release

The application is now available inside the ~/ej-workspace/ejkmer-builder/target/release folder.

Step 7: Update your EJB config

We can use this new application to handle every build and run configuration so now we need to tell EJB, through its config, to use it.

We can use some sed magic to avoid having to change every line manually:

sed -i 's/\/[b|r].*.sh/ejkmer-builder\/target\/release\/ejkmer-builder/g' ~/ej-workspace/config.toml

Here's the final result:

NOTE: Replace <user> with your username

[global]
version = "1.0.0"

[[boards]]
name = "Raspberry Pi"
description = "Raspberry Pi with Raspberry OS 64 bits"

[[boards.configs]]
name = "k-mer-original"
tags = ["arm64", "kmer unoptimized"]
build_script = "/home/<user>/ej-workspace/ejkmer-builder/target/release/ejkmer-builder"
run_script = "/home/<user>/ej-workspace/ejkmer-builder/target/release/ejkmer-builder"
results_path = "/home/<user>/ej-workspace/results_k-mer-original.txt"
library_path = "/home/<user>/ej-workspace/kmer"

[[boards.configs]]
name = "k-mer"
tags = ["arm64", "kmer optimized"]
build_script = "/home/<user>/ej-workspace/ejkmer-builder/target/release/ejkmer-builder"
run_script = "/home/<user>/ej-workspace/ejkmer-builder/target/release/ejkmer-builder"
results_path = "/home/<user>/ej-workspace/results_k-mer.txt"
library_path = "/home/<user>/ej-workspace/kmer"

[[boards.configs]]
name = "k-mer-omp"
tags = ["arm64", "kmer multi-threaded optimized"]
build_script = "/home/<user>/ej-workspace/ejkmer-builder/target/release/ejkmer-builder"
run_script = "/home/<user>/ej-workspace/ejkmer-builder/target/release/ejkmer-builder"
results_path = "/home/<user>/ej-workspace/results_k-mer-omp.txt"
library_path = "/home/<user>/ej-workspace/kmer"

[[boards.configs]]
name = "infinite-loop"
tags = ["arm64", "infinite-loop"]
build_script = "/home/<user>/ej-workspace/ejkmer-builder/target/release/ejkmer-builder"
run_script = "/home/<user>/ej-workspace/ejkmer-builder/target/release/ejkmer-builder"
results_path = "/home/<user>/ej-workspace/results_infinite-loop.txt"
library_path = "/home/<user>/ej-workspace/kmer"

TIP

Putting the application in our $PATH will make it easier to invoke it, for this we recommend installing it in our PC directly:

cargo install --path ~/ej-workspace/ejkmer-builder

With the application installed you can set every build and run scripts in your config file like this:

build_script = "ejkmer-builder"
run_script = "ejkmer-builder"

And of course a sed command to avoid having to do it manually:

sed -i 's/script = .*/script = "ejkmer-builder"/g' ~/ej-workspace/config.toml

This makes our config.toml easier to read and allows us to freely move our source code if we wish so.

Step 8: Test the new script

Make sure you've cleaned up the running process in your raspberry pi:

ssh ${PI_USERNAME}@${PI_ADDRESS} "killall infinite-loop"

cd ~/ej-workspace
ejb --config config.toml validate

We can again quit EJB with CTRL+C and we'll be able to see that the infinite-loop is not running on our Raspberry Pi even after abruptly quitting the whole process.

ssh ${PI_USERNAME}@${PI_ADDRESS} "killall infinite-loop"
infinite-loop: no process found

Advantages of using the EJ Builder SDK

  • Proper cancellation handling. When EJB sends an exit signal, your script can clean up running processes on target devices instead of leaving them orphaned
  • Single binary approach. One application handles both building and running (though you could do this with shell scripts too, it's just arguably harder)
  • Custom result formats. Our example just saves program output to a file, but we can collect and format results however makes sense for our use case
  • Easy integration testing. Write tests that spawn TCP listeners, launch our program on the target device, and verify the results in real-time
  • Unlimited possibilities. Once we're using a real programming language, we can do things like:
    • Monitor system resources (CPU, memory, network) during test execution
    • Send notifications to Slack when tests complete
    • Generate detailed HTML reports with charts and graphs
  • Job cancellation support with EJD (Guide 03)
  • You get to write rust code

Disadvantages of using the EJ Builder SDK

  • Setup overhead. It takes longer to get started compared to throwing together a quick shell script
  • Compile-test cycle: Every change requires a cargo build before you can test it, which slows down rapid iteration
    • Can be minimized by tools like cargo-watch
  • Rust knowledge required: You need to be comfortable with Rust syntax, and async programming
    • Though the SDK could be ported to other languages very easily. Contributions are welcome
  • Binary management: Need to keep track of compiled binaries and make sure they're available where EJB expects them.
    • Installing the application with cargo install solves this
  • Overkill for simple tasks: If we're just running basic commands and don't need to clean up any resources when a job fails, a shell script might be simpler
    • E.g., when running tests in an MCU where every deployment overwrites the board's flash memory.
  • You get to write rust code

Next Steps

At this point, we have a fully functional EJ Builder setup that can handle complex deployments with proper cancellation handling. EJB works perfectly fine as a standalone tool - we can integrate it into CI/CD pipelines or use it on our development machine to spin up integration tests in the background while we work on other tasks.

You may have noticed that throughout this guide, we haven't stored results anywhere and we've only worked with a single builder. This is completely fine for many use cases, but if you're looking at larger-scale deployments with multiple builders, you might want something more robust.

In Guide 03 - Dispatcher, we'll explore the EJ Dispatcher (EJD) a tool that can:

  • Manage multiple builders simultaneously
  • Queue and distribute jobs across your hardware fleet
  • Store and organize results from multiple test runs
  • Provide authentication and access control
  • Enable remote job submission and monitoring

The dispatcher transforms EJ from a single-builder tool into a powerful distributed testing platform, but it's entirely optional depending on our needs.

Best Practice: Always use the Builder SDK for production deployments, especially for long-running applications.

Guide 03: Setting Up an EJ Dispatcher

Overview

In the previous guides, we learned how to set up a single EJ Builder and run embedded tests on target hardware. While this works perfectly for individual development and experimentation, real-world embedded development often involves teams, multiple hardware configurations, and integration with CI/CD pipelines.

This guide introduces the EJ Dispatcher (EJD) - a centralized service that transforms our individual EJ Builder into a scalable, multi-user testing infrastructure. We'll learn how to set up a dispatcher, connect multiple builders, and coordinate testing across our development team.

Prerequisites

Before starting this guide, ensure you have:

  • Completed Guide 02: This guide builds directly on the EJ Builder setup from the previous guide
  • Docker and Docker Compose: Required for running the EJD service
  • Basic Linux permissions knowledge: We'll be working with Unix sockets and user groups
  • Network access: If deploying across multiple machines
  • Your working EJ Builder setup: From the previous guide, including the kmer project configuration and ej-workspace

What is the EJ Dispatcher?

The EJ Dispatcher acts as the central coordinator for your embedded testing infrastructure. Think of it as a "job scheduler" that sits between your development workflow and your physical hardware.

Without EJD: Each developer runs their own EJ Builder instance, manually coordinating access to hardware and managing their own test results.

With EJD: A single dispatcher manages a pool of builders and hardware, automatically queuing jobs, distributing work, and storing results in a centralized location.

Key capabilities include:

  • Centralized Job Management: Queue and automatically distribute test jobs across available builders
  • Multi-Builder Coordination: Manage fleets of builders with different hardware configurations
  • Team Collaboration: Multiple developers can submit jobs without hardware conflicts
  • Result Storage & History: Centralized database stores all test results with full historical tracking
  • Secure Access: Authentication system controls who can submit jobs and access results
  • CI/CD Integration: REST API enables integration with automated build pipelines

When You Need a Dispatcher

The EJ Dispatcher becomes essential when you encounter these scenarios:

Team Development: Multiple developers need to share the same embedded hardware without conflicts. The dispatcher automatically queues jobs and ensures fair access to resources.

CI/CD Integration: Your build pipeline needs to automatically run embedded tests on every commit. The dispatcher provides the API endpoints and job management needed for automation.

Hardware Scaling: You have multiple development boards, different hardware configurations, or geographically distributed test setups. The dispatcher can coordinate across all of them.

Result Management: You need to track performance trends, compare results across builds, or maintain historical test data. The dispatcher provides centralized storage and querying capabilities.

Production Testing: You're running embedded tests as part of your release process and need reliability, monitoring, and audit trails that a centralized system provides.

Architecture Overview

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Git Repo      │    │   CI/CD         │    │   Developer     │
│                 │    │                 │    │                 │
└─────────┬───────┘    └─────────┬───────┘    └─────────┬───────┘
          │                      │                      │
          │          ┌───────────┼───────────┐          │
          │          │           │           │          │
          └──────────┼───────────┼───────────┼──────────┘
                     │           │           │
                ┌────▼───────────▼───────────▼────┐
                │      EJD (Dispatcher)           │
                │                                 │
                │  ┌─────────────────────────────┐│
                │  │     Job Queue Manager       ││
                │  │  - Queue and dispatch jobs  ││
                │  └─────────────────────────────┘│
                │  ┌─────────────────────────────┐│
                │  │     Result Storage          ││
                │  │  - Database management      ││
                │  │  - Historical data          ││
                │  └─────────────────────────────┘│
                │  ┌─────────────────────────────┐│
                │  │     Authentication          ││
                │  │  - Builder Registration     ││
                │  │  - Client Access Control    ││
                │  └─────────────────────────────┘│
                └────┬───────────┬───────────┬────┘
                     │           │           │
         ┌───────────┼───────────┼───────────┼───────────┐
         │           │           │           │           │
    ┌────▼────┐ ┌────▼────┐ ┌────▼────┐ ┌────▼────┐ ┌────▼────┐
    │ EJB #1  │ │ EJB #2  │ │ EJB #3  │ │ EJB #4  │ │ EJB #N  │
    │ (Local) │ │ (RPi)   │ │ (Cloud) │ │ (Lab)   │ │ (...)   │
    └─────────┘ └─────────┘ └─────────┘ └─────────┘ └─────────┘

Step 1: Installing EJD

The easiest way to deploy EJD is using the official Docker setup. EJ provides a ready-to-use git repository that uses Docker and Docker Compose to setup EJD.

git clone https://github.com/embj-org/ejd-deployment ~/ejd-deployment
cd ~/ejd-deployment
# Follow the prompts to setup your env variables
./setup.sh
# Download and launch EJD along with a postgresql database
docker compose up -d

TLS Support

If you're looking into deploying EJD into a public network, we very highly recommend setting it up with TLS support. A skeleton example is available in the same git repository inside the tls-example folder. It uses Traefik to set up a reverse proxy which will provide TLS support for you.

NOTE: It's not a ready-to-use example. Many things depend on your specific setup like DNS provider and URL but it should help you getting started.

Step 2: Set up permissions to access the EJD socket

During setup, EJD create an Unix Socket that can be used to communicate with the tool. By default, we need root permissions to access this socket. This is a security measure as EJD doesn't perform any sort of authentication when communicating through the socket. Instead the authentication is managed by Linux. In most setups it'll be used to create the first application user and test our setup and for this we must be explicit about giving ourselves permissions.

Here we will create a new group called ejd and change the group ownership of the socket file to this new group. Once we have done that, we'll add the current user to the ejd group which will allow us to use the socket provided by EJD without having to prefix every command with sudo.

# Create the `ejd` group
sudo groupadd ejd

# Add yourself to the `ejd` group
sudo usermod -a -G ejd $USER

# Change the group ownership of the socket file to 'ejd'
# This allows members of the 'ejd' group to access the file if permissions allow
sudo chown :ejd ~/ejd-deployment/ejd/tmp/ejd.sock

# Grant write permissions to the group for the socket file
# This enables 'ejd' group members to write to the socket
sudo chmod g+w ~/ejd-deployment/ejd/tmp/ejd.sock

# Activate the group changes in the current shell session
# You may need to log out and login again
newgrp ejd

Step 3: Create your first user

EJ provides ejcli, a cli tool that interfaces with EJD. You can install it the same way we installed EJB in the first guide

cargo install ejcli

Now create your first user.

NOTE: Replace <username> in the command and enter your password when prompted.

ejcli create-root-user --socket ~/ejd-deployment/ejd/tmp/ejd.sock --username <username>
Creating user
Password >
CreateRootUserOk(EjClientApi { id: 63c16857-0372-4add-a5bf-c0bd266fe650, name: "<username>" })

Step 4: Register your builder

To create the builder, we'll use the rest API interface from EJD. This can be created from a different PC as long as we have the correct permissions and access to the port EJD is exposed to.

NOTE: Replace <username> in the command and enter your password when prompted.

ejcli create-builder --server http://localhost:3000 --username <username>
...
export EJB_ID=<builder_id>
export EJB_TOKEN=<builder_token>

The EJB_ID and EJB_TOKEN provided allow us to connect our EJB instance to EJD. Again, this connection will use the HTTP(s) interface.

Step 5: Connecting EJB to EJD

Export the two environment variables that we got from the last command and launch EJB:

export EJB_ID=<builder_id>
export EJB_TOKEN=<builder_token>
ejb --config ~/ej-workspace/config.toml connect --server http://localhost:3000

You should see that the websocket connection was established successfully.

2025-07-11T11:57:30.191943Z  INFO ejb::connection: WebSocket connection established

Once we start a connection, EJB will wait until a new job request comes from EJD.

Step 6: Dispatch your first build job

Every job that can be dispatched through EJD is associated with a specific git commit hash. This allows you to later check every job associated with a specific commit.

EJB will automatically check out the new version for you before building and running your application.

When dispatching a job we need to provide:

  • The commit hash associated with the current job.
  • The remote URL associated with this commit.
    • This allows us to run jobs from forks of our repository for instance.
  • The timeout time in seconds after which the job will be cancelled.

Optionally, if you have private repositories and don't want to set up an ssh key in the machine hosting our builder, you may also provide a token that would allow git to fetch our private repository using https.

Once again, ej-cli can be used to test your setup and making sure everything is working correctly.

Since our test application kmer is publicly available we don't need to provide a token:

ejcli dispatch-build \
 --socket ~/ejd-deployment/ejd/tmp/ejd.sock \
 --seconds 20 \
 --commit-hash eb7c6cbe6249aff4df82455bbadf4898b0167d09 \
 --remote-url https://github.com/embj-org/kmer

We should see something like this as a result:

=======================================
Build finished successfully with 4 log entries:
=======================================
d2a5ae66-5eab-493e-847c-af21a52455d6 - k-mer-original [arm64,kmer unoptimized]
=======================================
From https://github.com/embj-org/kmer
 * [new branch]      main       -> ejupstream/main
HEAD is now at eb7c6cb feat: add infinite loop example
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /home/andre/ej-workspace/kmer/build-pi
[ 25%] Built target infinite-loop
[ 50%] Built target k-mer
[100%] Built target k-mer-original
[100%] Built target k-mer-omp

=======================================
5a5df2de-9854-45b6-a4a5-0af58959a04b - k-mer-omp [arm64,kmer multi-threaded optimized]
=======================================
From https://github.com/embj-org/kmer
 * [new branch]      main       -> ejupstream/main
HEAD is now at eb7c6cb feat: add infinite loop example
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /home/andre/ej-workspace/kmer/build-pi
[ 75%] Built target k-mer
[ 75%] Built target k-mer-original
[ 75%] Built target k-mer-omp
[100%] Built target infinite-loop

=======================================
ffdc80b8-6e59-42e7-af82-3eef5d5ba6bd - k-mer [arm64,kmer optimized]
=======================================
From https://github.com/embj-org/kmer
 * [new branch]      main       -> ejupstream/main
HEAD is now at eb7c6cb feat: add infinite loop example
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /home/andre/ej-workspace/kmer/build-pi
[100%] Built target k-mer-omp
[100%] Built target k-mer-original
[100%] Built target k-mer
[ 62%] Built target infinite-loop

=======================================
aff0ed67-92aa-4011-876b-bf79c51a7210 - infinite-loop [arm64,infinite-loop]
=======================================
From https://github.com/embj-org/kmer
 * [new branch]      main       -> ejupstream/main
HEAD is now at eb7c6cb feat: add infinite loop example
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /home/andre/ej-workspace/kmer/build-pi
[ 50%] Built target k-mer-omp
[ 50%] Built target k-mer-original
[ 75%] Built target infinite-loop
[100%] Built target k-mer

=======================================

Understanding the internals

Congratulations, we have now seen how we can dispatch a new job to EJD that gets picked up by EJB and is run on our target board.

In our specific use case, we have one builder instance with one board connected to it but by now you should have a good understanding of how this whole setup expands to multiple builders and boards.

Job Cancellation and Timeouts

We can also use the dispatch-run command to build and run our application like we've done before.

The process is very similar to the dispatch-build command but it will also run the application on the board and wait for it to finish.

ejcli dispatch-run \
 --socket ~/ejd-deployment/ejd/tmp/ejd.sock \
 --seconds 20 \
 --commit-hash eb7c6cbe6249aff4df82455bbadf4898b0167d09 \
 --remote-url https://github.com/embj-org/kmer

In this case, since we still have our infinite-loop application being deployed, the job will eventually time out after 20 seconds.

Analyzing the results we'll be able to see 4 log entries for the 4 configs but only 3 result entries as the last config never actually produced any results before the job got cancelled.

This timeout feature relies on the EJ Builder SDK presented in the last guide. The same way EJD sent the Build command in the above example, after the timeout is reached, EJD will send a Cancel command to the builder. The builder will then notify the script performing the build that the job was cancelled. The Builder SDK will receive this cancellation command and run the cleanup code that provided in our ejkmer-builder application.

In cases we don't use the Builder SDK, the builder will eventually kill the process without giving it a chance to clean up, this is why we highly recommend using the Builder SDK for all our builds.

Next Steps

Congratulations! You have successfully set up an EJ Dispatcher and connected your first builder. This is a significant step towards building a scalable and manageable testing infrastructure.

You may have noticed that we haven't yet covered how to actually parse the results of the job and how to fetch results from previous jobs. EJ provides two solutions for this:

  • The first solution is to directly do it inside our ejkmer-builder application. This is a good approach if we don't need any history of the previous results and only care about what the current job produced.
  • The second solution is to use the EJ Dispatcher SDK to create a custom CLI tool that can interact with EJD programmatically. The same way the ejcli tool allows us to interact with EJD, we can create our own custom CLI tool that can submit jobs, query their status, and retrieve results. Additionally, we're free to implement any custom logic we need to parse and analyze the results.

In Guide 04 - Dispatcher SDK, we'll create a custom CLI tool that can:

  • Submit jobs programmatically
  • Query job status and results
  • Parse and analyze test data

This SDK-based approach enables powerful automation and integration possibilities.

Production Tip: When deploying EJD to an open network, always use HTTPS with a reverse proxy like Traefik or Nginx to secure our dispatcher. This prevents unauthorized access and ensures encrypted communication between builders and the dispatcher. A skeleton example using Traefik is available in the ejd-deployment repository

Guide 04: Building custom tools with the Dispatcher SDK

You can find the SDK's documentation in crates.io

Overview

In the previous guide, we successfully set up an EJ Dispatcher and connected builders to create a centralized testing infrastructure. While we can now submit jobs and see results through the ejcli tool, we haven't yet explored how to programmatically interact with the dispatcher or analyze the results it produces.

This guide demonstrates how to use the EJ Dispatcher SDK to build custom applications that can submit jobs, retrieve results, and perform automated analysis. We'll create a Rust application that connects to our dispatcher, fetches test results, and validates that different configurations of our embedded application produce consistent outputs.

By the end of this guide, we'll understand how to build custom tooling around EJ's dispatcher infrastructure, enabling powerful automation and analysis workflows for our embedded testing pipeline.

Prerequisites

Before starting this guide, ensure you have:

  • Completed Guide 03: This guide builds directly on the EJ Dispatcher setup from the previous guide
  • Working EJ Dispatcher setup: Including:
    • EJD running and accessible
    • At least one builder connected and working
    • Successful job submissions using ejcli
    • The kmer project results from previous guides
  • Rust toolchain installed: We'll need cargo and the Rust compiler for building the SDK application
    • Install via rustup.rs if you haven't already
  • Basic Rust knowledge

Step 1: Setting up your rust project

Let's create a custom CLI tool called ejkmer-dispatcher that will interact with EJD to submit jobs, retrieve results, and perform analysis on the results produced by the kmer project.

cd ~/ej-workspace
cargo init --bin ejkmer-dispatcher
cd ejkmer-dispatcher
cargo add ej-dispatcher-sdk
cargo add ej-config
cargo add clap -F derive
cargo add tokio -F macros -F rt-multi-thread

Step 2: Using the dispatcher sdk to start a new job

The dispatcher_sdk offers you a function to dispatch new jobs easily:

#![allow(unused)]
fn main() {
use ej_dispatcher_sdk::prelude::*;

async fn do_run(
    socket: PathBuf,
    seconds: u64,
    commit_hash: String,
    remote_url: String,
) -> Result<()> {

    let job_result = ej_dispatcher_sdk::dispatch_run(
        &socket,
        commit_hash,
        remote_url,
        None,
        Duration::from_secs(seconds),
    )
    .await?;

    println!("{}", job_result);
}
}

The dispatch_run function will connect to EJD using the Unix Socket and maintain a connection until the job either finishes or is cancelled.

The job can either be immediately dispatched or put into a queue if there are already running jobs. Additionally, the jobs can be cancelled if, by the time the job leaves the queue there are no builders available or if the job times out.

Once we get to this line :

#![allow(unused)]
fn main() {
    println!("{}", job_result);
}

We know the job has finished successfully and we're ready to start parsing the results.

Step 3: Parse the results

You may be wondering what the type of job_result is. If so, here it goes:

#![allow(unused)]
fn main() {
/// Run operation result.
#[derive(Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct EjRunResult {
    /// Run logs per board configuration.
    pub logs: Vec<(EjBoardConfigApi, String)>,
    /// Run results per board configuration.
    pub results: Vec<(EjBoardConfigApi, String)>,
    /// Whether the run was successful.
    pub success: bool,
}
}

We're mostly interested in the results: Vec<(EjBoardConfigApi, String)> which is a dynamic array of Board Config and String pairs.

  • The EjBoardConfigApi holds the config ID, name and tags
  • The String holds the job results. For our specific use case each String will have the following format. This is the content of the results_path when the run_script finishes.

The logs are useful to see what happened between every job phase, checkout, build and run and they're are printed automatically when doing this:

#![allow(unused)]
fn main() {
    println!("{}", job_result);
}

For our example, the results follow this format:

Results:
ABC: 2
BCD: 1
CDA: 1
DAB: 1

real	0m0.005s
user	0m0.000s
sys	0m0.005s

We'll focus on parsing the sequences found and their occurrences, we'll then check to make sure that every config finds the same sequences and the same number of occurrences per sequence.

Here's an example of a function that does just that:

#![allow(unused)]
fn main() {
struct ConfigResult {
    config: EjBoardConfigApi,
    data: HashMap<String, usize>,
}

fn parse_results(job_result: &EjRunResult) -> Vec<ConfigResult> {
    let mut parsed_results: Vec<ConfigResult> = Vec::new();
    for (board_config, result) in job_result.results.iter() {
        let mut occurrences_map: HashMap<String, usize> = HashMap::new();
        let mut found_start_of_results = false;
        for line in result.lines() {
            if line.contains("Results:") {
                found_start_of_results = true;
                continue;
            }
            if !found_start_of_results {
                continue;
            }
            if line.contains(':') {
                let splitted: Vec<&str> = line.split(": ").collect();
                assert_eq!(splitted.len(), 2);
                let sequence = splitted[0];
                let n_occurences = splitted[1]
                    .parse()
                    .expect("Expected number on right side of ':'");
                occurrences_map.insert(sequence.to_string(), n_occurences);
            }
        }
        parsed_results.push(ConfigResult {
            config: board_config.clone(),
            data: occurrences_map,
        });
    }
    parsed_results
}
}

Step 4: Check the results

Once we have the results parsed, it makes it easier to reason with the code that actually checks that the results are valid:

#![allow(unused)]
fn main() {
fn check_results(parsed_results: &Vec<ConfigResult>) {
    for i in 0..parsed_results.len() {
        for j in (i + 1)..parsed_results.len() {
            let config_i = &parsed_results[i].config;
            let config_j = &parsed_results[j].config;
            let data_i = &parsed_results[i].data;
            let data_j = &parsed_results[j].data;

            assert_eq!(
                data_i.len(),
                data_j.len(),
                "Different number of sequences for {} and {} {} vs {}",
                config_i.name,
                config_j.name,
                data_i.len(),
                data_j.len(),
            );

            for (sequence, expected) in parsed_results[i].data.iter() {
                let actual = data_j.get(sequence);
                assert!(
                    actual.is_some(),
                    "Couldn't find {} in {}",
                    sequence,
                    config_j.name
                );

                let actual = actual.unwrap();

                assert_eq!(
                    expected, actual,
                    "Expected {} occurrences for {}. Got {} ",
                    expected, sequence, actual
                );
            }
        }
    }
}
}

Step 5: Completing our do_run function

Once we can parse and check the results, the only thing left to do is to use these functions to make sure our run was successful:

#![allow(unused)]
fn main() {
async fn do_run(
    socket: PathBuf,
    seconds: u64,
    commit_hash: String,
    remote_url: String,
) -> Result<()> {
    let job_result = ej_dispatcher_sdk::dispatch_run(
        &socket,
        commit_hash,
        remote_url,
        None,
        Duration::from_secs(seconds),
    )
    .await?;
    println!("{}", job_result);

    if !job_result.success {
        return Err(Error::RunError);
    }
    let parsed_results = parse_results(&job_result);
    check_results(&parsed_results);

    println!("Results OK!");

    Ok(())
}
}

Step 6: Add a CLI interface

We'll also add a basic CLI interface that will make it easier to add new commands down the line.

For this we'll use clap that we've already added to our project.

The following code should be pretty straight forward to reason with:

#![allow(unused)]
fn main() {
use clap::{Parser, Subcommand};
#[derive(Parser)]
#[command(name = "ejkmer-dispatcher")]
#[command(about = "EJ Kmer Dispatcher - Job dispatcher and result handler for the Kmer project")]
struct Cli {
    #[command(subcommand)]
    pub command: Commands,
}

#[derive(Subcommand)]
enum Commands {
    DispatchRun {
        /// Path to the EJD's unix socket
        #[arg(short, long)]
        socket: PathBuf,
        /// The maximum job duration in seconds
        #[arg(long)]
        seconds: u64,
        /// Git commit hash
        #[arg(long)]
        commit_hash: String,
        /// Git remote url
        #[arg(long)]
        remote_url: String,
    },
}
}

Step 7: Putting everything together

#[tokio::main]
async fn main() -> Result<()> {
    let cli = Cli::parse();
    match cli.command {
        Commands::DispatchRun {
            socket,
            seconds,
            commit_hash,
            remote_url,
        } => do_run(socket, seconds, commit_hash, remote_url).await,
    }
}

clap will automatically parse the arguments for you to make sure that everything works correctly. We can now try our new application with the same arguments as the ej-cli

Step 8: Try it out

First, remove the infinite-loop config as it doesn't do anything useful.

Remove this entry from ~/ej-workspace/config.toml:

[[boards.configs]]
name = "infinite-loop"
tags = ["arm64", "infinite-loop"]
build_script = "ejkmer-builder"
run_script = "ejkmer-builder"
results_path = "/home/<user>/ej-workspace/results_infinite-loop.txt"
library_path = "/home/<user>/ej-workspace/kmer"

And now run our new program:

cd ~/ej-workspace/ejkmer-dispatcher/
cargo run -- dispatch-run \
    --socket ~/ejd-deployment/ejd/tmp/ejd.sock \
    --seconds 60 \
    --commit-hash eb7c6cbe6249aff4df82455bbadf4898b0167d09 \
    --remote-url https://github.com/embj-org/kmer

=======================================
Run finished successfully with 3 log entries:
=======================================
84b19f1e-66c1-4182-a428-c4357be4d9a4 - k-mer [arm64,kmer optimized]
=======================================
From https://github.com/embj-org/kmer
 * [new branch]      main       -> ejupstream/main
HEAD is now at eb7c6cb feat: add infinite loop example
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /home/andre/ej-workspace/kmer/build-pi
[ 50%] Built target infinite-loop
[ 50%] Built target k-mer-original
[ 75%] Built target k-mer-omp
[100%] Built target k-mer
Results:
ABC: 2
BCD: 1
CDA: 1
DAB: 1

real	0m0.003s
user	0m0.003s
sys	0m0.000s

=======================================
51ba26d0-b71e-444a-9732-9a33e51dd4dd - k-mer-omp [arm64,kmer multi-threaded optimized]
=======================================
From https://github.com/embj-org/kmer
 * [new branch]      main       -> ejupstream/main
HEAD is now at eb7c6cb feat: add infinite loop example
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /home/andre/ej-workspace/kmer/build-pi
[ 50%] Built target k-mer-original
[ 50%] Built target infinite-loop
[100%] Built target k-mer-omp
[100%] Built target k-mer
Results:
ABC: 2
BCD: 1
CDA: 1
DAB: 1

real	0m0.004s
user	0m0.004s
sys	0m0.003s

=======================================
734cdfb0-85aa-4405-af5a-752d68f5c003 - k-mer-original [arm64,kmer unoptimized]
=======================================
From https://github.com/embj-org/kmer
 * [new branch]      main       -> ejupstream/main
HEAD is now at eb7c6cb feat: add infinite loop example
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /home/andre/ej-workspace/kmer/build-pi
[ 75%] Built target k-mer-original
[ 75%] Built target k-mer
[ 75%] Built target infinite-loop
[100%] Built target k-mer-omp
Results:
ABC: 2
BCD: 1
CDA: 1
DAB: 1

real	0m0.003s
user	0m0.003s
sys	0m0.000s

=======================================

=======================================
Run finished successfully with 3 result entries:
=======================================
734cdfb0-85aa-4405-af5a-752d68f5c003 - k-mer-original [arm64,kmer unoptimized]
=======================================
Results:
ABC: 2
BCD: 1
CDA: 1
DAB: 1

real	0m0.005s
user	0m0.005s
sys	0m0.001s

=======================================
84b19f1e-66c1-4182-a428-c4357be4d9a4 - k-mer [arm64,kmer optimized]
=======================================
Results:
ABC: 2
BCD: 1
CDA: 1
DAB: 1

real	0m0.005s
user	0m0.000s
sys	0m0.005s

=======================================
51ba26d0-b71e-444a-9732-9a33e51dd4dd - k-mer-omp [arm64,kmer multi-threaded optimized]
=======================================
Results:
ABC: 2
BCD: 1
CDA: 1
DAB: 1

real	0m0.007s
user	0m0.000s
sys	0m0.010s

=======================================

Results OK!

Seeing Results OK! means that the job ran succesfully and the results were as expected !

What's Next

Congratulations! You've now built a complete embedded testing infrastructure with EJ, from basic builder setup to advanced dispatcher integration with custom analysis tools. However, this is just the beginning of what's possible with EJ.

Beyond Simple Applications

Throughout these guides, we've used the kmer application as our example - a relatively simple C program that processes text files. The real power of EJ becomes apparent when we apply it to more complex embedded applications:

  • Multi-component systems: Applications with multiple executables, libraries, and configuration files
  • Real-time systems: Programs that interact with hardware peripherals, sensors, or communication protocols
  • Performance-critical applications: Code that needs to be tested across different optimization levels and compiler flags
  • Cross-platform applications: Software that must run on multiple embedded architectures (ARM, RISC-V, x86 embedded, etc.)

Since EJ works with any application that can be built and deployed without manual intervention, we can integrate virtually any embedded project. The key is writing appropriate build and deployment scripts (whether shell scripts or using the Builder SDK) that handle our specific application's requirements.

Advanced Result Analysis and Integration

Our result validation example simply checked that different configurations produced identical outputs. In real-world scenarios, we can implement much more sophisticated analysis and integration workflows:

Communication and Notifications

  • Slack integration: Send notifications when tests complete, fail, or show performance regressions
  • Email reports: Generate detailed test summaries and email them to your team
  • GitHub/GitLab PR comments: Automatically comment on pull requests with test results and performance metrics

Continuous Integration Workflows

  • Performance trend analysis: Compare current results with historical data to detect regressions
  • Automated benchmarking: Track performance metrics over time and alert on significant changes
  • Cross-platform validation: Ensure our application behaves consistently across different hardware platforms

Result Presentation and Documentation

  • HTML report generation: Create rich, interactive web pages showing test results with charts and graphs
  • Dashboard creation: Build real-time dashboards showing the health of your embedded systems
  • Automated documentation: Generate performance reports and system specifications based on test results

Final Thoughts

EJ transforms embedded testing from a manual, error-prone process into a reliable, automated pipeline. By centralizing job management, providing proper cancellation handling, and offering programmatic access to results, EJ enables the same level of testing sophistication in embedded development that web developers take for granted.

The journey from a simple shell script to a full distributed testing infrastructure demonstrates how EJ scales with our needs - start simple, then add complexity as your requirements grow. From a solo developer with a single Raspberry Pi to a company managing hundreds of embedded devices, EJ provides the tools to build a testing infrastructure that grows with your project.

Happy testing!

Community: Share your custom tools and integrations with the EJ community to help others solve similar challenges.