D2.3 – MVP Alpha (SHRUTHI)

[dvn]

D2.3 – MVP Alpha (SHRUTHI)

tl;dr: 2021-12-09 – We are delighted to announce D2.3, the alpha release of Rhyzome, which marks the end of our third Milestone.

Introduction

Services hosting and system administration are integral elements in enabling community-run online group collaboration. The Self-Hosted, Robust Unikernel Testing & Hosting Infrastructure (SHRUTHI) effort encapsulates our exploration of this topic.

In D1.3 – SHRUTHI Service Prototype Release we were announcing

a practical and deterministic way for system administrators and collectives to self-host [DREAM] components.

Implementation

Based on our previous work, and incorporating subsequent refactoring to the Entanglement Garden self-hosting project, we have added private network provisioning for instances (VMs or Unikernels) with WireGuard endpoints for remote authentication and access.

This is implemented as the provisioning and resource management software Rhyzome.

Since D1.3, Rhyzome has been refactored into a modular system which consists of a core API that distributes jobs to purpose-specific components. We use the Rhyzome Libvirt component for provisioning instances (unikernels for VMs) paired with the Rhyzome Openwrt component to create and manage networks.

^ Basic diagram of SHRUTHI.

Instances (virtual machines or unikernels) may be clustered together into the same virtual network, allowing them to connect together directly in an isolated subnetwork.

Rhyzome API

This is the core of the system. It handles resource lifecycle management ans scheduling, and can be defined in two parts:

1. GRPC Server

All Rhyzome components connect to the core with gRPC, and use mutual TLS (mTLS) for authentication. Once components are connected, they wait for the server to send them jobs like creating new networks or VM instances.

2. HTTP API

This is how clients interact with Rhyzome. Allowing for a unified interface for CLI tools, web applications, or any other type of clients through a familiar REST API defined using OpenAPI specification.

Rhyzome Libvirt

This component is for provisioning and maintaining computing instances using libvirt . Instances may either be general purpose VMs, or highly specialized Unikernels.

The component expected to be used on a Linux Hypervisor with KVM support, but may support other OSes in the future due to its relatively portable nature; Being written in Go, and based on Libvirt.

Rhyzome OpenWrt

A Rhyzome component which runs in an OpenWrt VM. It handles lifecycle management operations of networks and WireGuard servers.

Try it

You can try the implementation on any Linux system which can use KVM.

Please see the Getting Started guide for further details on installing and configuring Rhyzome.

Click here for full-size

Highlights

OpenWrt + WireGuard®

SHRUTHI uses OpenWrt Linux for its networking component. This provides a stable and well documented interface for managing subnets, firewalls, VPNs (WireGuard®), and VLANs due to OpenWrt’s UCI system.^[1]

New networks have a WireGuard^[2] interface created and bridged with them. WireGuard is a light-weight VPN implementation using modern cryptography. It was incorporated into the Linux 5.6 Kernel in 2020.

When you provision a network with Rhyzome, the OpenWrt bridge and WireGuard tunnel bundle enables system administrators to connect remotely from their office into the deployed VLANs like they would in their own datacenter. This allows for a proper separation in the case of professional hosting, where customers should only be allowed to see and connect to their own VLANs and VMs.

OpenAPI

Rhyzome’s API is defined using the OpenAPI specification. This allows for designing a maintainable API that can be easily consumed by tools and developers alike. You can find it in its own rhyzome-openapi repository.

Future

This alpha release of SHRUTHI is notable for the addition of several useful features, however there is still work to be done to make the provisioning software more generalized, and feature-complete. Work focused on UI & UX is needed to make the software more approachable and easier to integrate with 3rd party software.

Thank you!

We welcome feedback and criticism! Our forum is open for friendly cooperation among DREAM Catchers. Do not hesitate to contact us and contribute to the code: our research is made to improve the digital commons.

This is experimental software made within the DREAM project.

This project has received funding from the European Union’s Horizon 2020 research and innovation programme within the framework of the NGI-POINTER Project funded under grant agreement No 871528.

---

1. “Unified Configuration Interface” ↩︎

2. “WireGuard” is a registered trademark of Jason A. Donenfeld. ↩︎

[dvn]

@how please take a look at the edits i’ve made to the top post.

I’m also editing in a pad, if you’d like to work there with me:

D2.3 – MVP Alpha (SHRUTHI) - HedgeDoc

[how]

The changes look good. I’ll try and figure out a way to flesh it out with real examples. Maybe running our own website as previously suggested could be useful. In the meantime I think the plan is great.

[how]

@dvn

message router

Maybe this needs to be named otherwise since it may become confusing with the UPSYCLE (message) router.

@dvn

route messages based on public keys between tunnels and unikernels.

What would be the purpose of such messages? And are we still talking about unikernels?

[dvn]

@how

Maybe this needs to be named otherwise since it may become confusing with the UPSYCLE (message) router.

I concur. +1

@how

What would be the purpose of such messages? And are we still talking about unikernels?

Legacy hangover.

[how]

I added a few words that hopefully disambiguate ‘message routing’.

[how]

The nanos book has a nice picture, can we make a similar one for SHRUTHI / Rhyzome – mostly adding a column to this one?

I mean, it makes sense to create a new image on this model, to show where Rhyzome plays, and that it can serve any type of virtualization on top, including on hardware or else.

[how]

@dvn, I’m also working at HedgeDoc - Collaborative markdown notes.

[dvn]

@how, I am pretty happy with the state of the announcement. Appreciate your edits. Please feel free to add more or give me comments. I’m a bit stuck at where else to expand it, or if it needs anything else beyond the screencast.

[how]

@dvn

TODO: Add a screencast and/or pointers to step-by-step documentation . IMO a screencast would work best as it was asked for D2.2 as well.

This must be replaced by the screencast.

After lunch I’ll have a look in detail at the announcement and repositories, try to build them, etc.
I think we’re very close. Thank you for the great work @dvn!

[dvn]

Thank you.

I took Saturday off, and yesterday I began working in the evening and realized that my approach to the WireGuard stuff was not working correctly, so I started refactoring it. I think I will be done with that today, and then can merge it into the main branches. Currently the main branches have no WireGuard stuff at all.

[how]

OK. Good to know.

I’m not finding the Rhyzome API spec in OpenAPI format, do you know where it is?

[dvn]

Have it defined in its own repository so that it can be easily imported between projects.

[how]

@dvn I started an MR for rhyzome-openapi to add HTML doc. It’s not ready yet, not sure how far we want to go there, where to host it, etc. The template also comes with default contact information that is not at all appropriate, e.g., email contact is missing (info@entanglement.garden sounds like a good default). Also the licensing is not indicated, maybe something like CC-BY-SA-4.0 for the API? I’m not sure what you’re using now for your docs.

@dvn, I made a few (light) changes to:

• properly document acronyms for screen readers
• add a table of contents
• move schema up a bit, after the explanation and before the details
• bundle OpenWrt + WireGuard into a single section to highlight the benefits for shared hosting

I guess the OpenAPI section could add a blurb on programmatic possibilities… I think something in the line of “how does it compare to Docker or HashiCorp?” maybe in terms of resources and specificity of the scope could be a nice addition.

Otherwise, some consolidated page on SHRUTHI/Rhyzome that includes integrated documentation on installation could be great. I’ll be trying to follow the install docs from zero to completion and see how to do that tomorrow.

Once the screencast is up and the WireGuard part is complete, we’re good to go.

@dvn

notable for the addition of several useful features

Maybe list them again? Repeating is never bad in this case. I find the “Future” section a bit too short. Especially when I read the prospect of “Resources” on the rhyzome API readme.

@dvn

easier to integrate with 3rd party software.

What kind of integration could be envisioned?

[natacha]

ok congratulations @dvn this is a great step, we now have most the building blocks of a DREAM project. Well well we are reaching a step in development and maybe now we can work to build a great narrative for the DEMO.

It feels it is now time to take a pause and think through what has been done, as all the pieces are separated and each affiliated to their own project, I am certain each will pursue its own dynamic, however maybe in some time from now, I hope we will sit together again and envision some common future for DREAM.

[how]

Installation Notes

These are my notes while I’m following the installation steps…

Too many documents to follow: we need a single one going from A to Z.

→ https://gitlab.com/public.dream/shruthi/docs/-/blob/main/getting-started.md

Step 1 - Rhyzome API

→ https://gitlab.com/public.dream/shruthi/rhyzome-api#install

Pre-Requisites

• Go: sudo apt install golang
• Step: see https://smallstep.com/docs/step-cli/installation

  wget https://dl.step.sm/gh-release/cli/docs-cli-install/v0.18.0/step-cli_0.18.0_amd64.deb
  sudo dpkg -i step-cli_0.18.0_amd64.deb
  

• PostgreSQL: sudo apt install postgresql-server

Install

git clone https://gitlab.com/public.dream/shruthi/rhyzome-api
cd rhyzome-api
go build -o rhyzome-api cmd/rhyzome-api/main.go
go build -o rhyzome-grpc cmd/rhyzome-grpc/main.go

Configure

@dvn Is it .conf or .json? The repo has .json and the README uses .conf. ← This is .json (the .conf is not read.) See Misleading configuration information (#1) · Issues · DREAM / SHRUTHI / Rhyzome API · GitLab

→ https://git.callpipe.com/entanglement.garden/rhyzome-api/-/wikis/PKI-Guide

Setup Root CA and Certificates

This step must be done once unless you already have certificates you can use. The following steps will be followed:

1. Create a Root CA
2. Create leaf certificate and bundle it
3. Verify certificate

Script to setup an Example CA for testing:

Steps to setup Root CA, and an example SSL certificate

#!/usr/bin/env bash
set -euo pipefail

mkdir -m 0700 -p ~/.step/certificates
cd $_
echo -e "Generated certs will be stored in: $PWD \n"

echo "# 1. Create a root CA"
cat > ../root.tpl <<EOD
{
  "subject": {
    "commonName": "Example Root CA"
  },
  "issuer": {
    "commonName": "Example Root CA"
  },
  "keyUsage": ["certSign", "crlSign"],
  "basicConstraints": {
    "isCa": true,
    "maxPathLen": 2
  }
}
EOD
step certificate create --insecure --no-password --template ../root.tpl --kty OKP --curve Ed25519 "Example Root CA" root_ca.crt root_ca.key

echo "# 2. Create leaf certificate and bundle it"
step certificate create --insecure --no-password --ca root_ca.crt --ca-key root_ca.key "Example Certificate" leaf.crt example.key
cat leaf.crt root_ca.crt > example.crt

echo "# 3. Verify certificate (returns zero for success)"
step certificate verify --roots root_ca.crt example.crt
if [ $? -eq 0 ];
then 
    echo "Success!"
else
    echo "Failed to verify certificate."
fi

Write configuration file

There is a rhyzome-api.example.json file. Simply copy it to rhyzome-api.json in the same directory and change the paths. With the previous example CA, it may look like:

{
    "db": "dbname=rhyzome host=/var/run/postgresql sslmode=disable",
    "pki": {
        "ca": "/home/demo/.step/certificates/root_ca.crt",
        "cert": "/home/demo/.step/certificates/example.crt",
        "key": "/home/demo/.step/certificates/example.key"
    }
}       

Initialize database

Run: ./db-reset.sh

Usage

@dvn, the configuration file seems not to be taken into account: the certificates are not loaded. Also the JWKS leads to EG, which is probably not what we want since it means nobody will be able to test the setup.

# Start the gRPC service
./rhyzome-grpc
INFO[2021-11-30T12:02:31+01:00]/home/how/src/dream/rhyzome-api/config/config.go Load() successfully read config from rhyzome-api.json
DEBU[2021-11-30T12:02:31+01:00]/home/how/src/dream/rhyzome-api/config/config.go Load() config as loaded: {HTTPBind::8080 GRPCBind::9090 DB:dbname=rhyzome host=/var/run/postgresql sslmode=disable TLS:{Cert: Key: CA:} Keycloak:{JWKS:https://auth.entanglement.garden/auth/realms/entanglement.garden/protocol/openid-connect/certs ResourceScope:} DisableAuthz:true}
FATA[2021-11-30T12:02:31+01:00]/home/how/src/dream/rhyzome-api/grpcserver/server.go ReloadCertificates() failed to read client ca cert:open : no such file or directory

# Start the HTTP API service
./rhyzome-api
INFO[2021-11-30T12:04:52+01:00]/home/how/src/dream/rhyzome-api/config/config.go Load() successfully read config from rhyzome-api.json
DEBU[2021-11-30T12:04:52+01:00]/home/how/src/dream/rhyzome-api/config/config.go Load() config as loaded: {HTTPBind::8080 GRPCBind::9090 DB:dbname=rhyzome host=/var/run/postgresql sslmode=disable TLS:{Cert: Key: CA:} Keycloak:{JWKS:https://auth.entanglement.garden/auth/realms/entanglement.garden/protocol/openid-connect/certs ResourceScope:} DisableAuthz:true}
INFO[2021-11-30T12:04:52+01:00]/home/how/src/dream/rhyzome-api/httpapi/httpapi.go Serve() starting http listener on :8080

Rhyzome LibVirt

→ https://gitlab.com/public.dream/shruthi/rhyzome-libvirt#install

Install

git clone https://gitlab.com/public.dream/shruthi/rhyzome-libvirt
cd rhyzome-libvirt
go build -o rhyzome-libvirt cmd/rhyzome/main.go
package cmd/rhyzome/main.go is not in GOROOT (/usr/lib/go-1.15/src/cmd/rhyzome/main.go)

@dvn ^^ it seems that rhyzome must be installed as a package?

Configure

@dvn here again it uses .conf and a pki key: I think it should be consistent and use .json and a tls key.

I’m using the following configuration:

cat rhyzome-libvirt.json
{
  "bridgeinterface": "virbr0",
  "imagehost": "http://127.0.0.1",
  "diskstoragepool": "virtualmachines",
  "metadataurl": "http://127.0.0.1:8081",
  "metadatabind": ":8081",
  "apiserver": "localhost:9090",
  "tls": {
      "ca": "/home/how/.step/certificates/root_ca.crt",
      "cert": "/home/how/.step/certificates/example.crt",
      "key": "/home/how/.step/certificates/example.key"
  }
}

Not sure about diskstoragepool though…

Image Host

→ https://git.callpipe.com/entanglement.garden/vm-images/image-host
→ https://git.callpipe.com/entanglement.garden/vm-images/common

Common configuration

This must be configured once.

git clone https://git.callpipe.com/entanglement.garden/vm-images/common
cd common
git checkout -b $USER
# Add yourself to the user-data-template.yaml
cat >> user-data-template.yaml <<EOD
  - name: how
    shell: /bin/bash
    groups: [sudo, adm, staff]
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICqimC0Ed1Xw/ghT349oc1Q1kWm+QS4Uq6QdmfAnff9n how@spine
EOD
git add user-data-template.yaml
git commit -m "Add how to the template"

Pre-requisites

• packer: sudo apt install packer

Install

make
packer build -timestamp-ui -var-file ../common/vars.json image-host.json
Error: Failed to prepare build: "qemu"

1 error occurred:
        * Deprecated configuration key: 'iso_checksum_type'. Please call `packer fix`
against your template to update your template to be compatible with the current
version of Packer. Visit https://www.packer.io/docs/commands/fix/ for more
detail.

@dvn: ^^ should probably be fixed, I got packer from Debian stable

[dvn]

Thank you @how. This is immensely helpful. Look at those rough edges!

@how

Too many documents to follow: we need a single one going from A to Z.

So you think the “Getting Started” doc should contain the step-by-step instructions instead of linking to the READMEs?

@how

@dvn Is it .conf or .json ? The repo has .json and the README uses .conf. ← This is .json (the .conf is not read.)

That’s a mistake. It should be .json everywhere.

@how

Setup Root CA and Certificates

@how

@dvn Is it really necessary to follow all these steps for a simple demo? Can’t we bundle an ‘example’ script that will make all the required setup with ‘demo’ passwords?

We can bundle a script, I agree. Thanks for starting on that.

@how

Usage

@dvn, the configuration file seems not to be taken into account: the certificates are not loaded. Also the JWKS leads to EG, which is probably not what we want since it means nobody will be able to test the setup.

Hmmm… I see the problem… I think you need to switch “pki” for “tls” in the config… This should be fixed, along with the JWKS.

[how]

I opened an issue WRT configuration. I now have the api service running but the gRPC remains stuck with not being able to read the private key for some reason (its mode is 0600 FWIW), although it now shows the right paths for all TLS-related configuration.

@dvn

So you think the “Getting Started” doc should contain the step-by-step instructions instead of linking to the READMEs?

Not necessarily. Right now I’m trying to go through the document and complete it. Not working so far, but certainly a step-by-step guide such as the one I’m making above would ease the newcomer’s grab of the software and facilitate people actually trying it out. Not everyone is as stubborn as I am when it comes to trying out new things.

I would have appreciated doing it along as you were coding it, so it would be much easier to understand, but at the same time, I’m now a virgin and encountering the whole thing as anyone else would.

@dvn, please grep your name in the doc above to see new comments.

I’ll probably move this doc to a new topic so we can have a table of contents.

[dvn]

Had a lot of other things come up to do the first half of this week.

Now I’m focused on this and working on fixing all the issues you identified @how.

If we finish Friday, when is the earliest you can submit M3?

[how]

I can only work on DREAM at best from Monday through Wednesday morning.