DREAM D1.2 - UPSYCLE Architecture & System Design Overview
tl;dr: 2021-04-30 – DREAM releases D1.2, the UPSYCLE Architecture & System Design Overview.
Introduction
In the context of DREAM, we’re aiming to enable offline-first, decentralized group collaboration.
This deliverable provides an overview of UPSYCLE design and how the various components of DREAM interact with each other.
Research Problem
The current Internet stack is based on a centralized, host-based, one-to-one communication pattern, while most of our communications involve interaction within groups.
To address the scalability and reliability issues of centralized approaches, publish-subscribe in a decentralized setting has been extensively researched, however often times they lack attack resiliency and privacy properties that would be necessary for real-world deployments.
Current public Peer-to-peer (P2P) networks lack support for private, secure, and scalable group collaboration, and their outlooks often derive from individualist concerns as exemplified by the adoption of self-interest-driven design considerations stemming from free market and game theories.
Desirable properties
Decentralized publish-subscribe overlays should satisfy a number of desirable properties:
- scalability in terms of number of nodes, topics, subscribers per topic, and subscriptions per node
- relay-free routing (or topic-connected overlay (TCO)): only subscribers of a topic participate in routing events for that topic
- bounded node degrees: the number of connections maintained by a node must be limited so that the network can scale
- completeness of event dissemination: all published events should be delivered to all subscribers
- low duplication factor of event dissemination
- low latency of event dissemination[1]
- robustness of the overlay: quick recovery after failures and churn.
Equally important but not often considered:
- attack resilience: the overlay should be resilient to known attacks
- subscription privacy: nodes’ subscriptions should be private, only common topic subscriptions between two nodes should be discoverable.
Research Focus
DREAM research aims to implement these desirable properties to solve a number of identified problems.
We propose an original commons-based approach to P2P network design building on different premises, where peers collaborate in groups around voluntary subscription to specific topics. Topic-based publish-subscribe has wide range of applications including event notifications, database updates, messaging, social interactions and group collaboration, and more recently in Internet of Things and Distributed Ledger Technologies as well.[2]
Problem 1: Assuming Always-On Connectivity
Most existing decentralized publish-subscribe designs rely on a synchronous P2P architecture and assume always-on connectivity, which limits the possibilities of collaboration among people.
A few notable exceptions to this pattern take an offline-first approach, notably Briar[3], Syncthing[4], or Secure ScuttleButt and the Dat protocol[5].
Problem 2: Inadequate Economic Models & Incentives
Although Peer-to-Peer networks can be considered as digital commons, most economic models stem from market theory and game theory: in public P2P systems, every peer provides service to everyone else, leading to an increasing free-rider problem as the networks grow in size, and diminishing incentives to provide more resources to the network.
In order to alleviate these and other issues, many systems require elaborate economic models based on “hypercompetitive” markets and incentive systems that rely on expensive distributed ledger technologies and consensus algorithms.
Earlier peer-to-peer systems were using best-effort algorithms and lacked incentives to integrate low-resource devices, either in storage capacity, disposable bandwidth, or connectivity, which were disadvantaged. New cryptocurrency-based incentives[6] try to mitigate this by allowing peers to get paid for sharing resources, creating a highly speculative storage market where larger relays gain a strong asymmetry over other alledgedly equal peers.
Problem 3: Lack of Strong Group Security Properties
Group collaboration in a decentralized environment remains a hard problem.
Research Result
Architecture Overview
1. An Edge Node
Alice is preparing a commons-based peer production (CBPP) workshop to locally contribute to a commons mapping project. Her laptop will serve during the workshop to aggregate information from her peers. Alice publishes a private “CBPP Workshop” topic. That creates a local container (DMC) specifically for this topic. Alice writes and publishes an intructions document.
2. An Edge Network
As Bob arrives at the workshop, Alice gives him a QR code to scan. Now Bob has subscribed to the “CBPP Workshop” topic from his phone, which now has its own local DMC replica. DMC uses the publish-subscribe protocol to disseminate operations to all group members, so Bob can now read Alice’s instructions for the workshop. There’s a section for self-introductions, so he adds his name and a short bio, and publishes the instructions back to the “CBPP Workshop” topic. As Alice started the topic, she’s also a subscriber, so she receives the update!
3. A Group
A group is the ensemble of publishers and subscribers to a topic. In our current story, Alice and Bob form the group around the “CBPP Workshop” topic.
They can work together freely and eventually invite new people to the group by announcing the topic (either privately, as when Bob joined the group, or publicly). In order to do this, they could gather more nodes on the edge network, but they could also connect to the core network and reach out to more people across the Internet.
4. The Core Network
The core network is comprised of core nodes that are called like this because altogether they form a backbone of the peer-to-peer network. Core nodes differ from edge nodes in several ways:
- a core node is always connected (or tries to be): that means it probably runs in a data center (ISP, hospital, university…), your local hackerspace, or on a SOHO connection that runs permanently ;
- a core node keeps track of edge nodes connecting to it, so it can store-and-forward messages that came for these edge nodes while they were offline ;
- a core node keeps track of topics that its edge nodes are subscribed to, and participate in the dissemination of these topic’s data to other core nodes and to its local edge subscribers ;
- a core node also transmits messages that it does not know what to do with to other core nodes until eventually one core node knows what to do with it (because it has subscribers).
So, the core network is intended to provide a permanent buffer for edge nodes so that no message is lost because of intermittent connectivity, thus enables asynchronous communication. In practice, their routing and store-and-forward activity is transparent to the network: they act as proxies or aggregators, but do not produce, consume, nor access any contents.[7]
Additionally, the core network work on a different overlay than edge networks, allowing edge nodes to preserve their location privacy.
5. Topic Synchronization and Storage
Now that the topic is announced to the core network, any edge node that knows about our “CBPP Workshop” topic can request its core node to subscribe to that topic. Only once a core node has been requested to subscribe to a specific topic by one of its edge nodes, does it start participating in the topic’s dissemination: this property is called relay-free dissemination and contrasts with other P2P networks where peers may relay any data to any other peer.
An edge node stores data from its subscriptions in a Distributed Mutable Container (DMC), one for each topic. Here this sample node has three subscriptions. In reality you could have hundreds or thousands of them.
DMC uses data structures that have a property called eventual consistency: if Alice and Bob lose track of each other and continue working on “CBPP Workshop” separately, when they come back together, the result of their distinct work is guaranteed to be mergeable without conflict (hence the name of Conflict-Free Replicated Datatype (CRDT)).
DMC uses the publish-subscribe protocol to send and receive CRDT operations that modify stored data structures in all replicas: each group member’s local copy is affected and synchronized. Not only for Alice or Bob, but for all group members, for all the time. You could go on a trail, document the flora for a week, come back to the city and bring your findings to your Flora Observation Group without a glitch – while receiving what the rest of the group would have produced in the meantime.
6. Topic Access Control[8]
Already this sounds mind-boggling, how can thousands of people share something and it keeps being consistent. Maybe you’ve participated in editing a wiki, a shared document, or code using Git. Merging concurrent changes is often a problem. We’re not going to dive into CRDTs just yet, but you can count on them to keep your mind straight.
Instead we’re going to make a step back and consider a private group. Because private groups are much more common in our daily lives: working with colleagues, planning a meeting with friends, etc. Until now we assumed a public topic “CBPP Workshop”.
So let’s make a private topic “Birthday Party”. This topic is only available to our small group of friends. One interesting properties of UPSYCLE is that if you don’t know what you’re looking for, unless it’s publicly discoverable, you will not be able to find it. So our “Birthday Party” topic is safe from prying eyes, and only these nodes that participate in the group can transmit data about it.
This means we need some way of making this private topic available for discovery by people we want to subscribe. For this there are different strategies under way, as part of NGI0 P2Pcollab research to implement several discovery and dissemination protocols. Until then we’ll use out-of-band (OOB) means, such as scanning a QR code that was mentioned earlier[1:1], or give a specially crafted URL to someone written on paper, spoken or exchanged over delta.chat or Signal. It’s important to keep it secret, otherwise it defeats the idea of a private group, right?
Access control is not limited to subscribing, it can as well concern the rights to publish (or otherwise change) what’s in the topic. Maybe you want to keep Carol from reading her own “Birthday Party” topic until next Friday, but still be able to send messages to it: so she may send directions and respond to gossip without actually know what Alice and Bob have been preparing for her… Access control deals with all kinds of restrictions, or, in more optimist terms: all kinds of capabilities.
7. Provisioning and Deployment
This system seems to be a complex one, right? Two-tier P2P network, public-key addressing, content-addressed storage, CRDTs, etc. the number of concepts can be daunting. Fortunately we also provide ways to create optimized machines dedicated to certain tasks in the network: some will run in the data center, others on your home router or your phone. With SHRUTHI (upcoming D1.3), we can declare what needs to be done and generate unikernels at will from code. Contributing to the network either as a service provider (for core nodes), as a publisher (for edge nodes, client applications) or to extend its capabilities can be done by packaging intent into dedicated virtual machines.
How UPSYCLE Solves our Research Problems
The main outcome of the research being conducted within DREAM is a specification for a set of protocols that address all these issues: a Ubiquitous Publish-Subscribe Infrastructure for Collaboration on Edge Networks (UPSYCLE), including the multicast message routing for P2P publish/subscribe paper that documents the specification for the message router service.
We present a design for a multicast message routing infrastructure to be used together with decentralized publish/subscribe protocols in a two-tier P2P system, composed of a stable core network and intermittently connected edge networks, where asynchronous communication is made possible by core nodes providing store-and-forward service to edge nodes with end-to-end security guarantees.
UPSYCLE is a P2P, relay-free, broker-based publish-subscribe protocol suite that provides a scalable, public key-addressed, end-to-end secure group communication primitive which keeps user location & subscriptions private. It enables direct communication of mobile nodes on edge networks and also offers asynchronous communication with remote nodes on the internet via store-and-forward proxies that form a core P2P network and keep encrypted messages for the mobile nodes while they are offline.
A proxy acts on behalf of a user in the P2P core network. It collects subscriptions and facilitates publishing, search, and discovery of content. A user may use multiple proxies for redundancy or for separate identities. Proxies can be run by users, communities, or commercial providers.
UPSYCLE aims to address all 3 research problems:
- A topic-connected overlay realizes relay-free routing: participating nodes only relay what they explicitly care about
- A two-tier P2P network enables asynchronous communication and offline-first capability.
- A service provider economic model ensures optimal quality of service and data sovereignty for users.
- A composable and modular system provide cryptographic, privacy, and other desirable properties.
- A multicast system supports groups, including member revocation and membership privacy.
UPSYCLE can solve these problems by composing with the following sorcery:
Relay-Free Routing
Topic-connected overlays (TCOs) mean that nodes only participate in the forwarding and storage of data that they explicitly subscribed to: peers do not relay traffic they are not interested in, thereby reducing latency and traffic overhead, and increasing privacy.
Two-Tier P2P Network
In order to enable asynchronous communication even on resource-constrained, mobile end-user devices we consider a two-tier P2P architecture, as proposed in UPSYCLE that consist of a stable, always online P2P core network, along with several ephemeral edge networks. The core and edge networks run distinct P2P dissemination protocols, and edge nodes reach remote nodes via a core node that provides a multicast routing service with store-and-forward message queues.
Next to connecting to proxies, mobile nodes can also maintain direct P2P connections with other nodes on the local network where they participate in similar P2P protocols to the ones in the core network. This allows local collaboration, even without internet connectivity.
Service Provider Model
In contrast with public P2P networks using market-based economic models, DREAM considers the P2P network as a common pool resource (CPR), and relies on mutual agreements among users of the CPR to ensure its permanence, following the model of institutional arrangements for the commons.
Relay-Free Routing provides an inherent incentive to participating nodes, since they only share resources for topics they care about (to which they subscribe to). Thus the incentive becomes, if you care about something you allocate resources for it.
Participants in the network can choose one or more core nodes according to their needs (trust, reliability…).
It’s important to note here that these [core network] proxies only perform store-and-forward message relaying, and cryptographic user and group identities are independent of them. This allows a mobile node to choose a different proxy at any moment, or even to use multiple proxies for redundancy.
DMC provide assurance to users that data remains independent from the choice of core nodes: they can change providers at any time, driving service quality towards excellence. SHRUTHI unikernel provisioning enables a level-playing fields where independent offers can thrive.
This approach avoids the need to maintain a costly economic model for resource allocation. The economic models proposed by IPFS and Maidsage with proof-of-storage/replication are useful for networks where data is dispersed in the whole network, while in DREAM’s case nodes have more influence on data locality and choosing intermediaries that are willing to host their data.
Composable and Modular System
The Software architecture running on each P2P node is based on the actor model where software components, both local and remote, communicate with each other via message passing.
This enables a composable and modular system, where message type definitions establish a clear interface between services, and allows flexibility in organizing and connecting the infrastructure to suit deployment in different environments, as well as makes it possible for different implementations in different languages to interface with each other. This architecture enables security and fault isolation of components provided services run in securely sandboxed environments.
UPSYCLE minimizes the amount of information available to intermediaries about communicating parties.
Native Decentralized Group Support
UPSYCLE is designed to enable end-to-end security for groups, and supports members leaving a group[9]:
Message payloads sent to the group are encrypted using a decentralized group encryption scheme […] [that] relies on a causal broadcast primitive that the underlying p2p pub/sub infrastructure provides, and implements a decentralized membership management protocol.
Each pub/sub topic uses a multicast group for message dissemination shared by the current members of the group. Whenever there’s a membership change that involves revoking access, a new multicast group is set up without the excluded members, using an end-to-end encrypted message to the remaining group members who instruct their local node and core nodes they’re relying on to join the new multicast group.
Eventual consistency with causal delivery is a key building block for this protocol suite, as well as public key-addressed user and group identities.
Applied to the two-tier P2P setting, this protocol suite enables end-to-end secure communication channels directly between end-user devices, without proxies being able to decrypt application messages.
Group memberships remain private: only group members are able to discover fellow group members.
- interest clustering enables nodes that subscribe to the same groups to discover each other: core nodes can discover other core nodes, while edge nodes can discover other edge nodes on the same LAN;
- explicit subscription of an edge node to a topic enable its core node to know about this group membership
The P2P transport provides client authentication and forward secrecy thanks to TLS 1.3.
Group membership management design follows the ground-breaking DSGM paper to provide confidentiality, authentication, integrity, forward secrecy, post-compromise security, and eventual consistency.
Previous Work
DREAM research on UPSYCLE benefits from long-standing research on P2P group collaboration conducted at P2Pcollab.net, and the related NGI0 P2Pcollab grant that provides the core libraries for gossip-based P2P discovery and dissemination protocols.
The grounded focus on local knowledge and collective practices would not be possible without the DREAM Team’s first-hand experience of grassroots living collectives and ongoing research about the commons and citizen engagement.
The interested reader can refer to the references section of the initial protocol design proposal and the related work sections of the design overview for background research that we build upon.
Outlook
D2.2 -- MVP Alpha (UPSYCLE) (due: 2021-08-27) will implement the publish-subscribe pattern on the edge network, including the Message Router (with TLS 1.3 for Perfect Forward Secrecy (PFS), CBOR) serialization, peer-sampling and clustering, a key-value store and unikernels for all components. It will integrate https://dream.public.cat/pub/dream-data-spec to support Distributed Mutable Containers (DMC).
https://dream.public.cat/t/d2-4-mvp-beta-release/156 (due: 2021-11-12) will implement Pub/Sub on the core network, including the Store-and-Forward Proxy (SFP), and unikernel provisioning with SHRUTHI. Some of these components will be demonstrated by the end of October in D3.1 – DREAM Beta Demo. The final D1.4 -- DREAM Specification (due: 2021-12-31) will consolidate this document with the advancements made through the NGI0 P2Pcollab research and DREAM development and community demo.
You may follow updates in the related software repositories.
We welcome feedback and criticism! Our forum is open for friendly cooperation among DREAM Catchers. Do not hesitate to contact us and contribute to the code: our research is made to improve the digital commons.
this only concerns the performance of the P2P dissemination protocol. It does not affect anonymity, but as core and edge nodes are on different overlays, an edge node could use a anonymity protocol (such as Tor) to connect to its core node(s). ↩︎ ↩︎
UPSYCLE: Ubiquitous Publish-Subscribe Infrastructure for Collaboration on Edge Networks ↩︎
Briar is closest to DREAM, but its data synchronization capabilities do not make use of CRDTs (see D1.1). Briar could eventually be used to synchronize Distributed Mutable Containers. ↩︎
Syncthing has limited functionality (file exchange), but shares some design decisions such as using cryptographic public keys to identify peers, and using TLS channels. ↩︎
Both SSB and DATuse one-to-many append-only logs, while DREAM uses CRDTs to enable group collaboration. ↩︎
BTT, Filecoin, Maidsafecoin (note that the successor of Maidsafecoin is not backed by a blockchain), etc. ↩︎
It may occur that a core node is also a member of a group, and then it can access the contents, e.g., in the case of serving a public website based on DMC data, e.g., a public map. ↩︎
Note that access control is not to be confused with ACLs. ↩︎
https://dream.public.cat/pub/multicast-message-routing-for-p2p-pubsub/#ref-dsgm ↩︎
