D2.1 - Towards the deadline

pukkamustard · 28 May 2021 10:16

An internal planning post for what is happening/planned for D2.1 – MVP Alpha (DROMEDAR).

Will post as response per mail. No stable internet.

pukkamustard · 31 May 2021 08:41

With the D2.1 deadline we want to have a minimal working DMC
implementation. The implementations consists of an OCaml library
and a CLI that uses the library.

ocaml-dmc

This is the repository of the OCaml DMC implementation.

See the meeting in
February for
an idea of what the CLI should be able to do. Some things we leave
out.

Todos

Persistent storage for ERIS blocks (@nemael is working on
this)
Persistent storage for DMC objects
Implementation of RDF Signify using ocaml-rdf and
ocaml-monocypher
Sub-commands
- init to initialize a new DMC repository
- add to add an element to a DMC set
- remove to remove an element from a DMC set
- state query the state of a DMC container
Documentation
- Simple walk-trough in the Readme

Things for later (not for D2.1)

DMC registers (D2.1 will only implement DMC sets)
Adding and removing additional authorized public keys to a DMC
container
Transport
Datalog REPL

Considerations

Implementation of DMC logic and sub-commands is fairly
straightforward with the ocaml-rdf library which includes a
Datalog engine. No troubles expected.

Tricky part might be persistent storage. @nemael is already
looking into this.

ocaml-monocypher

Bindings to the Monocypher cryptographic library.

Todos

Implement bindings to Ed25519 functions (required for RDF Signify/DMC).

This todo is straightforward but quite a bit of mechanical work. If anybody is
interested it is up for grabs.

ocaml-eris

OCaml implementation of ERIS.

Todos

Make it work with persistent storage (see notes on ocaml-dmc, @nemael
is on it)

ocaml-rdf

OCaml RDF library. This is the working horse of the DMC implementation and
includes the tricky parts (Turtle serializer and Datalog implementation).

Todos

RDF/Turtle serializer (@arie is working on this)
Content-addressable RDF (implementation of FragmentGraph)
Datalog. ocaml rdf already contains a working Datalog implementation
(taken from
ramsdell/ocaml-datalog).
However it lacks to essential features: Negation and built-in functions.
@pukkamustard is working on implementing Negation and built-in functions.

Considerations

We have fall-backs for the critical components:

If RDF/Turtle does not work out we use RDF/JSON which is already implemented
by @arie
If we fail at implementing negation/built-in functions in the existing
Datalog implementation we can use the
c-cube/datalog
implementation. c-cube/datalog has support for negation and built-in
functions however there are some other things that make it not suitable:
Implementation is fairly complex (compared to ramsdell/ocaml-datalog.
This is partly due to support for negation/built-ins but also because it
allows nested terms (something we do not need/want).
No support for persistent storage. ramsdell/ocaml-datalog also does not
support persistent storage, but it seems easier to implement there as the
implementation is much more readable and hackable (in @pukkamustard’s
opinion).
Nevertheless, @pukkamustard has experimented with c-cube/datalog and thinks
that it can be hacked to work as we want if necessary. This will be the
fallback if we do not manage to implement negation/built-ins by our self.

D2.1 post

A public post/page that introduces and summarizes the work done.

Todos

Write post. Deferred until ocaml-dmc takes on more shape.

pukkamustard · 7 June 2021 07:18

Apparently not the full email made it to the post (it was cut at ocaml-monocypher). I’ve added the missing sections from my original mail.

natacha · 7 June 2021 13:09

Thanks so much for this very clear explanation of the state of work, it is very helpful to see clearly where we’re at and what will be at hand to work with.
I appreciate much that you consider alternative options for the different cases.

This is an interesting choice, I wonder what are the differences between one and another.

Can’t wait to see this happen as it is at the heart of public usage I think.

pukkamustard · 23 June 2021 10:56

This thing is taking shape and I kindly request feedback.

Check it out from DREAM / DROMEDAR / ocaml-dmc · GitLab

and run guix environment -l guix.scm to get a suitable environment. (run guix pull before that - many recent fixes for OCaml in guix upstream).

If all goes well you should be able to run dune exec src/cli/dmc_cli.exe -- set_init to initalize a set.

pukkamustard · 24 June 2021 10:08

Just pushed CBOR serialization. This allows initial primitive transport.

Try and load this (with cbor_load -e raw -f FILENAME)

cbor.dump (4.5 KB)

The secret key for the set (dmc:AAAD3RMUCOD3CH3RO6FQFJ6465ZDLXJCGGBZRWH4HFKM4YEGOOW5C3X56MMTEOE7VWCBFSGBVQQMYTTAQHMT6KTD7KYB3A3RNPP7JVUR4M) is:

urn:ed25519:sk:U6ZU5MNIMQD55L3ZYCUOS6XSGNRXDCVCTR3EOEYTKCUI4I55UG3A

Send me a message!

EDIT: changed to Base32, Base64 was creating trouble
EDIT2: copying and pasting seems to be unreliable. Uploaded a file.

pukkamustard · 24 June 2021 15:13

Sorry for delayed response, but here goes:

A set can hold multiple values whereas a register only holds (at most) one value.

The reason for delaying registers is technical (as mentioned in a call I think). Computing the current value of a register requires taking the maximum over a set of values. Our representation of RDF literals is not yet quite capable of doing this nicely. Of course we have ideas on how to fix this … something for the D2.4 beta.

Usage wise I think sets are more interesting than registers. Sets can model something like a map where there are many parties involved in maintaining the data. In a register it is usually a smaller group of peoples that control the value (e.g. in a user profile it is a single person).

Yes, I agree. Maybe something can go into D2.1…

how · 24 June 2021 16:40

how · 24 June 2021 16:41

/me wants a pukkamustard tee-shirt

I’m on a mobile connection so I’ll check and feedback later.

Nemael · 28 June 2021 09:17

That’s awesome!

I ran guix pull and guix environment -l guix.scm, and they took a while to complete (about 1h30mins), I’m not sure why, maybe it is because I am running guix within a virtual machine.

I could initialize a set correctly!

Oh, and the command I ran was dune exec src/cli/dmc_cli.exe -- set_init instead of dune exec lib/cli/dmc_cli.exe -- set_init, the lib folder does not exist in the repository

Regarding CBOR, when running dune exec src/cli/dmc_cli.exe -- cbor_load -e raw -f cbor.dump, I only obtain this

nemael@nemael-VirtualBox:~/Desktop/DMCFeedback/ocaml-dmc$ dune exec src/cli/dmc_cli.exe -- cbor_load -e raw -f cbor.dump 
Reading from standard input (ctrl-d to terminate).
nemael@nemael-VirtualBox:~/Desktop/DMCFeedback/ocaml-dmc$

So, the standard input open and closes instantly, but I do not get any output. (I verified if the file is correctly located. If I remove it, I get an “Internal error, uncaught exception” on the open command

I also tried to not specify the filename, and use commands such as dune exec src/cli/dmc_cli.exe -- cbor_load -e raw < cbor.dump, or copy-pasting the content using xclip, but the result were the same.
If I do the same, but leave the standard input empty, I do get an “Invalid cbor” entry.

Maybe it is something on my side that’s cancelling the display of the output? I looked around, but I couldn’t see any tips on that. Maybe I also am using the wrong command, is there a different one specifically to display the output?

pukkamustard · 28 June 2021 10:11

That is a bit too long. Most stuff should be pre-compiled and available from substitutes. OCaml dependencies compile rather fast (even on low-powered machines). You may have recompiled the OCaml compiler which takes some time…but I don’t know why.

Fixed in post above. thanks!

This was an error in logging. dmc_cli says it was reading from stdin even if it was not. I pushed a fix with adafaceeab473f2ddf4d9ca915f3d8eeea77ca7b.

Try running this again:

dune exec src/cli/dmc_cli.exe -- cbor_load -e raw -f cbor.dump

The cli does not tell you that anything happens if it all goes ok (this should be improved!). But you can get some more information on what’s going on with:

dune exec src/cli/dmc_cli.exe -- cbor_load -e raw -f cbor.dump -v

(and even more with -v -v).

pukkamustard · 28 June 2021 10:13

Another thing to try:

dune exec src/cli/dmc_cli.exe -- authorized_keys_list dmc:AAAD3RMUCOD3CH3RO6FQFJ6465ZDLXJCGGBZRWH4HFKM4YEGOOW5C3X56MMTEOE7VWCBFSGBVQQMYTTAQHMT6KTD7KYB3A3RNPP7JVUR4M

And the authorize_new command…

Nemael · 28 June 2021 10:26

I pulled the new commit, setup guix environment again (it only took about 5 minutes this time, as you said, I maybe rebuilt the ocaml compiler, or maybe the os while I’m at it.)

After that, it worked flawlessly.
dune exec src/cli/dmc_cli.exe -- cbor_load -e raw -f cbor.dump gave me the output Reading from cbor.dump.. Using -v and -v -v did output a lot more debug information.

The authorize_new command also worked, and outputted an urn urn:ed25519:pk:SIYYE2Q6QLEL4Y2VZYIMMWSMMN7NNJWRQDM6RRV4ZQAGNFZ4QGTA

misterfish · 28 June 2021 15:24

Still having Guix problems

Output:

It is true that I change between OCaml switches fairly often, including changing compiler versions. But how to do make Guix aware of this? It is still pointing objects in /gnu/store which of course do not realise that the compiler has changed.

Note that adding --pure gives the same result.

misterfish · 28 June 2021 15:37

Fixed after convo with @pukkamustard. Solution:

guix environment -l guix.scm -- bash --noprofile --norc

misterfish · 28 June 2021 16:01

cbor.dump

how · 28 June 2021 16:48

Here is my results for the proposed commands. The last one ~~is failing, I tried authorizing @Nemael’s key posted above… Maybe the syntax is not right…~~ was failing but I could resolve why.

bash-5.0$ dune exec src/cli/dmc_cli.exe -- set_init
Set identifier: dmc:AAAPRUJTY772WDVDZU7YTFJ6FQH34HURZFSDB6LQ6XPZ5M6TGYKJ4QJT7VBCOUVZHCV3H5ZRVCBPZSELIPXIPTLYJ54ZQ34PGZSWRZDQLQ
Secret key (do not loose or share!): urn:ed25519:sk:SMU6FSM6QTSGCPW2OZE5ACAG675MAJEY2GXPEAOCQ3SSCFYKS2ZQ
<triple
        <iri
            dmc:AAAPRUJTY772WDVDZU7YTFJ6FQH34HURZFSDB6LQ6XPZ5M6TGYKJ4QJT7VBCOUVZHCV3H5ZRVCBPZSELIPXIPTLYJ54ZQ34PGZSWRZDQLQ>
        <iri http://purl.org/dmc/ns#rootPublicKey>
        <iri
            urn:ed25519:pk:TVHJB7ABJXMOACNCBKSGAM3ZA2Y3ZVNAKX2S2CL7HJJ6Z27Z2CYA>>

<triple
        <iri
            dmc:AAAPRUJTY772WDVDZU7YTFJ6FQH34HURZFSDB6LQ6XPZ5M6TGYKJ4QJT7VBCOUVZHCV3H5ZRVCBPZSELIPXIPTLYJ54ZQ34PGZSWRZDQLQ>
        <iri http://www.w3.org/1999/02/22-rdf-syntax-ns#type>
        <iri http://purl.org/dmc/ns#Set>>

$ dune exec src/cli/dmc_cli.exe -- cbor_load -e raw -f cbor.dump -v
Reading from cbor.dump.
dmc_cli.exe: [INFO] [dmc] v fresh=false readonly=false log_size=500000
dmc_cli.exe: [INFO] Loading objects and blocks for container dmc:AAAD3RMUCOD3CH3RO6FQFJ6465ZDLXJCGGBZRWH4HFKM4YEGOOW5C3X56MMTEOE7VWCBFSGBVQQMYTTAQHMT6KTD7KYB3A3RNPP7JVUR4M
dmc_cli.exe: [INFO] tree commited (message: cbor_load)

$ dune exec src/cli/dmc_cli.exe -- authorized_keys_list  dmc:AAAPRUJTY772WDVDZU7YTFJ6FQH34HURZFSDB6LQ6XPZ5M6TGYKJ4QJT7VBCOUVZHCV3H5ZRVCBPZSELIPXIPTLYJ54ZQ34PGZSWRZDQLQ
urn:ed25519:pk:TVHJB7ABJXMOACNCBKSGAM3ZA2Y3ZVNAKX2S2CL7HJJ6Z27Z2CYA

Failing (mostly for lack of docs)

$ dune exec src/cli/dmc_cli.exe -- authorize_new urn:ed25519:pk:SIYYE2Q6QLEL4Y2VZYIMMWSMMN7NNJWRQDM6RRV4ZQAGNFZ4QGTA
dmc: internal error, uncaught exception:
     Dmc__Dmc_eris.Invalid_read_capability

$ dune exec src/cli/dmc_cli.exe -- authorize_new dmc:AAAPRUJTY772WDVDZU7YTFJ6FQH34HURZFSDB6LQ6XPZ5M6TGYKJ4QJT7VBCOUVZHCV3H5ZRVCBPZSELIPXIPTLYJ54ZQ34PGZSWRZDQLQ
dmc: required option --secret-key is missing
Usage: dmc authorize_new [OPTION]... CONTAINER_IDENTIFIER
Try `dmc authorize_new --help' or `dmc --help' for more information.

Resolving

The first time I tried authorize_new $KEY, but after reading the help I figured it would be the dmc: container set identifier. Still, the help did not mention --secret-key so I still am confused about this command. If I were to add @Nemael’s key to the container set, I’d certainly have to do --secret-key $MYKEY first… So I realized I cannot authorize an existing key, but instead a new one that’s created on the fly. Makes sense once you know it

$  dune exec src/cli/dmc_cli.exe -- authorize_new --secret-key urn:ed25519:sk:SMU6FSM6QTSGCPW2OZE5ACAG675MAJEY2GXPEAOCQ3SSCFYKS2ZQ dmc:AAAPRUJTY772WDVDZU7YTFJ6FQH34HURZFSDB6LQ6XPZ5M6TGYKJ4QJT7VBCOUVZHCV3H5ZRVCBPZSELIPXIPTLYJ54ZQ34PGZSWRZDQLQ
Public key urn:ed25519:pk:EAD5G3RE42DRGEMVAK73XWEGHCLKVSWW4OOS75VWKPBJMQJWE7WA authorized for container dmc:AAAPRUJTY772WDVDZU7YTFJ6FQH34HURZFSDB6LQ6XPZ5M6TGYKJ4QJT7VBCOUVZHCV3H5ZRVCBPZSELIPXIPTLYJ54ZQ34PGZSWRZDQLQ

Secret key (do not loose or share!): urn:ed25519:sk:PCB2BUJZJ6K3MGNFE3YNMAJARQJ4REASJOHGVT6EWKIFDA2RYNZA

Still, if I need a secret key to authorize a new key, then the new secret key is known to me (so I can act as if I were the owner)… I guess this is something we figure out after alpha

pukkamustard · 29 June 2021 08:04

Good point. Luckily this is just a shoddy implementation in the CLI. The dmc library allows authorizing a public key without knowing the secret key (https://inqlab.net/projects/ocaml-dmc/dmc/Dmc/module-type-S/AuthorizedKeys/index.html#val-authorize).

Currently the CLI can not take a public key as argument but just generates a new random key (src/cli/dmc_cli.ml · main · DREAM / DROMEDAR / ocaml-dmc · GitLab). Implementing this in the CLI is a good first step into the codebase ;)…

pukkamustard · 29 June 2021 09:32

@arie just finished work on the RDF Turtle serialization and it is now in ocaml-dmc!

This makes the output much easier to read:

$ dune exec src/cli/dmc_cli.exe -- resolve dmc:AAAD3RMUCOD3CH3RO6FQFJ6465ZDLXJCGGBZRWH4HFKM4YEGOOW5C3X56MMTEOE7VWCBFSGBVQQMYTTAQHMT6KTD7KYB3A3RNPP7JVUR4M


<dmc:AAAD3RMUCOD3CH3RO6FQFJ6465ZDLXJCGGBZRWH4HFKM4YEGOOW5C3X56MMTEOE7VWCBFSGBVQQMYTTAQHMT6KTD7KYB3A3RNPP7JVUR4M>
 <http://purl.org/dmc/ns#member> <urn:erisx2:AAANBI56OPVF4DLM56RIGA33JHP4PQPXLCBR3SO2GB3L3OGTYMQMSGCAVZYC3RIWEOHJ2R57ALLIMB3Y6AB6ULZXNGPEYEYGAKMBAHEJMY>,
                                  <urn:erisx2:AAAPQ2OEC26URJJKOQ42TZG6H5S6WAGZITTSDAAS7V62OETFWNMZEEWYRC6FHEAMHLOSHDHITLUEWNRRLOPNY6AX4XD5NMDSWUM6UVLAKM> ;
  <http://purl.org/dmc/ns#rootPublicKey> <urn:ed25519:pk:SIYYE2Q6QLEL4Y2VZYIMMWSMMN7NNJWRQDM6RRV4ZQAGNFZ4QGTA> ;
  <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://purl.org/dmc/ns#Set> .

misterfish · 30 June 2021 09:25

you call that easy to read?