D2.1 - Towards the deadline

An internal planning post for what is happening/planned for D2.1 – MVP Alpha (DROMEDAR).

Will post as response per mail. No stable internet.

1 Like

With the D2.1 deadline we want to have a minimal working DMC
implementation. The implementations consists of an OCaml library
and a CLI that uses the library.

ocaml-dmc

This is the repository of the OCaml DMC implementation.

See the meeting in
February
for
an idea of what the CLI should be able to do. Some things we leave
out.

Todos

  • Persistent storage for ERIS blocks (@nemael is working on
    this)
  • Persistent storage for DMC objects
  • Implementation of RDF Signify using ocaml-rdf and
    ocaml-monocypher
  • Sub-commands
    • init to initialize a new DMC repository
    • add to add an element to a DMC set
    • remove to remove an element from a DMC set
    • state query the state of a DMC container
  • Documentation
    • Simple walk-trough in the Readme

Things for later (not for D2.1)

  • DMC registers (D2.1 will only implement DMC sets)
  • Adding and removing additional authorized public keys to a DMC
    container
  • Transport
  • Datalog REPL

Considerations

Implementation of DMC logic and sub-commands is fairly
straightforward with the ocaml-rdf library which includes a
Datalog engine. No troubles expected.

Tricky part might be persistent storage. @nemael is already
looking into this.

ocaml-monocypher

Bindings to the Monocypher cryptographic library.

Todos

  • Implement bindings to Ed25519 functions (required for RDF Signify/DMC).

This todo is straightforward but quite a bit of mechanical work. If anybody is
interested it is up for grabs.

ocaml-eris

OCaml implementation of ERIS.

Todos

  • Make it work with persistent storage (see notes on ocaml-dmc, @nemael
    is on it)

ocaml-rdf

OCaml RDF library. This is the working horse of the DMC implementation and
includes the tricky parts (Turtle serializer and Datalog implementation).

Todos

  • RDF/Turtle serializer (@arie is working on this)
  • Content-addressable RDF (implementation of FragmentGraph)
  • Datalog. ocaml rdf already contains a working Datalog implementation
    (taken from
    ramsdell/ocaml-datalog).
    However it lacks to essential features: Negation and built-in functions.
    @pukkamustard is working on implementing Negation and built-in functions.

Considerations

We have fall-backs for the critical components:

  • If RDF/Turtle does not work out we use RDF/JSON which is already implemented
    by @arie
  • If we fail at implementing negation/built-in functions in the existing
    Datalog implementation we can use the
    c-cube/datalog
    implementation. c-cube/datalog has support for negation and built-in
    functions however there are some other things that make it not suitable:
  • Implementation is fairly complex (compared to ramsdell/ocaml-datalog.
    This is partly due to support for negation/built-ins but also because it
    allows nested terms (something we do not need/want).
  • No support for persistent storage. ramsdell/ocaml-datalog also does not
    support persistent storage, but it seems easier to implement there as the
    implementation is much more readable and hackable (in @pukkamustard’s
    opinion).
    Nevertheless, @pukkamustard has experimented with c-cube/datalog and thinks
    that it can be hacked to work as we want if necessary. This will be the
    fallback if we do not manage to implement negation/built-ins by our self.

D2.1 post

A public post/page that introduces and summarizes the work done.

Todos

  • Write post. Deferred until ocaml-dmc takes on more shape.
2 Likes

Apparently not the full email made it to the post (it was cut at ocaml-monocypher). I’ve added the missing sections from my original mail.

Thanks so much for this very clear explanation of the state of work, it is very helpful to see clearly where we’re at and what will be at hand to work with.
I appreciate much that you consider alternative options for the different cases.

This is an interesting choice, I wonder what are the differences between one and another.

Can’t wait to see this happen as it is at the heart of public usage I think.

This thing is taking shape and I kindly request feedback.

Check it out from DREAM / DROMEDAR / ocaml-dmc · GitLab

and run guix environment -l guix.scm to get a suitable environment. (run guix pull before that - many recent fixes for OCaml in guix upstream).

If all goes well you should be able to run dune exec src/cli/dmc_cli.exe -- set_init to initalize a set.

1 Like

Just pushed CBOR serialization. This allows initial primitive transport.

Try and load this (with cbor_load -e raw -f FILENAME)

cbor.dump (4.5 KB)

The secret key for the set (dmc:AAAD3RMUCOD3CH3RO6FQFJ6465ZDLXJCGGBZRWH4HFKM4YEGOOW5C3X56MMTEOE7VWCBFSGBVQQMYTTAQHMT6KTD7KYB3A3RNPP7JVUR4M) is:

urn:ed25519:sk:U6ZU5MNIMQD55L3ZYCUOS6XSGNRXDCVCTR3EOEYTKCUI4I55UG3A

Send me a message!

EDIT: changed to Base32, Base64 was creating trouble
EDIT2: copying and pasting seems to be unreliable. Uploaded a file.

Sorry for delayed response, but here goes:

A set can hold multiple values whereas a register only holds (at most) one value.

The reason for delaying registers is technical (as mentioned in a call I think). Computing the current value of a register requires taking the maximum over a set of values. Our representation of RDF literals is not yet quite capable of doing this nicely. Of course we have ideas on how to fix this … something for the D2.4 beta.

Usage wise I think sets are more interesting than registers. Sets can model something like a map where there are many parties involved in maintaining the data. In a register it is usually a smaller group of peoples that control the value (e.g. in a user profile it is a single person).

Yes, I agree. Maybe something can go into D2.1…

1 Like

/me wants a pukkamustard tee-shirt :slight_smile:

I’m on a mobile connection so I’ll check and feedback later.

1 Like

That’s awesome!

I ran guix pull and guix environment -l guix.scm, and they took a while to complete (about 1h30mins), I’m not sure why, maybe it is because I am running guix within a virtual machine.

I could initialize a set correctly!

Oh, and the command I ran was dune exec src/cli/dmc_cli.exe -- set_init instead of dune exec lib/cli/dmc_cli.exe -- set_init, the lib folder does not exist in the repository :slight_smile:


Regarding CBOR, when running dune exec src/cli/dmc_cli.exe -- cbor_load -e raw -f cbor.dump, I only obtain this

nemael@nemael-VirtualBox:~/Desktop/DMCFeedback/ocaml-dmc$ dune exec src/cli/dmc_cli.exe -- cbor_load -e raw -f cbor.dump 
Reading from standard input (ctrl-d to terminate).
nemael@nemael-VirtualBox:~/Desktop/DMCFeedback/ocaml-dmc$ 

So, the standard input open and closes instantly, but I do not get any output. (I verified if the file is correctly located. If I remove it, I get an “Internal error, uncaught exception” on the open command

I also tried to not specify the filename, and use commands such as dune exec src/cli/dmc_cli.exe -- cbor_load -e raw < cbor.dump, or copy-pasting the content using xclip, but the result were the same.
If I do the same, but leave the standard input empty, I do get an “Invalid cbor” entry.

Maybe it is something on my side that’s cancelling the display of the output? I looked around, but I couldn’t see any tips on that. Maybe I also am using the wrong command, is there a different one specifically to display the output?

That is a bit too long. Most stuff should be pre-compiled and available from substitutes. OCaml dependencies compile rather fast (even on low-powered machines). You may have recompiled the OCaml compiler which takes some time…but I don’t know why.

Fixed in post above. thanks!

This was an error in logging. dmc_cli says it was reading from stdin even if it was not. I pushed a fix with adafaceeab473f2ddf4d9ca915f3d8eeea77ca7b.

Try running this again:

dune exec src/cli/dmc_cli.exe -- cbor_load -e raw -f cbor.dump

The cli does not tell you that anything happens if it all goes ok (this should be improved!). But you can get some more information on what’s going on with:

dune exec src/cli/dmc_cli.exe -- cbor_load -e raw -f cbor.dump -v

(and even more with -v -v).

Another thing to try:

dune exec src/cli/dmc_cli.exe -- authorized_keys_list dmc:AAAD3RMUCOD3CH3RO6FQFJ6465ZDLXJCGGBZRWH4HFKM4YEGOOW5C3X56MMTEOE7VWCBFSGBVQQMYTTAQHMT6KTD7KYB3A3RNPP7JVUR4M

And the authorize_new command…

I pulled the new commit, setup guix environment again (it only took about 5 minutes this time, as you said, I maybe rebuilt the ocaml compiler, or maybe the os while I’m at it.)

After that, it worked flawlessly.
dune exec src/cli/dmc_cli.exe -- cbor_load -e raw -f cbor.dump gave me the output Reading from cbor.dump.. Using -v and -v -v did output a lot more debug information.

The authorize_new command also worked, and outputted an urn urn:ed25519:pk:SIYYE2Q6QLEL4Y2VZYIMMWSMMN7NNJWRQDM6RRV4ZQAGNFZ4QGTA

Still having Guix problems :confused:

Output:

It is true that I change between OCaml switches fairly often, including changing compiler versions. But how to do make Guix aware of this? It is still pointing objects in /gnu/store which of course do not realise that the compiler has changed.

Note that adding --pure gives the same result.

Fixed after convo with @pukkamustard. Solution:

guix environment -l guix.scm -- bash --noprofile --norc

1 Like

cbor.dump

:partying_face:

1 Like

Here is my results for the proposed commands. The last one is failing, I tried authorizing @Nemael’s key posted above… Maybe the syntax is not right… was failing but I could resolve why.

bash-5.0$ dune exec src/cli/dmc_cli.exe -- set_init
Set identifier: dmc:AAAPRUJTY772WDVDZU7YTFJ6FQH34HURZFSDB6LQ6XPZ5M6TGYKJ4QJT7VBCOUVZHCV3H5ZRVCBPZSELIPXIPTLYJ54ZQ34PGZSWRZDQLQ
Secret key (do not loose or share!): urn:ed25519:sk:SMU6FSM6QTSGCPW2OZE5ACAG675MAJEY2GXPEAOCQ3SSCFYKS2ZQ
<triple
        <iri
            dmc:AAAPRUJTY772WDVDZU7YTFJ6FQH34HURZFSDB6LQ6XPZ5M6TGYKJ4QJT7VBCOUVZHCV3H5ZRVCBPZSELIPXIPTLYJ54ZQ34PGZSWRZDQLQ>
        <iri http://purl.org/dmc/ns#rootPublicKey>
        <iri
            urn:ed25519:pk:TVHJB7ABJXMOACNCBKSGAM3ZA2Y3ZVNAKX2S2CL7HJJ6Z27Z2CYA>>

<triple
        <iri
            dmc:AAAPRUJTY772WDVDZU7YTFJ6FQH34HURZFSDB6LQ6XPZ5M6TGYKJ4QJT7VBCOUVZHCV3H5ZRVCBPZSELIPXIPTLYJ54ZQ34PGZSWRZDQLQ>
        <iri http://www.w3.org/1999/02/22-rdf-syntax-ns#type>
        <iri http://purl.org/dmc/ns#Set>>
$ dune exec src/cli/dmc_cli.exe -- cbor_load -e raw -f cbor.dump -v
Reading from cbor.dump.
dmc_cli.exe: [INFO] [dmc] v fresh=false readonly=false log_size=500000
dmc_cli.exe: [INFO] Loading objects and blocks for container dmc:AAAD3RMUCOD3CH3RO6FQFJ6465ZDLXJCGGBZRWH4HFKM4YEGOOW5C3X56MMTEOE7VWCBFSGBVQQMYTTAQHMT6KTD7KYB3A3RNPP7JVUR4M
dmc_cli.exe: [INFO] tree commited (message: cbor_load)
$ dune exec src/cli/dmc_cli.exe -- authorized_keys_list  dmc:AAAPRUJTY772WDVDZU7YTFJ6FQH34HURZFSDB6LQ6XPZ5M6TGYKJ4QJT7VBCOUVZHCV3H5ZRVCBPZSELIPXIPTLYJ54ZQ34PGZSWRZDQLQ
urn:ed25519:pk:TVHJB7ABJXMOACNCBKSGAM3ZA2Y3ZVNAKX2S2CL7HJJ6Z27Z2CYA

Failing (mostly for lack of docs)

$ dune exec src/cli/dmc_cli.exe -- authorize_new urn:ed25519:pk:SIYYE2Q6QLEL4Y2VZYIMMWSMMN7NNJWRQDM6RRV4ZQAGNFZ4QGTA
dmc: internal error, uncaught exception:
     Dmc__Dmc_eris.Invalid_read_capability
$ dune exec src/cli/dmc_cli.exe -- authorize_new dmc:AAAPRUJTY772WDVDZU7YTFJ6FQH34HURZFSDB6LQ6XPZ5M6TGYKJ4QJT7VBCOUVZHCV3H5ZRVCBPZSELIPXIPTLYJ54ZQ34PGZSWRZDQLQ
dmc: required option --secret-key is missing
Usage: dmc authorize_new [OPTION]... CONTAINER_IDENTIFIER
Try `dmc authorize_new --help' or `dmc --help' for more information.

Resolving

The first time I tried authorize_new $KEY, but after reading the help I figured it would be the dmc: container set identifier. Still, the help did not mention --secret-key so I still am confused about this command. If I were to add @Nemael’s key to the container set, I’d certainly have to do --secret-key $MYKEY first… So I realized I cannot authorize an existing key, but instead a new one that’s created on the fly. Makes sense once you know it :slight_smile:

$  dune exec src/cli/dmc_cli.exe -- authorize_new --secret-key urn:ed25519:sk:SMU6FSM6QTSGCPW2OZE5ACAG675MAJEY2GXPEAOCQ3SSCFYKS2ZQ dmc:AAAPRUJTY772WDVDZU7YTFJ6FQH34HURZFSDB6LQ6XPZ5M6TGYKJ4QJT7VBCOUVZHCV3H5ZRVCBPZSELIPXIPTLYJ54ZQ34PGZSWRZDQLQ
Public key urn:ed25519:pk:EAD5G3RE42DRGEMVAK73XWEGHCLKVSWW4OOS75VWKPBJMQJWE7WA authorized for container dmc:AAAPRUJTY772WDVDZU7YTFJ6FQH34HURZFSDB6LQ6XPZ5M6TGYKJ4QJT7VBCOUVZHCV3H5ZRVCBPZSELIPXIPTLYJ54ZQ34PGZSWRZDQLQ

Secret key (do not loose or share!): urn:ed25519:sk:PCB2BUJZJ6K3MGNFE3YNMAJARQJ4REASJOHGVT6EWKIFDA2RYNZA

Still, if I need a secret key to authorize a new key, then the new secret key is known to me (so I can act as if I were the owner)… I guess this is something we figure out after alpha :slight_smile:

1 Like

Good point. Luckily this is just a shoddy implementation in the CLI. The dmc library allows authorizing a public key without knowing the secret key (AuthorizedKeys (dmc.Dmc.S.AuthorizedKeys)).

Currently the CLI can not take a public key as argument but just generates a new random key (src/cli/dmc_cli.ml · main · DREAM / DROMEDAR / ocaml-dmc · GitLab). Implementing this in the CLI is a good first step into the codebase ;)…

@arie just finished work on the RDF Turtle serialization and it is now in ocaml-dmc! :tada:

This makes the output much easier to read:

$ dune exec src/cli/dmc_cli.exe -- resolve dmc:AAAD3RMUCOD3CH3RO6FQFJ6465ZDLXJCGGBZRWH4HFKM4YEGOOW5C3X56MMTEOE7VWCBFSGBVQQMYTTAQHMT6KTD7KYB3A3RNPP7JVUR4M


<dmc:AAAD3RMUCOD3CH3RO6FQFJ6465ZDLXJCGGBZRWH4HFKM4YEGOOW5C3X56MMTEOE7VWCBFSGBVQQMYTTAQHMT6KTD7KYB3A3RNPP7JVUR4M>
 <http://purl.org/dmc/ns#member> <urn:erisx2:AAANBI56OPVF4DLM56RIGA33JHP4PQPXLCBR3SO2GB3L3OGTYMQMSGCAVZYC3RIWEOHJ2R57ALLIMB3Y6AB6ULZXNGPEYEYGAKMBAHEJMY>,
                                  <urn:erisx2:AAAPQ2OEC26URJJKOQ42TZG6H5S6WAGZITTSDAAS7V62OETFWNMZEEWYRC6FHEAMHLOSHDHITLUEWNRRLOPNY6AX4XD5NMDSWUM6UVLAKM> ;
  <http://purl.org/dmc/ns#rootPublicKey> <urn:ed25519:pk:SIYYE2Q6QLEL4Y2VZYIMMWSMMN7NNJWRQDM6RRV4ZQAGNFZ4QGTA> ;
  <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://purl.org/dmc/ns#Set> .

2 Likes

you call that easy to read? :laughing:

1 Like