ERIS implementation

Hey everybody!
The ERIS library is now usable :partying_face:
It is available on the ocaml-eris repository
I have a few small touch-ups that I would like to do, mainly:

  • I would like to make ERIS output json that would fit pukkamustard’s test vectors (instead of the mess of variables the ouput is right now), which then could be fed back into ERIS using two new encode_to_json and decode_to_json functions.
  • I would like to restrict Eris’ use a bit (to limit the choice of block size, for example)
  • I would like to clean the functions up, to improve readability (especially the decode and encode functions
  • I would like to improve the documentation. Documentation is not yet complete or pushed and I would like to work on it more in the coming days (this is the most important part, I mainly have to put together a bunch of notes that I took while working on ERIS)
  • I would like to improve the automatic testing, maybe learn to use Alcotest

With a bit more tasks of the same scale, it shouldn’t take a long time.
I’ll keep this post updated in the next few days on what has been done.
I am pretty happy to have (almost) reached the end of this part, and feel more confident toward my ocaml skills.
The “ERIS Quickstart Guide” is a bit of a mess, but it contains information on how to build and run ERIS. The information in this file will be moved to the README.md file.
I don’t know much what else to say, please check it out on its repository!

3 Likes

Congratulations @nemael :cowboy_hat_face: :muscle: :fireworks:

I added encode_to_json function that ouput a json string containing the encoded information according to the test vectors.

1 Like

Nemael via DREAM noreply@dream.public.cat writes:

[…]

The ERIS library is now usable :partying_face:

Fantastic! :tada:

Some feedack:

ocp-indent

Could you please use ocp-indent to format your OCaml code? This
ensures that we are all used an established standard way of
formatting and prevent git noise due to indentation or other
formatting issues.

I have also asked @ablom in private communication to use
ocp-indent. Maybe we can all (cc @dream) agree to use ocp-indent.

Testing

  • I would lie to improve the automatic testing, maybe learn to
    use Alcotest

:+1:

Could you move the bin/main.ml to test/main.ml and make it run
on all test vectors automatically? Preferably using Alcotest.

Interface

  • I would like to clean the functions up, to improve readability
    (especially the decode and encode functions
  • I would like to improve the documentation. Documentation is
    not yet complete or pushed and I would like to work on it more
    in the coming days (this is the most important part, I

:+1:

Suggestion:

module ReadCapability : sig
  type t

  val to_string : t -> string
  val of_string : string -> (t, string) result
end

type blocks = (string * string) list

val encode : string -> ?convergence_secret:string -> 
?block_size:int -> blocks * ReadCapability.t
val decode : ReadCapability.t -> blocks -> string

Also move the encode_to_json to the tests. That way installing the
ERIS library does not have a depenendency on Yojson.

1 Like

:+1:

ocp-indent is easy to use with Emacs :slight_smile: and other editors.

echo '(load-file "'"$(opam config var prefix)"'/share/typerex/ocp-indent/ocp-indent.el")' >>~/.emacs

Thank you for your feedback!

I moved the encode_to_json function to tests, and moved the dependencies around. Now Eris itself only requires Ocaml-Monocypher and Ocaml-Base32.

Does it really makes sense to put the encode_to_json function into the test.ml file? Because, as I understand it, the encode_to_json function will be used to store Eris data in the long run, and I think that it makes sense to keep it within Eris (despite adding a library as requirement) Another way would be to put this function in a new folder/file, which would contain (about) only this function. Would it make sense?

I also added automatic testing with Alcotest, which run without fail, that’s very cool!

For the tests, my first approach was to compare the whole json produced with the json stored in test-vectors, but it was a problem because the blocks were not always in the same order (they were always accounted for, but not in the same order, which caused problem with the json string comparison). So I changed it a bit.

  • Encode tests takes content, convergence-secret and block-size from the json string, encode the content and compare the produced urn to the urn stored in the json file.
  • Decode tests takes blocks, block-size, level, root-reference, root-key from the json string, decode the content and compares it to the content stored in the json file.

I also have a question with ocp-indent. I ran it in command-line, and it worked well, but it uses an indent space of 2. 2 is very tiny for me and it feels cramped. Would it be ok for me to change the ocp-indent settings to use an indent space of 4 (I think it is doable in the settings), or is it important also to keep such a small indent space?

I have not yet added the types you are suggesting, but I will take the time to implement them to improve readability.

Here is a patch to make the flake.nix work:
0001-Add-base32-and-monocypher-to-flake.nix.patch (3.2 KB)

(ignore the .pdf extension, thats just to defeat upload rules)

Thank you!
That flake was a relic that I should have removed a while ago, but it’s great to have it working with your patch :slight_smile:

Nemael via DREAM noreply@dream.public.cat writes:

I moved the encode_to_json function to tests, and moved the
dependencies around. Now Eris itself only requires
Ocaml-Monocypher and Ocaml-Base32.

Nice.

Does it really makes sense to put the encode_to_json function
into the test.ml file? Because, as I understand it, the
encode_to_json function will be used to store Eris data in the
long run, and I think that it makes sense to keep it within Eris
(despite adding a library as requirement) Another way would be
to put this function in a new folder/file, which would contain
(about) only this function. Would it make sense?

I don’t think so. The JSON format that is implemented in
encode_to_json is only meant for the test-cases. I don’t think
it should be used for real applications as there are much better
ways of storing necessary information.

I also added automatic testing with Alcotest, which run without
fail, that’s very cool!

Nice! Could you add following snippet to test/dune:

(rule
 (alias runtest)
 (action (run ./test.exe)))

This will allow you to run dune runtest (or dune test) in the
repo root. However there seems to be an issue that the
test-vectors are not available when running with dune runtest.
Maybe they need more rules in the dune file to install them in the
proper location? Could you try and get that running?

For the tests, my first approach was to compare the whole json
produced with the json stored in test-vectors, but it was a
problem because the blocks were not always in the same order
(they were always accounted for, but not in the same order,
which caused problem with the json string comparison). So I
changed it a bit.

  • Encode tests takes content, convergence-secret and block-size
    from the json string, encode the content and compare the
    produced urn to the urn stored in the json file.
  • Decode tests takes blocks, block-size, level, root-reference,
    root-key from the json string, decode the content and compares
    it to the content stored in the json file.

:+1:

I also have a question with ocp-indent. I ran it in
command-line, and it worked well, but it uses an indent space of
2. 2 is very tiny for me and it feels cramped. Would it be ok
for me to change the ocp-indent settings to use an indent space
of 4 (I think it is doable in the settings), or is it important
also to keep such a small indent space?

:slight_smile: I’d prefer sticking with the defaults. They encapsulate years
of discussion on space vs. tab and 4 spaces vs 2 spaces…

Hey, nice work, Nemael!
I have a question though; I tried

guix environment -l guix.scm

which seems to be ok. After that, I try

dune build

which gives an error:

File "_none_", line 1:
Error: The file /home/arie/.opam/default/lib/biniou/biniou.cmxa is not a compilation unit description

Any ideas on how to fix this? I don’t know that much about guix. Trying to install the missing libs via opam is not trivial, because you need the libs that are written by pukkamustard (I think), and it might work, but takes some time.

Can you try:

dune clean; guix environment -l guix.scm --pure -- dune build

Suggestion for next step:

Implement an erissum utility (as also implemented by @emery https://dream.public.cat/t/rfc-eris-0-2-0/93/41?u=pukkamustard) in ocaml-dmc that can be used as following:

dmc erissum my-file.txt
erisx2:AAAD77QDJMFAKZYH2DXBUZYAP3MXZ3DJZVFYQ5DFWC6T65WSFCU5S2IT4YZGJ7AC4SYQMP2DM2ANS2ZTCP3DJJIRV733CRAAHOSWIYZM3M my-fille.txt

This requires implementing the dmc command line interface using Cmdliner and using the ERIS implementation on real files.

Ok, I tried this. It gives

File "/gnu/store/w50w5k8iri09y7cbdanbgcy5q16irn8b-profile/lib/ocaml/site-lib/yojson/META", line 1, characters 0-0:
Error: Library "easy-format" not found.
-> required by library "yojson" in
   /gnu/store/w50w5k8iri09y7cbdanbgcy5q16irn8b-profile/lib/ocaml/site-lib/yojson
-> required by executable test in test/dune:2
Hint: try:
  dune external-lib-deps --missing @@default

Maybe it needs a dependency in yojson?

Hm, ocaml-easy-format is a propagated-input to ocaml-yojson and should be in the environment if you have ocaml-yojson - which is the case because ocaml-yojson is listed in the guix.scm file.

Maybe you are running a old version of Guix (the patch to fix the dependency of yojson on easy-format was only merged in December).

Could you try running:

guix time-machine
    --url=https://gitlab.com/pukkamustard/guix.git \
    --branch=assorted-ocaml-patches \
    --disable-authentication \
    -- environment -l guix.scm --pure -- dune build

Which exactly pins the version of OCaml package definitions to a version that should work.

Also maybe try running guix pull and the retry the command above.

1 Like

Hey, thanks for this. It works now. I installed guix with apt (on debian testing), maybe that’s why it was using an older version? Anyway, guix pull did the trick. Then guix environment -l guix.scm is sufficient.