Oh, that’s too bad. It’s annoying that it doesn’t give you more information.
On my computer it works, also after throwing everything away, and reinstalling.
What do you get when you type
opam switch
and do you perhaps have a guix installation as well? (That gave me some problems yesterday).
( how )% opam switch
# switch compiler description
-> 4.12.0 ocaml-base-compiler.4.12.0 4.12.0
default ocaml-system.4.05.0 default
I do have a Guix installation but am not using it really. What did pose you problem?
Well, either dune or opam where confused about which libraries to use, the ones that were installed in guix, or the ones that were installed in the regular opam directory.
I suppose it had something to do with environment variables like $OCAML_TOPLEVEL_PATH, but in the end I couldn’t get it to work and removed guix altogether (it uses quite some space as well).
But I got some messages related to this issue.
It might be that some packages (that are used by conduit-dream) have the wrong version numbers in your case. But to be fair I wouldn’t know how to find out.
yeah my bad.
if you’re comfortable with nix (i’m not really) @tg-x is maintaining the nix flake which in principle should circumvent these problems.
what is the output of opam list and opam pin? i’ll try to look later
I had to reinstall opam from a binary distribution since guix does not seem to install it for me, or not with the right ocaml version. I’m using ocaml v4.12.0.
-
opam list
# Packages matching: installed
# Name # Installed # Synopsis
alcotest 1.4.0 Alcotest is a lightweight and colourful test framework
angstrom 0.15.0 Parser combinators built for speed and memory-efficiency
angstrom-lwt-unix 0.15.0 Lwt_unix support for Angstrom
asn1-combinators 0.2.5 Embed typed ASN.1 grammars in OCaml
astring 0.8.5 Alternative String module for OCaml
async_kernel v0.14.0 Monadic concurrency library
async_rpc_kernel v0.14.0 Platform-independent core of Async RPC library
base v0.14.1 Full standard library replacement for OCaml
base-bigarray base
base-bytes base Bytes library distributed with the OCaml compiler
base-threads base
base-unix base
base64 3.5.0 Base64 encoding for OCaml
base_bigstring v0.14.0 String type based on [Bigarray], for use in I/O and C-bindings
base_quickcheck v0.14.1 Randomized testing framework, designed for compatibility with Base
batteries 3.3.0 A community-maintained standard library extension
bigarray-compat 1.0.0 Compatibility library to use Stdlib.Bigarray when possible
bigstringaf 0.7.0 Bigstring intrinsics and fast blits based on memcpy/memmove
bin_prot v0.14.0 A binary protocol generator
bos 0.2.0 Basic OS interaction for OCaml
ca-certs 0.2.1 Detect root CA certificates from the operating system
ca-certs-nss 3.66 X.509 trust anchors extracted from Mozilla's NSS
cbor 0.4 CBOR encoder/decoder (RFC 7049) - native OCaml implementation
cmdliner 1.0.4 Declarative definition of command line interfaces for OCaml
conduit 4.0.1 A network connection establishment library
conduit-dream 4.0.1-6 pinned to version 4.0.1-6 at git+file:///home/how/src/dream/ocaml-conduit-dream#master
conduit-lwt 4.0.1 A portable network connection establishment library using Lwt
conduit-lwt-dream 4.0.1-6 pinned to version 4.0.1-6 at git+file:///home/how/src/dream/ocaml-conduit-dream#master
conduit-lwt-unix 4.0.1 A network connection establishment library for Lwt_unix
conduit-mirage-dream 4.0.1-6 pinned to version 4.0.1-6 at git+file:///home/how/src/dream/ocaml-conduit-dream#master
conf-gmp 3 Virtual package relying on a GMP lib system installation
conf-gmp-powm-sec 3 Virtual package relying on a GMP lib with constant-time modular exponentiation
conf-libffi 2.0.0 Virtual package relying on libffi system installation
conf-libssl 3 Virtual package relying on an OpenSSL library system installation
conf-pkg-config 2 Check if pkg-config is installed and create an opam switch local pkgconfig folder
conf-which 1 Virtual package relying on which
core v0.14.1 Industrial strength alternative to OCaml's standard library
core_kernel v0.14.2 Industrial strength alternative to OCaml's standard library
cppo 1.6.7 Code preprocessor like cpp for OCaml
csexp 1.5.1 Parsing and printing of S-expressions in Canonical form
cstruct 6.0.0 Access C-like structures directly from OCaml
cstruct-sexp 6.0.0 S-expression serialisers for C-like structures
ctypes 0.18.0 Combinators for binding to C libraries without writing any C
ctypes-foreign 0.18.0 Virtual package for enabling the ctypes.foreign subpackage.
decoders 0.6.0 Elm-inspired decoders for Ocaml
decoders-cbor 0.6.0 CBOR backend for decoders
dns 5.0.1 An opinionated Domain Name System (DNS) library
dns-client 5.0.1 Pure DNS resolver API
domain-name 0.3.0 RFC 1035 Internet domain names
dune 2.8.5 Fast, portable, and opinionated build system
dune-configurator 2.8.5 Helper library for gathering system configuration
duration 0.1.3 Conversions to various time units
eqaf 0.7 Constant-time equal function on string
fieldslib v0.14.0 Syntax extension to define first class values representing record fields, to get and set record fields, iterate and fold over all fields of a record and create new record values
fmt 0.8.9 OCaml Format pretty-printer combinators
fpath 0.7.3 File system paths for OCaml
gmap 0.3.0 Heterogenous maps over a GADT
hacl-star 0.4.1 OCaml API for EverCrypt/HACL*
hacl-star-raw 0.4.1 Auto-generated low-level OCaml bindings for EverCrypt/HACL*
hex 1.4.0 Library providing hexadecimal converters
hkdf 1.0.4 HMAC-based Extract-and-Expand Key Derivation Function (RFC 5869)
integers 0.4.0 Various signed and unsigned integer types for OCaml
io-page 2.4.0 Support for efficient handling of I/O memory pages
ipaddr 5.0.1 A library for manipulation of IP (and MAC) address representations
ipaddr-sexp 5.0.1 A library for manipulation of IP address representations usnig sexp
jane-street-headers v0.14.0 Jane Street C header files
jst-config v0.14.1 Compile-time configuration for Jane Street libraries
logs 0.7.0 Logging infrastructure for OCaml
lru 0.3.0 Scalable LRU caches
lwt 5.4.0 Promises and event-driven I/O
lwt_ppx 2.0.2 PPX syntax for Lwt, providing something similar to async/await from JavaScript
macaddr 5.0.1 A library for manipulation of MAC address representations
metrics 0.3.0 Metrics infrastructure for OCaml
mirage-clock 3.1.0 Libraries and module types for portable clocks
mirage-crypto 0.10.3 Simple symmetric cryptography for the modern age
mirage-crypto-ec 0.10.3 Elliptic Curve Cryptography with primitives taken from Fiat
mirage-crypto-pk 0.10.3 Simple public-key cryptography for the modern age
mirage-crypto-rng 0.10.3 A cryptographically secure PRNG
mirage-device 2.0.0 Abstract devices for MirageOS
mirage-flow 2.0.1 Flow implementations and combinators for MirageOS
mirage-flow-combinators 2.0.1 Flow implementations and combinators for MirageOS specialized to lwt
mirage-kv 3.0.1 MirageOS signatures for key/value devices
mirage-no-solo5 1 Virtual package conflicting with mirage-solo5
mirage-no-xen 1 Virtual package conflicting with mirage-xen
mirage-protocols 5.0.0 MirageOS signatures for network protocols
mirage-random 2.0.0 Random-related devices for MirageOS
mirage-stack 2.2.0 MirageOS signatures for network stacks
mirage-time 2.0.1 Time operations for MirageOS
mmap 1.1.0 File mapping functionality
mtime 1.2.0 Monotonic wall-clock time for OCaml
num 1.4 The legacy Num library for arbitrary-precision integer and rational arithmetic
ocaml 4.12.0 The OCaml compiler (virtual package)
ocaml-base-compiler 4.12.0 Official release 4.12.0
ocaml-compiler-libs v0.12.3 OCaml compiler libraries repackaged
ocaml-config 2 OCaml Switch Configuration
ocaml-migrate-parsetree 2.1.0 Convert OCaml parsetrees between different versions
ocaml-options-vanilla 1 Ensure that OCaml is compiled with no special options enabled
ocaml-syntax-shims 1.0.0 Backport new syntax to older OCaml versions
ocamlbuild 0.14.0 OCamlbuild is a build system with builtin rules to easily build most OCaml projects.
ocamlfind 1.9.1 A library manager for OCaml
ocplib-endian 1.1 Optimised functions to read and write int16/32/64 from strings and bigarrays
octavius 1.2.2 Ocamldoc comment syntax parser
parsexp v0.14.0 S-expression parsing library
pbkdf 1.1.0 Password based key derivation functions (PBKDF) from PKCS#5
ppx_assert v0.14.0 Assert-like extension nodes that raise useful errors on failure
ppx_base v0.14.0 Base set of ppx rewriters
ppx_bench v0.14.1 Syntax extension for writing in-line benchmarks in ocaml code
ppx_bin_prot v0.14.0 Generation of bin_prot readers and writers from types
ppx_cold v0.14.0 Expands [@cold] into [@inline never][@specialise never][@local never]
ppx_compare v0.14.0 Generation of comparison functions from types
ppx_cstruct 6.0.0 Access C-like structures directly from OCaml
ppx_custom_printf v0.14.1 Printf-style format-strings for user-defined string conversion
ppx_derivers 1.2.1 Shared [@@deriving] plugin registry
ppx_deriving 5.2.1 Type-driven code generation for OCaml
ppx_deriving_yaml 0.1.0 Yaml PPX Deriver
ppx_enumerate v0.14.0 Generate a list containing all values of a finite type
ppx_expect v0.14.1 Cram like framework for OCaml
ppx_fields_conv v0.14.2 Generation of accessor and iteration functions for ocaml records
ppx_fixed_literal v0.14.0 Simpler notation for fixed point literals
ppx_hash v0.14.0 A ppx rewriter that generates hash functions from type expressions and definitions
ppx_here v0.14.0 Expands [%here] into its location
ppx_inline_test v0.14.1 Syntax extension for writing in-line tests in ocaml code
ppx_jane v0.14.0 Standard Jane Street ppx rewriters
ppx_js_style v0.14.1 Code style checker for Jane Street Packages
ppx_let v0.14.0 Monadic let-bindings
ppx_module_timer v0.14.0 Ppx rewriter that records top-level module startup times
ppx_optcomp v0.14.3 Optional compilation for OCaml
ppx_optional v0.14.0 Pattern matching on flat options
ppx_pipebang v0.14.0 A ppx rewriter that inlines reverse application operators `|>` and `|!`
ppx_sexp_conv v0.14.3 [@@deriving] plugin to generate S-expression conversion functions
ppx_sexp_message v0.14.1 A ppx rewriter for easy construction of s-expressions
ppx_sexp_value v0.14.0 A ppx rewriter that simplifies building s-expressions from ocaml values
ppx_stable v0.14.1 Stable types conversions generator
ppx_string v0.14.1 Ppx extension for string interpolation
ppx_typerep_conv v0.14.2 Generation of runtime types from type declarations
ppx_variants_conv v0.14.1 Generation of accessor and iteration functions for ocaml variant types
ppxlib 0.22.0 Standard library for ppx rewriters
protocol_version_header v0.14.0 Protocol versioning
psq 0.2.0 Functional Priority Search Queues
ptime 0.8.5 POSIX time for OCaml
randomconv 0.1.3 Convert from random byte vectors (Cstruct.t) to random native numbers
re 1.9.0 RE is a regular expression library for OCaml
result 1.5 Compatibility Result module
rresult 0.6.0 Result value combinators for OCaml
seaboar 0.6.8 pinned to version 0.6.8 at git+file:///home/how/src/dream/ocaml-seaboar#master
seq base Compatibility package for OCaml's standard iterator type starting from 4.07.
sexplib v0.14.0 Library for serializing OCaml values to and from S-expressions
sexplib0 v0.14.0 Library containing the definition of S-expressions and some base converters
spawn v0.14.0 Spawning sub-processes
splittable_random v0.14.0 PRNG that can be split into independent streams
ssl 0.5.10 Bindings for OpenSSL
stdint 0.7.0 Signed and unsigned integer types having specified widths
stdio v0.14.0 Standard IO library for OCaml
stdlib-shims 0.3.0 Backport some of the new stdlib features to older compiler
stringext 1.6.0 Extra string functions for OCaml
textutils v0.14.0 Text output utilities
time_now v0.14.0 Reports the current time
timezone v0.14.0 Time-zone handling
tls 0.14.0 Transport Layer Security purely in OCaml
tls-mirage 0.14.0 Transport Layer Security purely in OCaml, MirageOS layer
topkg 1.0.3 The transitory OCaml software packager
typerep v0.14.0 Typerep is a library for runtime types
uchar 0.0.2 Compatibility library for OCaml's Uchar module
uri 4.1.0 An RFC3986 URI/URL parsing library
uuidm 0.9.7 Universally unique identifiers (UUIDs) for OCaml
uutf 1.0.2 Non-blocking streaming Unicode codec for OCaml
variantslib v0.14.0 Part of Jane Street's Core library
vchan 6.0.0 Xen Vchan implementation
x509 0.14.1 Public Key Infrastructure (RFC 5280, PKCS) purely in OCaml
xenstore 2.1.1 Xenstore protocol in pure OCaml
xenstore_transport 1.3.0 Low-level libraries for connecting to a xenstore service on a xen host
yaml 2.1.0 Parse and generate YAML 1.1 files
zarith 1.12 Implements arithmetic and logical operations over arbitrary-precision integers
-
opam pin
conduit-async-dream.4.0.1-6 (uninstalled) git git+file:///home/how/src/dream/ocaml-conduit-dream#master
conduit-dream.4.0.1-6 git git+file:///home/how/src/dream/ocaml-conduit-dream#master
conduit-lwt-dream.4.0.1-6 git git+file:///home/how/src/dream/ocaml-conduit-dream#master
conduit-lwt-unix-dream.4.0.1-6 (uninstalled) git git+file:///home/how/src/dream/ocaml-conduit-dream#master
conduit-mirage-dream.4.0.1-6 git git+file:///home/how/src/dream/ocaml-conduit-dream#master
seaboar.0.6.8 git git+file:///home/how/src/dream/ocaml-seaboar#master
- installation instructions work
(will update this post as info comes)
Looks like the pinning is wrong.
#=== ERROR while compiling conduit-lwt-unix-dream.4.0.1-6 =====================#
# context 2.1.0 | linux/x86_64 | ocaml-base-compiler.4.12.0 | pinned(git+file:///home/how/src/dream/ocaml-conduit-dream#master#cc40f1a9)
# path ~/.opam/4.12.0/.opam-switch/build/conduit-lwt-unix-dream.4.0.1-6
# command ~/.opam/opam-init/hooks/sandbox.sh build dune build -p conduit-lwt-unix-dream -j 3
# exit-code 1
# env-file ~/.opam/log/conduit-lwt-unix-dream-257239-f015b8.env
# output-file ~/.opam/log/conduit-lwt-unix-dream-257239-f015b8.out
### output ###
# 296 | >>= fun (fd, ic, oc, info) ->
# [...]
# Error: This pattern matches values of type 'a * 'b * 'c * 'd
# but a pattern was expected which matches values of type
# Lwt_unix.file_descr * Lwt_io.input_channel * Lwt_io.output_channel
# ocamlopt src/conduit-lwt-unix/.conduit_lwt_unix.objs/native/conduit_lwt_unix.{cmx,o} (exit 2)
# (cd _build/default && /home/how/.opam/4.12.0/bin/ocamlopt.opt -w -40 -g -I src/conduit-lwt-unix/.conduit_lwt_unix.objs/byte -I src/conduit-lwt-unix/.conduit_lwt_unix.objs/native -I /home/how/.opam/4.12.0/lib/angstrom -I /home/how/.opam/4.12.0/lib/asn1-combinators -I /home/how/.opam/4.12.0/lib/astring -I /home/how/.opam/4.12.0/lib/base/caml -I /home/how/.opam/4.12.0/lib/base64 -I /home/how/.op[...]
# File "src/conduit-lwt-unix/conduit_lwt_unix.ml", line 296, characters 10-28:
# 296 | >>= fun (fd, ic, oc, info) ->
# ^^^^^^^^^^^^^^^^^^
# Error: This pattern matches values of type 'a * 'b * 'c * 'd
# but a pattern was expected which matches values of type
# Lwt_unix.file_descr * Lwt_io.input_channel * Lwt_io.output_channel
<><> Error report <><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
+- The following actions failed
| - build conduit-lwt-unix-dream 4.0.1-6
+-
+- The following changes have been performed
| - recompile ctypes 0.18.0
| - recompile ppx_deriving_yaml 0.1.0
| - recompile seaboar 0.6.8
| - recompile stdint 0.7.0
| - recompile yaml 2.1.0
| - install conf-libffi 2.0.0
| - install conf-which 1
| - install ctypes-foreign 0.18.0
| - install hacl-star 0.4.1
| - install hacl-star-raw 0.4.1
+-
The former state can be restored with:
/home/how/bin/opam switch import "/home/how/.opam/4.12.0/.opam-switch/backup/state-20210915111905.export"
Yesterday I fixed my Ocaml install and was able to install and run the Message Router. In the afternoon, @arie, @misterfish, @natacha and I convened for a demo of the MR so far. It was very good to reconnect and see some awesome progress from Alleycat. We managed to find a bug – as always, they come to me – but it seems to be a minor one and will be fixed by the end of this month. We decided on a date for delivery of D2.2: September 29th.
Here is a copy of the pad where we took notes:
2021-09-15 D2.2
due date D2.2
state of software
what does it do exactly (in layman’s terms)
what’s next?
seems like messages are expiring when the service reconnects
which subset of spec and why, so developers know how to contribute or get interested
readme, how to set up & run, scenarios
@how: try running on different IPs
So I’ll be setting up a couple or three machines each running one message router from the demo software (MR A, B and C) so we can test across the network. @dvn would it be possible to use rhyzome to deploy these machines? My plan is to make something like a minimalist Alpine Linux setup with OCaml and the DREAM software, snapshot it, and reproduce it twice (so we have three hosts). But it would be much more efficient to have a single declarative repository with all configurations so we can easily setup/destroy the hosts on-demand. This would also help me go through rhyzome and entanglement.garden software to get started with it – I’d like in the near future to turn all my infrastructure into a declarative state, and using rhyzome makes sense to me to support DREAM software in the wild.
1 Like
I’m currently setting up some virtual machines with Qemu to test a setup using Alpine Linux.
The idea is to create a minimal snapshot that I can deploy to the Hetzner cloud, eventually from Gitlab CI using rhyzome. We’re not there yet, but that could be an interesting development for the demo.
In this testing phase, my goal is to figure out the best way to build the message router using Alpine so I can deploy it to 3 VMs on the Hetzner Cloud, and have them talk to each other from different hosts.
@misterfish, I know the current interface requires pressing keys to launch pre-defined sequences, such as sending messages to the queue. Is there a way to do this programmatically without keyboard interaction? Then we could have the machines talking to each other without human intervention. Maybe using a kill USR1 signal or something? I presume there’s no actual sending to port from external programs for now since it would require crafting the correct messages, or can we already?
Cool. Curious to see how far you get.
It would require some work, but we could have the MR’s, or services, automatically send a couple of messages every n seconds, using Lwt from Ocaml. Perhaps that could also be a start for testing the system (although most likely we won’t have time to implement those before the end of September)
Not sure what you mean with your last sentence. @misterfish always has a free Tuesday, so you’re not lucky with that.
Thank you @arie. I meant that, since the MR is listening, it could accept any validly crafted message. But I’m not sure how or whether we can do those from outside the current MR code itself.
Well, when you start a service, that is an external program (from the point of view of a message router).
Of course, right now, only message routers and services can send valid messages (since the encoding functions are indeed in those programs). You could put an encoding function in another program, let it make a connection and send a message, but would be exactly the same as what a service does.
What does ‘external’ mean? Not written by us?
No, it means what you said about the service. Actually some CLI where you can pipe in some random message would be perfect. Then we can just send “Hello, world.” and it goes through. That’s perfect.
@Nemael could you look into this code and extract the encoding part so we can have just that? I can imagine something like this: cat file.txt | upsycle-send mr-a.dream.public.cat that writes out the response from the router.
All right. The thing is, is that we already have services; although they do not work
exactly as you describe (you cannot give a string or a file.txt as argument for a message),
but they send random messages (some of the arguments are randomly generated) to MR’s after key-strokes.
Tweaking this service to do what you want (i.e. be able to give a file.txt as argument) would be way easier than to write a new service, since the whole ‘establishing a connection’, and 'make a valid message from a string" part is already done there.
1 Like
Having a simple way to demonstrate a “live message” among random ones would offer a great confirmation for the sceptical mind. Another step would be for the destination service to be able to print it out in the log once it received it, or somehow display the content of this specific message, I guess that would require a specific destination service that not only count messages but decodes them.
Sure, I agree with that. The printing step we already have. It is printed on the screen right now; you should be able to find it once you have an MR and a service running (e.g. MR a and Svc a1), and send a message from the service to the MR (use h for a help message to see how to do this).
I guess it would be good to do a git pull and opam install --deps-only --with-test . in the upsycle-router directory, to be sure that h works.
The body of the message can be seen in the screen of the MR.
@how if I understand correctly you want to simulate what the keyboard does, without needing a keyboard, so you can demonstrate different scenarios. Yes we can do this. I think a TCP socket would work better than signals, also because it needs to work between machines and not just on the same machine.
So your ‘file.txt’ consists of, say:
# --- contents of file.txt:
n
n
n
and then you want to do
cat file.txt | upsycle-send mr-a.dream.public.cat
and it will have the same effect as if you had pressed ‘n n n’ on the keyboard. Is this correct?
Great that you made a fix quickly! Thank you.
Yes, that’s a first step, and the description you make is correct. Let’s go for it.
BTW, once we have this in place, what’s the catch of having the services talk to each other on their own?
Could you clarify?
See the new configuration field ‘control-interface’ in the message router yaml files. If it’s lets’s say 7001, then you can do:
echo 'n' | nc localhost 7001
which will have the same effect as pressing n.
1 Like