Hello Stratus - toying around with Nimbus and QML

experiment
nimbus

#1

Intro

Felt like a good time to take a breather from Eth2 for a day, so I started looking at some of the Whisper code that recently was contributed it to Nimbus - it’s a first cut so I thought it would be fun to take it for a spin - and it works surprisingly well!

Another thing I’d been meaning to do was to try out QT - never really gotten around to write an app in it, and given how they added QtQuick and claimed it was good for rapid prototyping, it seemed like a good match for the day.

Thus, without further ado, I’d like to present to you the outcome of the experiment: Stratus - one more desktop chat application (well, a viewer at least!) for the Status Network :astonished: - obviously borne out of jealousy that @iurimatias & co have their own Status protocol implementation…

Technology

The stack is thankfully fairly simple: Nim for networking and logic with a C++-based UI framework that is customized through QML and JavaScript.

P2P layer

At the bottom we have nim-eth-p2p which handles the Ethereum devp2p protocol - serialization, encryption, discovery, multiplexing - that sort of stuff.

Incidentally, this is the part that many want to replace with libp2p eventually - though for our first beacon node demo in Nimbus, we’ll likely keep using it simply because the code is written already.

It’s generally a good thing that it’s getting replaced - for us it’s causing some trouble, such as the non-standard port that prevents our chat from being used on many public wifi’s, or the encryption that’s not really doing its job.

Whisper layer

nim-eth-p2p is also where the core Whisper implementation lives in Nimbus - raw Whisper packets are decoded and matched against available decryption keys, bloom filters are formed to shape bandwidth consumption and the database of live messages buzzing around in the system is kept here.

For example when you send a whisper message, the nim-eth-p2p code takes a structure like the following:

  Envelope* = object
    ## What goes on the wire in the whisper protocol - a payload and some
    ## book-keeping
    ## Don't touch field order, there's lots of macro magic that depends on it
    expiry*: uint32 ## Unix timestamp when message expires
    ttl*: uint32 ## Time-to-live, seconds - message was created at (expiry - ttl)
    topic*: Topic
    data*: Bytes ## Payload, as given by user
    nonce*: uint64 ## Nonce used for proof-of-work calculation

and turns it into RLP-encoded bytes for further processing.

Status layer

Ok - we arrive at the first half of the thing that was hacked together this time, with emphasis on hacked. Once a Whisper message arrives at the application layer, it has already been decrypted, the key of the sender is available and now it’s time to turn raw bytes into chat bubbles.

Status encodes messages using json in a peculiar shape, created by a particular library - I can’t really think of anything positive to write about it, so I’ll just leave a link.

  try:
    # Sample message:
    # ["~#c4",["dcasdc","text/plain","~:public-group-user-message",
    #          154604971756901,1546049717568,[
    #             "^ ","~:chat-id","nimbus-test","~:text","dcasdc"]]]
    let
      src =
        if msg.decoded.src.isSome(): $msg.decoded.src.get()
        else: ""
      payload = cast[string](msg.decoded.payload)
      data = parseJson(cast[string](msg.decoded.payload))
      channel = data.elems[1].elems[5].elems[2].str
      time = $fromUnix(data.elems[1].elems[4].num div 1000)
      message = data.elems[1].elems[0].str

    info "adding", full=(cast[string](msg.decoded.payload))
    rootItem.add(channel, src[0..<8] & "..." & src[^8..^1], message, time)
  except:
    notice "no luck parsing", message=getCurrentExceptionMsg()

The message arrives and you have a nested list of heterogeneous items in it. You’ll immediately notice the shoddy hackathon-style code there - all based on the honorable practice of reverse engineering instead of reading documentation. I’m not sure which of those two time-looking numbers are actually the message time - close enough for now though :slight_smile:. Status once had a funny bug where messages would be sorted by this field - meaning that anyone could craft a message that would get stuck at the beginning of the list and never scroll away. This code is no better - breaks a cardinal rule of internet applications: you can’t trust what the remote end is sending you… That cast looks fishy and going by index just like that… well… and so it goes on.

UI Layer

The second part of the hack, the UI, is done using the QtQuick framework. Really nice! The idea is that you write a mostly declarative UI description, annotate it with some JavaScript to handle visuals like animation and shuffling of data from widget to model, and then keep the application logic apart.

My initial plan for the day was to write the data model in C++, that being the native language for QT, and then take the Nim-C interop features out for an exercise to bring in the rest of Nimbus. Luck would have it however that someone’s written a QML wrapper for Nim, and I could stay in Nim-land throughout. Unusually, it even came with documentation and examples!

Let’s take a look at a snippet from the QML:

ListView {
    id: messagesView
    model: root.messageList
    // model: sampleMessages
    Layout.fillWidth: true
    Layout.fillHeight: true
    delegate: RowLayout {
        width: messagesView.width
        Text { text: "👾"; font.pointSize: 28 }

        ColumnLayout {
            Layout.fillWidth: true
            RowLayout {
                Text { text: channel; font.bold: true }
                Text { text: " ("; }
                Text { text: source; }
                Text { text: ")"; }
                Item { Layout.fillWidth: true }
                Text { text: time; }
            }

            Text { Layout.fillWidth: true; text: message; wrapMode: Text.WordWrap }
        }
    }
}

Above we can see the bit that creates the layout for the rows of the message view, including the cute alien which in discuss looks more like a space invader.

First with have the List itself, and it’s annotated with parameters informing the resizable layout engine. You can also see the model which is how content is pulled in. Finally, the delegate is a template for what each row should look like.

To get the message content to show up, in Nim we fill out an object that has the right fields, and stuff it in the messageList:

  proc add*(
      self: MessageList, channel: string, source: string, message: string,
      time: string) {.slot.} =
    let sm = newStatusMessage(channel, source, message, time)

    self.beginInsertRows(newQModelIndex(), self.messages.len, self.messages.len)
    self.messages.add(sm)
    self.endInsertRows()

There’s a bit of book-keeping going on there as well, to let the UI know it has to update the visuals for the new data.

In general, the QML approach has two killer features:

  • you can churn something that looks decent in no time
  • designers can work on the UX independently - the layout can done in a visual designer tool and is separate from the code

… and then the third half

Of course, like any software development story there’s a snag somewhere around the time when you actually have to build the thing, and this time it was no different. As commonly known, build system complexity grows exponentially with the number of frameworks and languages you throw into the mix…

With the app mostly done, my first thought was to just statically link Qt and not have to bother with versions. Turns out that Qt is distributed as a collection of dynamic libraries and comes with a custom build system on top. Plan B suddenly became much more appealing: create an AppImage - should be simple, just compile and zip it up that’s it. Haha.

I’m generally not a fan of Docker - it’s one of those smells that says that something else has become too complex to manage and now the least bad solution is to throw another piece of complexity at it - but whatever, it’s hack day (more night now) and we want results. Fortunately, I’m not the only one that has had this need and conveniently, there exists a docker image that has a dated Ubuntu with a fresh Qt on it (that’s the right mix for AppImages!). As a cherry on top, it comes with instructions how to use it to deploy a QT app on Android.

It’s ugly, and it works. Long story short, the repo now holds the magic bits to create a mostly reproducible AppImage build (some Nim parts still would need locking down - that’s a work in progress in the community).

That’s it

Wanna give it a try? Here’s what works so far:

  • Connect to the status whisper nodes and listen for messages
  • Subscribe to channels and look at messages
  • Pure nimbus! no geth involved :partying_face:!
  • Only tested on Linux - in theory it should work elsewhere as well with the right magic incantations - even Android and iOS - though you know what they say about theory and practice…

From here, it should be fairly easy to add any of the following:

  • Send messages
  • View other whisper traffic - think wireshark, but for whisper - can’t do much with it but interesting for exploring the protocol
  • work on UI - don’t even have to know how to code!

:warning: Yeah, the code is untested and all that, and I know there are some nasty crashes and races etc :warning:

You can get the AppImage from Swarm (for that warm, fuzzy, distributed feeling) - might need reseeding occasionally though, just let me know.

Download: Stratus-20181229-0.0.0.0.0.0.1-x86_64.AppImage
Code: https://github.com/status-im/nim-eth-p2p/tree/stratus/examples/stratus (note the branch)

I’ll be happy to walk you through it as well, should you want to get your hands dirty - it’s turned out quite educational :slight_smile:


#2

Thanks for the post, is nice to see that nimbus is making progress and good effort!

Disclaimer, I haven’t picked the library myself, nor I would pick it if I had to make the choice, but I understand (while disagree) with the choice.

I believe JSON was what was used before and I understand there was much focus in reducing bandwidth consumption. So knowing that, reading the rationale https://github.com/cognitect/transit-clj, and understanding the people behind it https://cognitect.com/ (aka the clojure gang), it’s easy to see at least a few benefits:

  1. Less bandwidth consumption, while maintaining a fairly human-readable format

  2. Encourages maintaining backward compatibility, something that I know is very dear to both you and Rich Hickey ( I would not be surprised if you both had this slide hanging as a poster on your wall https://youtu.be/oyLBGkS5ICk?t=2492 :wink: ) , by basically making accretion easy (append an element to an array), and breaking changes hard, although possible (you are not supposed to modify any field, they ought to be considered immutable, schema-wise, append only), if a breaking change is needed a new name (i.e. tag) needs to be created

  3. formalized way to add custom types on top of it

  4. Easy to parse (any JSON parser will do) and fairly large multi-language support

The issues we had with the data format I think were mainly due to us not being very careful with the way we changed the protocol (we modified fields, instead of adding new fields, and message-id language dependent, but that was/is out-of-band, so not related to the data-format).

I don’t think we have much benefits in using transit, only 1 seems to be beneficial to us but not unique to transit, 2 can be addressed with some discipline, 3 we don’t use, 4 can be said the same for many other data formats. but hey, it’s a choice, and I can see in some cases where transit might be a good one.

I have mentioned this before, but I would encourage you to have less blunt opinions, especially when it comes to other devs work, who might not have the years of experience you have, as clearly someone has spent some time picking this library and implementing it, and I believe it deserves more than a dismissive comment.
Rather some guidance in why the choice might have been the best, so next time a different choice might be made.
I personally have learned much by using transit, as you often learn a lot from suboptimal choices (I am avoiding the word mistake on purpose :slight_smile: ).

I’m not sure which of those two time-looking numbers are actually the message time

It’s the one you are using, but it’s not a time timestamp and should not be used as such.
It’s a a logical clock (lamport timestamp), which is the only way to ensure casual and total ordering across messages in a distributed, non centralized system, as opposed to receiving time (no causal, nor total ordering, any out-of-order message will do) or sender (no causal, but total-ordering).
The tradeoff as you pointed out, is that it is possible to forge and backdate events (not possible with receiving time), so that’s something that needs mitigation but can’t be completely avoided, but well worth the benefits in my opinion.

cheers


#3

think of anything positive

Whoa, I see that came across out of proportion. Writing the comment I merely went over a similar list like you just wrote down for this particular case, and like you ended up qualifying each potential benefit with a meh - - the solution might be perfectly fine depending on the context, and the link to the source is there for you to make your own call.

Nonetheless, it also remains a critique of what the protocol still looks like today, not the effort or the developers behind it - the end result is what we pass on to our fellow users and developers and someone reading this post and looking for inspiration should come away knowing that here be dragons - it would be irresponsible not to pass on the “learning point” we’ve arrived at through a history of discussions on the subject. Thanks for holding me to high standards - ack there are less direct ways of passing it on.

That’s a really good talk though, nice link :slight_smile: - hope more people get to see and hear Rick because he very eloquently puts a finger on these tricky issues.

I’m confused by his harsh comments on SemVer - I see it as a social construct to inform someone what your intent was - so that you can reap the coordination benefit of that local team standup, but asynchronously - “hey, the thing I just saved on disk looks more useful than all the other versions I saved - I gave it this handy number to tell you how much breakage to expect if you start using it, on a scale of 3 - major, minor, or hopefully painless”. It’s up to you to decide how to use it - if you don’t want to trust it, just use hashes and pay the tax - similar to like when you limit yourself to functional / immutable solutions, you end up writing less efficient software (efficient in the sense of “performance when running on actual hardware”) - comes with the territory, and you have to weigh the advantages. Come to think of it, mutability and trust are remarkably similar in that sense.

lamport timestamp

Looks an awful lot like a unix timestamp, which is why I was able to hack it and write some bad code around it that will fool the uninitiated :slight_smile:

More seriously though, the issue with my code is that it mis-represents guarantees in the data and passes on those implicit guarantees to the next layer that simply cannot know the difference - the Lamport clocks I’ve used in the past use increments of 1 to the last seen timestamp to prevent that kind of abuse by construction. For honest clients, you keep ordering but lose earth-time. Maybe that’s a good trade-off, considering the adversarial setting.

Eventually we can also go with something like Introducing a data sync layer (hello @oskarth - haven’t forgotten) - sync is nice, dag order retained cryptographically.


#4

Thanks for the explanation, I felt it was good to clarify

I’m confused by his harsh comments on SemVer

I have the same feeling, sometimes it seems that Rich lives on a different planet, or does not get his hands dirty with working class code anymore, which for a developer, is essentially the same thing


#5

It’s a bit difficult to see or to say but I don’t think that he lives on another planet. Instead it helps to try to understand his perspective. As a industry programmer he has seen too much „working class code“ and as as a resolution created Clojure. Think about that, think what that means. It means that you can working software when you choose good foundations to build on.

If you consider Rich Hickey to be on a mountain above, you may check on the conversations he had with Alan Kay on hackernews.


#6

Have we come together across the three implementations (kinda) to discuss what is absolutely pivotal to have across platforms, and what is implementation independent?

I’m hoping as these projects get worked on and developed, we come up with a increasingly detailed “protocol-esque” description on how Status works at it’s most lowest level.

If these things aren’t being done, I propose that we spend some time on them. I’d imagine Brussels will have a strong amount of this. If so, let’s sure we document the living hell out of it.


#7

Yes and no. The real asset here is the protocol, which is why the protocol working group was started.
Once you have a protocol, which is really the essence of the solution, implementations are trivial to do and therefore matter little, relatively speaking - they can be lost, blocked or forgotten, but the essence remains.

In the meantime, there’s a lot of good work going on, formalizing these various parts, including key negotiation, synchronization etc, and all else that at its core is actually done in an implementation-neutral way, though the implementations help inform that discussion and act as catalysts. Agree that Brussels is a good place to continue these discussions and lay down some writing, focusing one what is truly distinct.

The main line of thinking right now would be to split things up a little so as to layer the functionality and make it available in pieces for apps / extensions to consume as they wish, such that we provide a strong foundations and others can make the right trade-offs for their own products.


#8

By popular demand, the code can now be found here:

Along with AppImage downloads here:

These may or may not work - the build is not yet reproducible, unfortunately.


#9

Very cool, may a 1000 clients bloom :slight_smile:

Ideally we create a guide on ‘how to create your own Status’, ie an Application Architecture and Designs/Requirements.

Yes, the real asset is incentivzation in the network itself, and ideally a well-thought out protocol that encapsulates that.