Extensions - scripting proposal

Here are just some of my thoughts about problems I see with the current extension design (https://github.com/status-im/pluto):

Problematic points

  • No much thought about who will be extension writers actually and what’s the purpose of extensions. Are they primary meant for core developers to be able to make changes faster/more dynamically ? 3rd party Web3 devs ?
    Empower non/less technical people to contribute directly by writing simple functionality ?
    It’s quite important, as it has design implications and things which can make one group happy can make the other group angry and vice versa.

  • Scripting (eq basic programming constructs) not supported with no clear plan how to introduce it later.
    This has multiple implications, one of them is that we are many times re-inventing the wheel and introducing incomplete/buggy implementations while parsing extensions and generating views (lexical scopes, destructuring, etc.) second and most important is that extensions are almost unusable in current form for authors not having access to status-react codebase as well (where they can “cheat” and make specialised components later referenced in extension).
    This could be of course addressed by slowly adding functionality (like logical operators, numeric operators, for operator…) to the view “language” but we will spent a lot of time re-inventing the wheel to create a subset of Clojure with little bit altered syntax (which will feel super weird for any extension writer who already knows Clojure).

Ideas how to mitigate them

  • Discuss and think much more about who will be extensions writers :
    Are we supposed to use them internally as a tool to boost productivity and/or ship (optional) functionality without going through traditional release cycle and app store installation ? In that case any non-standard syntax and unnecessary deviations from Clojure/Re-frame/Reagent conventions are problematic and not desired.
    Will we target Web3 devs, most of them fluent in JS ? In that case we should probably re-evaluate the decision to push EDN (Clojure syntax) in extensions on them altogether.

  • In case we decide that EDN syntax is indeed the best, I propose to go the full way and actually include restricted cljs execution environment for extensions.

Restricted cljs environment for extensions

The way how I imagine it could work would be that for certain extension contexts (like views/view-id, events/event-id) the forms would be evaluated in restricted cljs execution environment, with top-level bindings provided by us and every symbol restricted to either refer our top-level bindings or symbols from clojure.core namespace (we can fine tune that).
Due to the fact that code is data in lisp and awesome existing projects like clojure.tools.analyzer, we can run the form through analyser which will produce enriched AST like this:

clojure.tools.analyzer.jvm> (ast/children (analyze '(do 1 2 :foo)))
[{:op   :const,
  :id   0,
  :type :number,
  :val  1,
  :form 1,
  ...}
 {:op   :const,
  :id   1,
  :type :number,
  :val  2,
  :form 2,
  ...}
 {:op   :const,
  :id   3,
  :type :keyword,
  :val  :foo,
  :form :foo,
  ...}]

With this result, where each symbol is enriched with important information, we can trigger warning and reject the code every time non supported operation is used and basically restrict the code to functional, pure subset of clojure just producing data - no side effect calls, no platform interop calls, just pure functions from clojure.core namespaces + our provided functionality which will be injected into execution env as top level bindings.

Given that piece of code passes the security analysis, we can then safely evaluate and execute it.

So this in extensions:

views/preview
(let [{:keys [a b c]} params]
  [react/text a])

Will be evaluated as

(fn [params]
  custom-code-written-by-extension-writer) ;; params provided in top-level bindings

Advantages of such approach:

  • No more reinventing the wheel by re-implementing lexical scoping, necessary operators, etc.
  • For simple things, the custom code will stay exactly as simple as before, so writing simple view won’t be any more daunting or discouraging for the beginners compared to current solution.
    But need to “cheat” and write custom components for extensions will be greatly reduced and with little bit time and practice completely eliminated (when we fine tune what bindings will be actually provided for the custom code)
  • Because we will evaluate just specific parts of the extension (like views, events and subscriptions) in the restricted cljs execution environment, we won’t loose most of the existing validation functionality, where we validate hook properties, etc.
    Given that clojure.tools.analyzer is super flexible and provides awesome data-rich output, we can easily do same analysis as before, eq checking if subscribing to subscription actually refers subscription exposed by host (capacities map), etc.
  • There will be no extra “scripting” resource to refer and going from simple extension with barely any logic to more advanced with custom events/subscriptions and/or view leveraging map for repeating elements will feel natural.
  • Extension writing will be great experience even for experienced users (while less experienced users won’t be penalised), because it will be basically functional programmers paradise - writing pure functions, producing just data (view markup for views, effect map for events…) with host developers (core status devs) responsible for ensuring that such data are meaningfully translated into necessary side-effects.
3 Likes

Thanks @jan.herich

From a product/strategy perspective, I think that extensions are/should be BOTH a way for core devs to experiment with new and different features that will prevent the 6-monthly cycle of discussing something new but then maybe not getting it into production AND to allow 3rd party devs to interface easily with Status and learn about what building really decentralized products/services might actually look like. The two are not mutually exclusive to me, just different iterations of the same idea. We:

  1. Prove the use cases ourselves,
  2. Partner closely with some selected others and help them implement,
  3. Leverage the architecture to allow for greater organisational distribution, and
  4. Then figure out how to make it SUPER easy for 3rd party devs to use it, maybe with some kind of auto-generation of the .edn file?

For me, the goal has always been to stay away from requiring external developers to write Clojure, as it is likely just too high a barrier for the mass adoption we want to achieve in multiple spheres. I hope that @jeluard can answer some of the concerns you raise here, as I am very unconvinced that even a restricted cljs environment is a good idea for the kind of toolset I’d like to see available for “normal” developers :wink:

Understood and if that’s the case (clojure/edn being too much/alien to mainstream devs) I propose to abandon .edn format altogether and consume extensions defined as JSON, also get rid of the things like let blocks for lexical bindings.
We also need to think about suitable and lightweight (eq not like Otto) sandboxed js execution environment for scripting.

1 Like

We also need to think about suitable and lightweight (eq not like Otto) sandboxed js execution environment for scripting.

We didn’t use otto for a while, i suppose JSCore would be the best choice for us.