The road to Nix, a functional package manager to rule them all


#41

Looks like the issue with Status Desktop is related to the embedded ubuntu-server (some issue running under a Nix environment that I’ll need to look into). This became obvious when I noticed that the only time I could get the app UI to show up was when I was running the release Status Desktop in the background (and hence the debug Nix Desktop app could find an ubuntu-server process to talk to).

cc @volodymyr.kozieiev

If I run LD_DEBUG=all ./ubuntu-server.js I get

Loaded realmConstructor: function Realm() { [native code] }
-- Server starting
(node:7894) UnhandledPromiseRejectionWarning: Error: Error while obtaining machine id: Error: stderr maxBuffer exceeded
    at Socket.onChildStderr (child_process.js:352:14)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at addChunk (_stream_readable.js:263:12)
    at readableAddChunk (_stream_readable.js:246:13)
    at Socket.Readable.push (_stream_readable.js:208:10)
    at Pipe.onread (net.js:601:20)
    at /home/pedro/src/github.com/status-im/status-react/node_modules/node-machine-id/dist/index.js:1:7564
    at ChildProcess.exithandler (child_process.js:288:5)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at maybeClose (internal/child_process.js:915:16)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:209:5)
(node:7894) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:7894) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

I believe this is a good time to create a Nix package for ubuntu-server, so that it links to the right libraries and can run inside a Nix shell. Here’s the current output for realm.node:

ldd /home/pedro/src/github.com/status-im/status-react/node_modules/realm/compiled/node-v57_linux_x64/realm.node
	linux-vdso.so.1 (0x00007ffe523d8000)
	libstdc++.so.6 => not found
	libm.so.6 => /nix/store/0fv2lnbr02qzh3l387rvd6vl8443mpq7-glibc-2.27/lib/libm.so.6 (0x00007f08ae57d000)
	libgcc_s.so.1 => /nix/store/0fv2lnbr02qzh3l387rvd6vl8443mpq7-glibc-2.27/lib/libgcc_s.so.1 (0x00007f08ae367000)
	libpthread.so.0 => /nix/store/0fv2lnbr02qzh3l387rvd6vl8443mpq7-glibc-2.27/lib/libpthread.so.0 (0x00007f08ae148000)
	libc.so.6 => /nix/store/0fv2lnbr02qzh3l387rvd6vl8443mpq7-glibc-2.27/lib/libc.so.6 (0x00007f08add94000)
	/nix/store/0fv2lnbr02qzh3l387rvd6vl8443mpq7-glibc-2.27/lib64/ld-linux-x86-64.so.2 (0x00007f08af53d000)

#42

Daily update:

  • Made some progress after investigating further yesterday. Using the realm.node built in Nix in this PR (thanks hlolli!) I could run ubuntu-server.js on a separate terminal that is used by the debug Status Desktop app.
  • Still running into issues opening the realm database. For instance, if I copy my regular database to another location and point STATUS_DATA_DIR to that, I get:
DEBUG [status-im.utils.keychain.core:156] - no key exists, creating...
DEBUG [status-im.utils.keychain.core:148] - storing encryption key
WARN [status-im.utils.keychain.events:11] - invalid key detected
DEBUG [status-im.utils.handlers:34] - Handling re-frame event:  :init.callback/get-encryption-key-success
INFO [status-im.data-store.realm.core:124] - realm: moving all realms
ERROR [status-im.data-store.core:25] - Could not move realms TypeError: re-matches must match against a string.
DEBUG [status-im.data-store.realm.core:199] - Opening base realm... (first run)
INFO [status-im.data-store.realm.core:177] - migrate-realm
WARN [status-im.init.core:43] - Could not decrypt database Error: Unable to open a realm at path '/home/pedro/src/github.com/status-im/status-react/.status/Status//default.realm/default.realm': Invalid mnemonic. top_ref[0]: BC8252C000001A25, top_ref[1]: 3DCB61257237786E, mnemonic: 91 5D 19 BD, fmt[0]: 54, fmt[1]: 122, flags: F6.
DEBUG [status-im.utils.handlers:34] - Handling re-frame event:  :init.callback/init-store-error
-- Session ended
Terminated

#43

just thinking aloud, haven’t followed the conversation, but is that on desktop? it might might due to react-native-keychain maybe not generating a correct password? I would look into what is been passed to us, just to make sure it looks ok


#44

Yeah, this is only on desktop. I have just tested on a mint VM and have the same issue, so it’s indeed real.

that sounds very likely, will look into that.


#45

OK, my suggestion is the following: the PR has been reviewed and tested since the beginning of the week, ready to be merged. We should go ahead and merge it, and continue looking into this issue when @volodymyr.kozieiev is back from vacations, if not earlier. There is already an external contributor offering a PR to improve the Nix branch, so the sooner we make this the default branch, the better. I’ll post transition instructions in a follow up post.


#46

Without knowing too much about what could go wrong, in general it sounds good to me. If you want, we can also take it up on Core Dev call Monday. Up to you, or if someone else strongly disagrees.


#47

When you read this, the feature/nix branch will have been merged. You might be asking yourself, how does this impact me? Here are a few points:

  • NVM, homebrew and Conan are no longer used, you can remove any you don’t use from your machine if you’d like;
  • The Android SDK location has moved from ~/Android/Sdk to ~/.status/Android/Sdk (when running make setup the SDK will be downloaded to the new location);
  • The Android NDK has moved from under ~/Android/Sdk/ to the Nix store, so you can remove it from ~/Android/Sdk/ in order to save disk space;
  • If you downloaded Qt for Status Desktop, this is no longer used and can be uninstalled.

To set up your environment, follow the updated instructions in https://status.im/build_status/ (should be live soon), basically:

  • Run make setup
  • Run make shell any time you need to interact with the project (so you’re inside a Nix environment).

Let me know if you run into any issues or rough edges. A few of them are already known and I expect to work on them in the upcoming days.


#48

I’d like to thank you Pedro for this amazing work. This should in the long term make our builds(both manual and CI) more reliable and predictable. As with any big change this might cause some issues initially, but all in all this is definitely a move in the right direction. I owe you a beer, or probably several.


#49

I’ve had a call this morning with @volodymyr.kozieiev and the nix branch is working for him, so it won’t be an immediate blocker for him. It will be a nuisance for new developers wanting to debug Status Desktop, but I feel like that will be greatly outweighed by the number of external contributors now able to install and run the mobile app from the first attempt.


#50

I had managed to build from develop and able to successfully create Status-x86_64.AppImage however I was unable to execute and successfully run the AppImage

NixOS isn’t really designed to run arbitrary binaries, so I am using appimage-run utility, further discussion here.

$ appimage-run Status-x86_64.AppImage
./AppRun: error while loading shared libraries: libharfbuzz.so.0: cannot open shared object file: No such file or directory

It’s unclear to me if harfbuzz is a missing dependency that appimage-run is missing, or something our AppImage is missing?

I’ve had success with appimage-run on other AppImages, such as Turtl.

Alternatively it would be great to have a status nixpkg to install.


#51

I just confirmed that libharfbuzz.so.0 is not included in the AppImage. In fact, linuxdeployqt blacklists some libraries which it assumes are present in all supported distros. We could modify linuxdeployqt to not blacklist those DLLs (at the expense of the size of the AppImage), however for NixOS, I think the right thing to do is to just publish a package on nixpkgs and avoid patching things back and forth.


#52

Daily progress update:

  • In an effort to make the whole pipeline reproducible, I’ve started looking into writing a Nix expression to build status-go. This should make it easier to test status-react with user status-go branches, since Nix will know how to build it if necessary (if it is not yet on the Nix cache). It will also simplify the distribution model and eventually make it easy to have a single distribution model for Desktop and Mobile.
  • In view of this, I’ve determined that in order to build Mobile status-go, we’ll also need to build and package the gomobile project. This is something that in time we can contribute back to the Nixpkgs community.
  • Today I’ll look into building the gomobile package, which seems to require some improvements in the android-ndk package.

#53

Daily progress update:

  • After fixing the Android NDK package and building a gomobile one, I’ve been able to use that to generate a status-go Android artifact, but only from the nix-shell. If It try to run the same command in the status-go expression’s buildCommand attribute, I get some failures in cgo. Currently trying to figure out the cause.

    # runtime/cgo
    fork/exec /nix/store/ik293nv1ynswd6q97laz02bys598spgy-android-ndk/toolchains/llvm/prebuilt/linux-x86_64/bin/armv7a-linux-androideabi16-clang: no such file or directory
    

#54

Daily progress update:

  • Yesterday was a very productive day, marked by being able for the first time to produce a statusgo.aar artifact in the /nix/store. After much spelunking in the source code of gomobile and nixpkgs, and with the help of @jakubgs with his suggestion to employ strace to figure out the missing file, I was able to patch gomobile (it wasn’t able to use our downloaded Android SDK and needed patching to deal with the symlinking of nixpkgs’s Android SDK).
  • The road ahead is now much clearer and allows me to establish a plan:
    • Expose the generated .aar file to the Gradle files, and replace code that downloads the artifact from Digital Ocean;
    • Figure out why libgojni.so files in statusgo.aar are 50% larger than existing ones. Probably strip is not being run on them. Update: Unable to strip library '/home/pedro/src/github.com/status-im/status-react/android/app/build/intermediates/transforms/mergeJniLibs/release/0/lib/armeabi/libconceal.so' due to missing strip tool for ABI 'ARMEABI'. Packaging it as is.
    • Try to use Android NDK package from nixpkgs. This will inform the same improvement for the rest of status-react;
    • Replace bundle-status-go.sh with a convenient way to build a local status-go;
    • Fine tune the Android SDK component versions installed by Nix using androidenv.composeAndroidPackages.
    • Build status-go for iOS.
      • Replicate necessary fixes on Android NDK for Darwin platform;
      • The problem here is how to expose XCode to the status-go recipe, which already uses a derivation specific to Go compilation;

Link to new Pivotal Tracker story: https://www.pivotaltracker.com/story/show/164984148


#55

Good news, I’ve just installed my first Status app building status-react and status-go from scratch with a single make shell / make release-android sequence! Nix will build gomobile and use it to build the status-go version specified by the STATUS_GO_VERSION file if that hasn’t been built before.

CI logs here for the mind-blowing experience of seeing status-go and status-react building on the same go.


#56

Daily progress update:

  • Built status-go package consisting of Android libraries (/lib)and Linux/macOS binaries (in /bin, for statusd, bootnode, node-canary, etc);
  • Replaced usage of React Native’s Gradle wrapper with Nix’s Gradle package. That way we avoid one additional download;
  • Finished gomobile package, leveraging Nix’s Android NDK and SDK packages instead of our custom ones.

Next I’ll pair with @jakubgs to do the same for iOS.


#57

Hey @zahary, just so you’re aware of the current CI status, we built a custom Docker image based on NixOS’s which has the cache prepopulated. We also run a nightly Jenkins job which incrementally populates https://nix-cache.status.im. I’m hoping to work with @jakubgs soon to find a way for our NixOS docker image to be able to share the Nix store with other builds so we don’t need a fat prepopulated Docker image.


#58

Daily progress update:

  • Fixed iOS build by wrapping system XCode in a Nix package (with version-check). Looks like the nixpkgs xcode doesn’t contain the mobile toolchain;
  • Adapted Desktop CMake file to leverage the Nix status-go package. This means that the only platform which builds a custom status-go now is Windows. All the others rely on the Nix package;
  • Did a 1-1 with Jakub to go over the structure of the upcoming PR.

#59

Daily progress update:

  • Worked with Jakub to get nix build command to work properly on both macOS and Linux, and to fetch/build all packages. It previously was only fetching some, so that wasn’t ideal to e.g. populate the Nix cache.
  • Met with Jakub and Hlöðver Sigurðsson (https://github.com/hlolli) who’s versed Nix/NixOS so he could look into how we are doing things, and apparently we’re on a good path.

#60

oh, cool!

I still have some problems with running the app for Genymotion (react-native can’t find the server), iOS works just fine now.