The road to Nix, a functional package manager to rule them all


#81

@yalu I’ve created a status-react branch called chore/nix-gc-roots that adds a Makefile target to prevent garbage collection of status-react. Feel free to give it a try and then call nix-collect-garbage -d to see if it works (it does for me).


#82

i’ve changed CN cache to be first.
e.g.
substituters = https://nix-cache-cn.status.im/ https://nix-cache.status.im/ https://cache.nixos.org
i’m trying branch nix-cn-cache, it shows it will still access https://cache.nixos.org randomly which is the slow part. is this behavior correct?
1


#83

failed at last.
1


#84

I guess it would be best for you to remove the slow servers from your list. Even a single server should be enough and will probably end up being faster in your case.


#85

@pedro Awesome!!! Works! :star_struck:
feels so fresh and so clean
Thank you!!!


#86

Great to hear that! Merged to develop.


#87

Daily progress report:

  • Investigate work required to get to reproducible builds (e.g. one step is to add SHA values to existing canary build which always builds same historic commit, this way we’ll know when an unknown external factor - like a download - has caused the artifacts to change).

#88

Great news, I think! As for reproducible Android builds, with the Nix effort, it looks like the only difference right now between 2 apks of separate CI builds is in index.android.bundle and specifically in how the calls to the ClojureScript logging library (timbre) are transpiled. If we fix this one issue, we’ll have bitwise identical Android builds:

image

from what I see here https://github.com/ptaoussanis/timbre/blob/master/src/taoensso/timbre.cljx#L370, the changing parameters correspond to ?file and callsite-id. callsite-id is computed from ?file in https://github.com/ptaoussanis/timbre/blob/master/src/taoensso/timbre.cljx#L558 and the unstable filename seems to be the reason why it is also changing.

Update: found out where the temp path is being created (in Leiningen): https://github.com/technomancy/leiningen/blob/f7e1adad6ff5137d6ea56bc429c3b620c6f84128/leiningen-core/src/leiningen/core/eval.clj#L238


#89

A good video by Sander van der Burg about replacing NPM with Nix, with an introduction to the Nix ecosystem at the beginning: https://www.youtube.com/watch?v=Rh-CSJL1Q-U


#90

incredible job :slight_smile: great work @pedro


#91

I’ll be tracking the work/discussion on Android reproducible builds in https://github.com/status-im/status-react/issues/8203


#92

Came across this post by the Pinterest engineering team about how they integrated Nix and Buildkite to end the duality between dev and CI machine setup: https://medium.com/@Pinterest_Engineering/continuous-integration-for-ios-with-nix-and-buildkite-ef5b36c5292d

Interesting to see the very similar path taken, namely with Fastlane and Bazel.


#93

Daily progress update:

  • Having React Native being built through npm is causing build paths to get pulled in to the lib*jni.so modules. This is done by RN dependencies, so a simple commit in the RN fork won’t fix it. I’m going to try starting with a sample RN project and attempt to build RN from the ground up with Nix. That should enable us to have reproducible build paths, as well as a myriad other advantages, beginning with improved setup times (no need to compile boost, for example).

Update: I’ve set up a public Git repo where I’m working on a Hello Word Nix-based React Native app here: https://github.com/PombeirP/nix-react-native-test. I was able to successfully build it and deploy to Android, so clearly something else is the problem in status-react, most likely one (or several) of the RN modules that we import.


#94

In view of the goal of migrating from yarn install to yarn2nix (which should allow us to get stable paths inside RN dependencies), I think it is helpful to have an overview of the whole process happening when you run TARGET_OS=android make shell for the first time:

Overview of status-react setup

  1. make shell is called
    1. Pure dependencies are setup by nix-shell
      1. node
      2. yarn
      3. Android NDK/SDK, etc.
    2. Impure dependencies are setup through prepare-for-platform.sh
      1. Create symlinks for npm/yarn pointing to mobile_files
      2. yarn install --frozen-lockfile
        1. react-native and react-native-* installed along with all other dependencies in package.json
      3. gradlew react-native-android:installArchives
        • NOTE: This is the point at which previous attempts with yarn2nix have failed, since there is a local build of the react-native fork happening in Gradle.
        • node_modules/react-native/android/com/facebook/react/react-native/x.xx.x is
          created (referenced as a maven repo in android/build.gradle). QUESTION: At which point exactly is it created?
        • node_modules/react-native/ReactAndroid/build is created (sub-folders: docs, downloads, generated, intermediates, libs, outputs, poms, react-ndk, third-party-ndk, tmp)
        1. android/app/settings.gradle
          • Configure project :app
        2. android/app/build.gradle
          1. apply from: project(':react-native-config')
          2. getVersionCode
          3. Configure projects
            1. react-native-background-timer
            2. react-native-status
            3. react-native-webview
            4. etc…
          4. Tasks from :react-native-android:
            1. androidJavadoc, androidJavadocJar, androidSourcesJar, preBuild, preReleaseBuild, compileReleaseAidl, compileReleaseRenderscript,
            2. checkReleaseManifest, generateReleaseBuildConfig, generateReleaseResValues, generateReleaseResources, packageReleaseResources, processReleaseManifest, generateReleaseRFile
            3. prepareLintJar, generateReleaseSources, javaPreCompileRelease
            4. createNativeDepsDirectories
              1. Boost
              2. DoubleConversion
                1. downloadDoubleConversion, prepareDoubleConversion
              3. Folly
                1. downloadFolly, prepareFolly
              4. Glog
                1. downloadGlog, prepareGlog
              5. JSC
                1. downloadJSC, prepareJSC
            5. buildReactNdkLib
              1. make builds node_modules/react-native/ReactAndroid/src/main/jni/react/jni for 4 platforms
            6. packageReactNdkLibs
            7. compileReleaseJavaWithJavac, extractReleaseAnnotations, mergeReleaseConsumerProguardFiles, mergeReleaseShaders, compileReleaseShaders, generateReleaseAssets, packageReleaseAssets, packageReleaseRenderscript, processReleaseJavaRes, transformResourcesWithMergeJavaResForRelease, transformClassesAndResourcesWithSyncLibJarsForRelease, compileReleaseNdk, mergeReleaseJniLibFolders, transformNativeLibsWithMergeJniLibsForRelease, transformNativeLibsWithStripDebugSymbolForRelease, transformNativeLibsWithSyncJniLibsForRelease, bundleReleaseAar, signArchives
            8. installArchives
    3. Pure shell is made available

#95

wow, we might want to have this somewhere in the docs also!


#96

I’d agree, however I’m hoping to make this information irrelevant (at least in it’s current form) soon :slight_smile:


#97

Snapshot of what gradlew react-native-android:installArchives builds:

image


#98

Daily progress update:

  • Got a first successful gradle build of our RN fork, even though it’s not anywhere near perfect (some parts of it need to be impure - as commands such as installRelease will attempt to write inside node_modules/react-native -, and need to find a way to keep the rest pure).

#99

@pedro I only skimmed through so it might be offtopic, I came accross https://github.com/callstack/haul as an alternative react-native packager, maybe it can help you in your quest?


#100

Thanks @yenda, that could be a promising way out (having glanced at the source code for a bit). I’m trying it on my sample RN project but ran into an error, looking into it:

./App.js
Module not found: Error: Can't resolve 'react-hot-loader' in '/home/pedro/src/github.com/status-im/nix-react-native-test'
resolve 'react-hot-loader' in '/home/pedro/src/github.com/status-im/nix-react-native-test'
 Parsed request is a module
 using description file: /home/pedro/src/github.com/status-im/nix-react-native-test/package.json (relative path: .)
   Field 'browser' doesn't contain a valid alias configuration

UPDATE: Managed to fix that. Apparently react-dom and react-hot-loader were missing from devDependencies.