The road to Nix, a functional package manager to rule them all


#121

@andrey Two things are strange in the message you’re seeing:

  • It’s showing you that both tools are missing from your system. I took a look to confirm and this is not a catch-all message, there are tests behind it, so it does mean it didn’t find Xcode 10.1 (by invoking xcrun xcodebuild -version) nor the iPhone SDK (by running xcrun --sdk iphoneos --show-sdk-version);
  • It doesn’t seem to have even started building status-go. How did you tell status-react to use the status-go branch, did you run scripts/update-status-go.sh onb.v.0.1? I did that on both Linux and macOS and it built fine.

#122

yes i did

unpacking...
[13.3 MiB DL]
path is '/nix/store/dkfqc0sdvqszvrssqr8rwv31hihr03g4-onb.v.0.1.zip'

#123

i’m building android why it wants ios tools, there is a full log

andreys-MBP:status-react andrey$ make startdev-android-avd
these derivations will be built:
  /nix/store/spq4p8h9xv2vd73m9kxm2hdlh6cvdn6m-status-go-onb.v.0.1-android.drv
building '/nix/store/spq4p8h9xv2vd73m9kxm2hdlh6cvdn6m-status-go-onb.v.0.1-android.drv'...
unpacking sources
unpacking source archive /nix/store/bhgd62k3n16hc4hrplqkjg6m1wk4v6d5-status-go-source
source root is status-go-source
patching sources
configuring
There are some required tools missing in the system:
- [ ] Xcode 10.1
- [ ] iPhone SDK
Please install Xcode 10.1 from the App Store.
builder for '/nix/store/spq4p8h9xv2vd73m9kxm2hdlh6cvdn6m-status-go-onb.v.0.1-android.drv' failed with exit code 1
error: build of '/nix/store/spq4p8h9xv2vd73m9kxm2hdlh6cvdn6m-status-go-onb.v.0.1-android.drv' failed
make: *** [_startdev-android-avd] Error 100

#124

is there a way to use a package https://ci.status.im/job/status-go/job/prs/job/ios/job/PR-1464/5/ and do not build status-go ?


#125

Well, there’s a CI job that will populate the Nix cache with the results of a given branch, so if you have a branch that is pointing to the right status-go, you could trigger it with https://ci.status.im/job/status-react/job/nix-cache/job/macos/build?delay=0sec
It won’t use the result from the status-go CI build, but rather will build status-go itself like you’d do it on your machine (that’s the standard way now anyway).


#126

This is what I’m seeing, btw:

make startdev-android-avd                             SIGINT(2) ↵  5.72G RAM  558  12:35:15
these derivations will be built:
  /nix/store/spq4p8h9xv2vd73m9kxm2hdlh6cvdn6m-status-go-onb.v.0.1-android.drv
building '/nix/store/spq4p8h9xv2vd73m9kxm2hdlh6cvdn6m-status-go-onb.v.0.1-android.drv'...
unpacking sources
unpacking source archive /nix/store/bhgd62k3n16hc4hrplqkjg6m1wk4v6d5-status-go-source
source root is status-go-source
patching sources
configuring
building

Building mobile library for android

errors
internal/cpu
internal/race
math/bits
unicode/utf8
runtime/internal/sys
sync/atomic
math
encoding
unicode
runtime/internal/atomic
internal/bytealg
unicode/utf16
crypto/internal/subtle
crypto/subtle
internal/testlog
vendor/golang_org/x/net/dns/dnsmessage
internal/nettrace
container/list
vendor/golang_org/x/crypto/cryptobyte/asn1
github.com/status-im/status-go/vendor/golang.org/x/net/html/atom
runtime/cgo
github.com/status-im/status-go/vendor/golang.org/x/text/encoding/internal/identifier
github.com/status-im/status-go/vendor/golang.org/x/text/internal/utf8internal
github.com/status-im/status-go/vendor/github.com/hashicorp/golang-lru/simplelru
runtime
container/ring

#127

https://ci.status.im/job/status-react/job/nix-cache/job/macos/246/ ok we have a build but it still trying to build it locally, are there extra steps to do?

EDITED: can build with cache now, maybe it takes some time idk


#128

btw i have XCode installed, but it’s 10.2


#129

Then you can search for "10.1" in the code and change it to "10.2" locally, so it doesn’t complain. If we see that most people are on 10.2 nowadays, we should probably upgrade our CI agents and then commit that.


#130

can’t find “10.1” anywhere :frowning:


#131

#132

btw, can we allow some kind of local config for it? like local.properties in Java or some env variables? at least when I build my local dev builds, I don’t care that much about reproducibility of then, so I can use tools from my toolbox.

alternatively, we can make some strict mode that checks tools versions and default dev mode that allows, say, Xcode 10.x.


#133

Yeah, we can certainly add some more logic to this assertion. For now, we could also check that it is >= 10.1.


#134

Daily progress update:

  • Started implementing Nix wrapper for lein prod-build-android;
  • In order to achieve that in a pure environment, I need to wrap clojars maven dependencies with clj2nix.

#135

Hmm, looks like the nixpkgs recipe doesn’t support that for Xcode, we’d need to roll our own recipe (not hard, but probably something for later, as I’m deep in the weeds in the clojars repo stuff).


#136

Daily progress update:

  • Managed to run lein prod-build-android as a Nix expression, missing only accepting node_modules as an argument so that it is present in the intermediate build.

#137

Daily progress update:

  • Cleaned up the scripts to run lein prod-build-android and created scripts to automate the generation of Nix expressions by discovering the necessary Maven dependencies for the project.
  • Will now focus on adding the node_modules to the project and finishing a Nix lein prod-build-android build successfully, to then move on to gradle assembleRelease.

#138

Daily progress update:

  • Made good progress on the port to Nix of gradle assembleRelease task. It already configures most of the react native projects (including react-native-status), but is coughing on react-native-webview:
> Configure project :react-native-webview-bridge
WARNING: Configuration 'compile' is obsolete and has been replaced with 'implementation' and 'api'.
It will be removed at the end of 2018. For more information see: http://d.android.com/r/tools/update-dependency-configurations.html
Could not resolve compiler classpath. Check if Kotlin Gradle plugin repository is configured in project ':react-native-webview'.

FAILURE: Build failed with an exception.

* What went wrong:
Could not determine the dependencies of task ':react-native-webview:compileReleaseKotlin'.
> path may not be null or empty string. path='null'

If I change the way the kotlin-android plugin is imported in react-native-webview as per https://kotlinlang.org/docs/reference/using-gradle.html#configuring-dependencies, then I get the following error:

FAILURE: Build failed with an exception.

* Where:
Build file '/build/status-react-release-android/node_modules/react-native-webview/android/build.gradle' line: 27

* What went wrong:
Plugin [id: 'kotlin-android'] was not found in any of the following sources:

- Gradle Core Plugins (not a core plugin, please see https://docs.gradle.org/4.10.2/userguide/standard_plugins.html for available core plugins)
- Plugin Repositories (plugin dependency must include a version number for this source)

This is probably due to missing Maven dependencies, so I’ll be working to automate further the capture of Gradle dependencies using the gradle dependencies command or something like gradle :app:dependencyInsight --configuration compile --dependency <name>.


#139

Daily progress update:

  • Finally figured out what was happening with Gradle. I’m putting the thought process below just in case this happens to bite us (or someone else) in the future.

The error with :react-native-webview was only visible because I had temporarily commented https://github.com/status-im/react-native/blob/status-0.59.3-1/ReactAndroid/release.gradle#L77 in order to progress further. Clearly this line was needed a bit further down the line after other things were fixed, so I uncommented it. After that I started getting this error:

FAILURE: Build failed with an exception.

* Where:
Script '/build/status-react-release-android/node_modules/react-native/ReactAndroid/release.gradle' line: 77

* What went wrong:
A problem occurred configuring project ':react-native-android'.
> path may not be null or empty string. path='null'

* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Exception is:
org.gradle.api.ProjectConfigurationException: A problem occurred configuring project ':react-native-android'.

After much experimentation (and realizing that the build actually proceeded well if I ran a nix-shell and tried it again on the same temp build directory), I decided to comment the line again and add gradle app:dependencies --stacktrace to the Nix build. This yielded the same error, but with a different callstack, no longer pointing to :react-native-android. The stack trace interestingly mentioned org.gradle.api.internal.artifacts.repositories.DefaultFlatDirArtifactRepository.getDirs. Looking at our own codebase, we do use flatDir in https://github.com/status-im/status-react/blob/develop/android/build.gradle#L50. So the issue turned out to be a missing dependency on status-go which would set the STATUS_GO_ANDROID_LIBDIR env var. How useless can gradle error messages be? To prevent this from happening silently again, I’ve added our own logic to report if that environment variable is not set.


#140

A few days ago I was going through a video from Nix’s founder on Nix’s roadmap for 2019, and was happy to see they are planning to introduce content addressable derivations (i.e. packages). This means that a derivation’s address will no longer depend on the derivations’ inputs and configuration, but instead on the result of the derivation. In practical terms, that means that if you change a space in an expression (let’s say in a shell script) and that doesn’t impact anything on the output, none of the dependencies will get rebuilt.