The road to Nix, a functional package manager to rule them all

Hi @pedro , nix packages is very big…it’s not convenient for devs in China to build cause of the GFW block. It would be great if there is an speed up image source within China to download the nix packages… i’m struggling with building atm… :expressionless:

copying path ‘/nix/store/nymkh0w8h0h70jmfyll1ivw1436v42vl-platforms-28’ from ‘https://nix-cache.status.im’…
copying path ‘/nix/store/bqn10ibvsf6wz8hqx51pl71lg2i0p80x-ndk-bundle-19.2.5345600’ from ‘https://nix-cache.status.im’…
warning: unable to download ‘https://nix-cache.status.im/nar/bqn10ibvsf6wz8hqx51pl71lg2i0p80x.nar’: HTTP error 200 (curl error: Stream error in the HTTP/2 framing layer); retrying in 309 ms
non-zero padding
copying path ‘/nix/store/hv95kz66aibdf9ckbnv5nsjgm2i1335l-nghttp2-1.36.0-lib’ from ‘https://cache.nixos.org’…
error: build of ‘/nix/store/19srs5p4257r1sjx6rvcvih2xd9s85pz-curl-7.64.0.drv’, ‘/nix/store/2dflx8j1g3n0jcivf6if87k961l4n161-status-go-v0.23.0-beta.10.drv’, ‘/nix/store/2vd4nkkw4vk42gmv113m7q35dmma0gpn-ps-1003.1-2008.drv’, ‘/nix/store/54rhqsrqnpapfk0xmqfms9n2jbgm0zlh-git-2.19.2.drv’, ‘/nix/store/9p4cyy0685n688cbsjcj1l7w69y04bi7-stdenv-darwin.drv’, ‘/nix/store/i590s78f99y2717jpmapfglfipjl8ih3-unzip-6.0.drv’, ‘/nix/store/ixfv1ghaih8nb87jykvvk7p3z3ghi6nz-node-react-native-cli-2.0.1.drv’, ‘/nix/store/jqmrra5616znb7hx16pb249f89j2g17s-ruby-2.5.3.drv’, ‘/nix/store/k8q3yla32j4r5cjcbgz8x2ba1xmsbbnh-nodejs-10.15.0.drv’, ‘/nix/store/ln690p68g41393mvmi9dpw24ds0n9z82-bundler-1.17.2.drv’, ‘/nix/store/n89ws2zm2ax58azbqhsbi9p20wvp4ax5-wget-1.20.1.drv’, ‘/nix/store/p7j89xyf5qv4qyd8jqr4l2nbzmydrd3h-androidsdk.drv’, ‘/nix/store/qspwik4smlbdyjifmwsl9id2d82l3nm9-jq-1.6.drv’, ‘/nix/store/sn9zxjphg1x7231yv4n4x42mc5g02hav-yarn-1.13.0.drv’ failed
make: *** [shell] Error 1

1 Like

finally, i used vpn to get through this…it’s big!!..> 3 GB

1

i won’t success without a vpn, as i said before it would be great if there is an image source choice:roll_eyes:

1 Like

I will look into a possibility of doing a read-only mirror in China/near China.

The best way would be to have multiple servers behind a load balancer that distributes traffic based on geographical location, but I don’t know how doable that would be with Nix. I’d need some kind of syncing solution between the servers.

1 Like

Daily progress update:

  • Making progress towards a pure macOS build, seems that the last remaining roadblocks are the CMake builds of qtkeychain and SnoreNotify.

Daily progress update:

  • Finished macOS pure environment update PR.
1 Like

Hey man, I’ve created a new Nix cache located in HongKong: https://nix-cache-cn.status.im/

I’ve added it to status-react in the following branch: nix-cn-cache (PR: #8170)

If you could try it out and move the cn one to be first in order in nix.conf and report back with results of using it I’d really appreciate it.

2 Likes

download from nix-cache-cn.status.im is fast,but download from cache.nixos.org is very very slow and timeout at last causing make shell failed, so it would be great if there is also a mirror of cache.nixos.org … and if there is a datetime field output before every log row, that would looks better?

1

2

2 Likes

I’m in Asia, and I can confirm cache.nixos.org is slow for me, in that, sometimes I have to restart make shell for it to complete downloading packages(if packages are to be downloaded). If I do make clean-nix and restart, it takes “eons” to download all the packages and the build system to be ready for work.

Another thing. I started using Nix as my MacOS package manager and completely got rid of Homebrew. How can I prevent nix-collect-garbage -d to not wipe out status-react packages?

1 Like

Well, then I guess our cache should be first, not second in order, so I’ve changed the nix-cn-cache branch to set substituters instead of extra-substituters and put https://cache.nixos.org as second in order:
https://github.com/status-im/status-react/commit/715c9c032846d25bd617d3c0cb4a6af2248251a0
You can try it out, but I’d move the CN cache to be first in your case.

I don’t think you can. The only option could possible help you is --delete-older-than.

Smart move :-). Preventing garbage collection might be possible with --add-root (https://nixos.wiki/wiki/Storage_optimization). I’ll have to read about that.

1 Like

@yalu I’ve created a status-react branch called chore/nix-gc-roots that adds a Makefile target to prevent garbage collection of status-react. Feel free to give it a try and then call nix-collect-garbage -d to see if it works (it does for me).

1 Like

i’ve changed CN cache to be first.
e.g.
substituters = https://nix-cache-cn.status.im/ https://nix-cache.status.im/ https://cache.nixos.org
i’m trying branch nix-cn-cache, it shows it will still access https://cache.nixos.org randomly which is the slow part. is this behavior correct?
1

failed at last.
1

I guess it would be best for you to remove the slow servers from your list. Even a single server should be enough and will probably end up being faster in your case.

@pedro Awesome!!! Works! :star_struck:
feels so fresh and so clean
Thank you!!!

3 Likes

Great to hear that! Merged to develop.

1 Like

Daily progress report:

  • Investigate work required to get to reproducible builds (e.g. one step is to add SHA values to existing canary build which always builds same historic commit, this way we’ll know when an unknown external factor - like a download - has caused the artifacts to change).
1 Like

Great news, I think! As for reproducible Android builds, with the Nix effort, it looks like the only difference right now between 2 apks of separate CI builds is in index.android.bundle and specifically in how the calls to the ClojureScript logging library (timbre) are transpiled. If we fix this one issue, we’ll have bitwise identical Android builds:

image

from what I see here https://github.com/ptaoussanis/timbre/blob/master/src/taoensso/timbre.cljx#L370, the changing parameters correspond to ?file and callsite-id. callsite-id is computed from ?file in https://github.com/ptaoussanis/timbre/blob/master/src/taoensso/timbre.cljx#L558 and the unstable filename seems to be the reason why it is also changing.

Update: found out where the temp path is being created (in Leiningen): https://github.com/technomancy/leiningen/blob/f7e1adad6ff5137d6ea56bc429c3b620c6f84128/leiningen-core/src/leiningen/core/eval.clj#L238

4 Likes

A good video by Sander van der Burg about replacing NPM with Nix, with an introduction to the Nix ecosystem at the beginning: https://www.youtube.com/watch?v=Rh-CSJL1Q-U

2 Likes

incredible job :slight_smile: great work @pedro

2 Likes