Experimenting with building a Go binary

[vrischmann]

Hello,

I saw your project some time ago on HN and it looked interesting, thank your for making it ! I have no experience with Nix but I like the idea, however I never tried it because of the Nix language so the fact that you're using Lua is great in my opinion.

Anyway, I started experimenting with writing a derivation to build a go binary from a local module, just for fun.
I have this right now after a lot of trial and error:

local function goBuild(args)
  local actualArgs = {
    name = args.name;
    ["in"] = args.input;
    builder = "/bin/sh";
    system = "x86_64-linux";
  }

  local script = "\z
      export GOCACHE=$(pwd)/cache\n\z

      /usr/bin/go build -C $in -o $out "..args.name.."\n"

  actualArgs.args = { "-c", script }

  return derivation(actualArgs)
end

return goBuild {
  name = "foo";
  input = path "foo";
}

This works but I have some questions:

First, is it expected to write a shell script that does the actual call to go build ? At first I used builder = "/usr/bin/go" and expected to be able to put my files at the right place somehow, but looking at bootstrap.lua it seems like shell scripts are the way to go.
I'm also confused how and where $in and $out get replaced, because it seems like I have to use a shell to get their value: for example I tried command = "build -o $out $in" with the go builder above and the command gets the raw strings $out and $in, not their values.

Second, I had to explicitly set GOCACHE because $HOME is not accessible, which makes sense, but I'm wondering if there is (or will be) a way to have a cache for go builds across multiple invocations of zb build ?

[zombiezen]

This warms my heart to receive this message. Thank you for trying out this little experiment! 💜

I have no experience with Nix but I like the idea, however I never tried it because of the Nix language so the fact that you're using Lua is great in my opinion.

I'm so very glad to hear that! That's exactly the sort of experience I'm hoping to provide.

I have this right now after a lot of trial and error:

Yup, this looks about right for an unsandboxed build. My goal with #30 is to have a hermetic C and Go compiler (and bash + standard Unix utilities) that can be dropped in without the user having to install anything.

And I really ought to write some docs. Tracking mostly in #7. It's very encouraging to hear you were able to figure this out even without that! Bravo.

First, is it expected to write a shell script that does the actual call to go build ?

Yes, having the builder be a shell script is pretty common in Nix land. Each derivation call runs a single program, where builder is argv[0], args is argv[1:], and the rest of the parameters are plumbed through as environment variables. One trick you can use for inputs is the toFile function: it will write its string argument to a file at build time and then you can use the path as an argument to bash.

At first I used builder = "/usr/bin/go" and expected to be able to put my files at the right place somehow, but looking at bootstrap.lua it seems like shell scripts are the way to go. I'm also confused how and where $in and $out get replaced, because it seems like I have to use a shell to get their value [...]

bash is performing the expansion. Hypothetically, as long as you have a builder that writes the output to the out environment variable, you're good, but in practice, scripts are the easiest way to do that.

Second, I had to explicitly set GOCACHE because $HOME is not accessible, which makes sense, but I'm wondering if there is (or will be) a way to have a cache for go builds across multiple invocations of zb build ?

Things like GOCACHE probably won't be supported generally, because they would make the builds not hermetic. In my experience with Nix, this is seldom an issue because it becomes easier to bootstrap a development shell with the exact same tools as are used in zb build, and you can run your build process with make or go build to make incremental builds of a single derivation faster.

[vrischmann]

Thanks for the answers.

One trick you can use for inputs is the toFile function

toFile looks interesting, I didn't quite get it before but looking at bootstrap.lua I think I do now.

Hypothetically, as long as you have a builder that writes the output to the out environment variable, you're good, but in practice, scripts are the easiest way to do that

oh, I didn't realize it was in an environment variable but I see it now.

In my experience with Nix, this is seldom an issue because it becomes easier to bootstrap a development shell with the exact same tools as are used in zb build, and you can run your build process with make or go build to make incremental builds of a single derivation faster.

so I think I understand this for a local development environment but this wont apply for a CI execution right ? While caching is not as dramatic with Go as other languages, at work we still get a significant speedup if we can reuse objects from previous compilations. Does Nix just eat the cost of not having a cache ?

[zombiezen]

zb/Nix caching is intentionally at the derivation level. If you can break apart your build into multiple derivations, those intermediate derivations will be cached. A common example is downloading the Go modules in one derivation (i.e. go mod download) and then using the result in the build.

[vrischmann]

I see. So in theory we could have one derivation per single go module which contains both its source code and its object files, we then take these derivations as input for the final "main" module and we could reconstruct the full GOCACHE from the derivations cached content ?

So for example, let's say I have a go program, that needs two modules, that would mean two derivations:

github.com/spf13/cobra v1.8.1
  src
    ...
  cache
    00
    aa
    ...
github.com/redis/go-redis/v9 v9.7.0
  src
    ...
  cache
    20
    bf
    ...  

Then in the final derivation for my program I could reconstruct the GOCACHE by merging the content of every cache in the modules derivations.
Is that basically the gist of it ?

Thanks for answering my questions by the way, I realize these may be generic questions about the Nix model rather than specific to zb.

[zombiezen]

The more typical approach is having a derivation per each compiler invocation (this is roughly what Bazel does, for instance), but that has a lot of overlap with what the go build tool already does, so it's a bit of a tradeoff.

Happy to answer! It's a great starting point for documentation, so thanks for the questions. (And apologies for the delayed response.)

[vrischmann]

I see, thanks for the insights.