Skip to content
This repository has been archived by the owner on Jan 25, 2023. It is now read-only.

Cache dependencies for clojure projects using Tools.deps.alpha in addition to Leiningen and boot #485

Open
armincerf opened this issue Oct 27, 2020 · 11 comments · May be fixed by #831
Open

Cache dependencies for clojure projects using Tools.deps.alpha in addition to Leiningen and boot #485

armincerf opened this issue Oct 27, 2020 · 11 comments · May be fixed by #831

Comments

@armincerf
Copy link

Currently Netlify will only cache dependencies in Clojure projects if there is a valid project.clj or build.boot file, which are used by the Leiningen and Boot build tools.

However Clojure now has an official API for dependency management and uses a deps.edn file rather than the project.clj or build.boot files used in older build tools.

It makes sense to add support for deps.edn files as this is the official way to manage dependencies in Clojure now.

Relevant PR https://github.com/netlify/build-image/pull/117/files
@armincerf armincerf changed the title Cache dependencies for clojure projects using Tools.deps.alpha instead of Leiningen Cache dependencies for clojure projects using Tools.deps.alpha in addition to Leiningen and boot Oct 27, 2020
@kennyjwilli
Copy link

Are there any known workarounds for the time being?

@armincerf
Copy link
Author

The only one I can think off is maintaining a project.clj as well, but that's a bit of a pain

@armincerf
Copy link
Author

Actually just looked at the code and seems like it doesn't actually matter what's in the project.clj file, just that it exists. I'll test with an empty project.clj to make sure tomorrow but if that works then it will be easy to check for project.clj or deps.edn as they will both be cached in the same way.

I will make a PR when I get around to it if that's the case.

@kennyjwilli
Copy link

I think that will work well for most cases. Our build currently eats up ~2.5 mins of downloading deps on every build haha.

The one exception is that deps.edn allows for Git deps which are stored in ~/.gitlibs (by default. See docs).

@minikomi
Copy link

A completely empty (touch project.clj) project.clj causes lein to try and run it, which results in an error.

@mainej mainej mentioned this issue Apr 29, 2021
Closed
@mainej
Copy link

mainej commented Apr 29, 2021

I believe this would be fixed by #553.

@kennyjwilli
Copy link

Hi @mainej. Thanks for working on that PR. The Clojure CLI supports Git Deps. I'd suggest adding the ~/.gitlibs directory to the cached set of dirs (see docs).

@mainej
Copy link

mainej commented Apr 29, 2021

@kennyjwilli Good suggestion. 👍 I've added caching of ~/.gitlibs to the PR and verified that it works by using the instructions for testing builds with the cache.

@barrelltech
Copy link

barrelltech commented Feb 18, 2022 •
edited
Loading

What is the status of this? I would like to use netlify but am building a project with the clojure cli

Nevermind, it's been merged, there was just an issue in my build script 😂 I reckon this issue can be closed! :)

@mainej
Copy link

mainej commented Feb 22, 2022

@bbarrell this issue shouldn't be closed. It isn't fixed.

As I believe you discovered, Netlify does install the clojure cli for building deps.edn based projects (though see caveats below).

However, 485 was opened to address a different problem—build times of Clojure projects are unnecessarily slow, because ~/.m2 and other directories aren't cached at the end of a build.

(Note... what follows is addressed to Netlify, not you.)

Because of this and other bugs, it can be hard to build Clojure projects on Netlify. Netlify isn't paying attention to these issues, nor have they offered guidance or workarounds. It's a shame because I would really like Netlify to be an easy place to deploy Clojure projects. As you can see below, I and several other people have tried reporting issues, creating PRs, or looking for other solutions, but we've gotten very little response.

By my accounting there are two important outstanding bugs related to Clojure:

  1. PR 484: the default clojure cli is very outdated (Netlify's default version of the CLI was built Nov 25, 2019). Other tooling, including shadow-cljs expects newer versions of the CLI.
  2. Issue 485, this issue: builds do not cache dependencies. Would be fixed by PR 553.

I tried to write a "build plugin"—Netlify's mechanism for extending the build process—to work around the caching issue, but was never able to get ~/.m2 cached. I filed an issue about that, but never got any response from Netlify. Discouraged by that experiment, I've never tried writing a build plugin that would install an updated version of the Clojure CLI. I doubt it would work, because I think that install needs sudo, which you don't have in the plugins, AFAIK. The build plugin infrastructure is cool, but it doesn't address some of the fundamental needs that language communities need to build their projects.

Anyway, if you've gotten your project to build @bbarrell, nice! It is possible. Cross your fingers that it keeps working!

mainej added a commit to mainej/build-image that referenced this issue Aug 26, 2022
@mainej
Cache clojure dependencies
6b09a45 
This speeds up builds that use the [Clojure CLI][clojure-cli] by
avoiding network calls to download dependencies.

When a repo has a `deps.edn` file, we assume it is a Clojure repo. We
cache its dependencies by caching the following files:

* `.m2` As a JVM language, most Clojure dependencies are stored in the
  global `.m2` directory.
* `.gitlibs` The Clojure CLI can also download dependencies directly
  from git repositories rather than from a package manager. It caches
  these downloads in the `.gitlibs` directory.
* `.cpcache` Clojure uses the `.cpcache` directory to avoid
  unnecessarily re-calculating the JVM classpath. Caching this directory
  won't prevent downloads, but does improve build time slightly.

Fixes netlify#485.

[clojure-cli]: https://clojure.org/reference/deps_and_cli
@mainej mainej linked a pull request Aug 26, 2022 that will close this issue
Open
3 tasks
mainej added a commit to mainej/build-image that referenced this issue Aug 26, 2022
@mainej
fix: Cache clojure dependencies
8c569dc 
Fixes netlify#485.

This speeds up builds that use the [Clojure CLI][clojure-cli] by
avoiding network calls to download dependencies.

When a repo has a `deps.edn` file, we assume it is a Clojure repo. We
cache its dependencies by caching the following files:

* `.m2` As a JVM language, most Clojure dependencies are stored in the
  global `.m2` directory.
* `.gitlibs` The Clojure CLI can also download dependencies directly
  from git repositories rather than from a package manager. It caches
  these downloads in the `.gitlibs` directory.
* `.cpcache` Clojure uses the `.cpcache` directory to avoid
  unnecessarily re-calculating the JVM classpath. Caching this directory
  won't prevent downloads, but does improve build time slightly.

[clojure-cli]: https://clojure.org/reference/deps_and_cli
@mainej
Copy link

mainej commented Aug 26, 2022

The caching fix in #553 was closed because it wasn't for focal, the current version of Ubuntu targeted by Netlify. I've re-opened a simplified version of #553 as #831, based off the focal branch.

#484 has also been closed, for the same reason. It could technically be re-opened against focal, but the workaround is another option for getting an updated Clojure.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(caching): Cache clojure dependencies mainej/build-image
5 participants
@mainej @minikomi @kennyjwilli @armincerf @barrelltech