CPM rocks; these things could help (and how can I help)

[rpav]

I recently switched all my stuff from Conan to this. The original conan setup took a couple weeks or more; I moved over in under a day. It's great and now no one needs 3rd party deps and I love it.

A few things I've been doing and/or would like to see:

1. "Package repository;" I've been building my own in https://github.com/rpav/CpmPackages, but it'd be neat to see an official/community one too. This is neat because you can use it with essentially no further constructs, if you put it in a CMake path:

include(pkg/cxxopts)
include(pkg/nlohmann_json)
# etc

2. Going along with this, I noticed that there was a "single-arg" CPMAddPackage enhancement, but what would be neat is if there was something like:

cpm(pkg/cxxopts TAG xyz FOO bar ...)

I.e., a small function around include that pass along the entire arglist to be parsed as desired .. for more flexible "packages":

cmake_parse_arguments(mypkg_ ... ${CPM_PACKAGE_ARGS})

if(mypkg_FOO)
  # also depend on...
  cpm(...)
endif()

CPMAddPackage(
  NAME ..
  TAG ${mypkg_TAG}
  OPTIONS "... "
  )

I'll probably hack this up in my cmake utilities if nothing else, but I'd be happy to contrib.

3. I know there's ccache, but it'd be neat if, alongside source cache, CPM could also share builds. This would reduce the need for a big 3rd party dep and specialized configuration.
   • This is a pretty big feature request, but I have some interest in seeing it done and would definitely be willing to help.
   • It may also be less hard than it seems: both users and packages could supply a "build string" to be combined and hashed; rather than "analyze all of CMake and determine what is relevant," this reduces the problem to "tell me what's important, CPM only handles the build"
4. Some kind of out-of-the-box integration for non-cmake build systems.
   • It would reduce the annoyance of these and be pretty neat to be able to be able to say e.g. MESON_BUILD ... and have this call the appropriate things to build at build-time
   • It would not be necessary (perhaps "eventually neat") to deduce targets, libraries, headers, etc .. these could still be manually added in the package file. This would rather mostly be a bridge for compiler and options.

[TheLartians]

Hey @rpav, thanks for the feedback, I'm really happy CPM.cmake is useful for you! Also great ideas and suggestions! Some thoughts:

1. "Package repository;"

Imo that's a really nice idea and another step towards a more convenient PM solution (relevant: #99).

Instead of includes, how do you feel about using a single CMake file containing a bunch of CPMDeclarePackage calls, similar to a package lock file? This would allow adding your repo using CPM.cmake and using CPMGetPackage for actually fetching a dependency.

# use package repository 
# essentially contains a single CMakeLists.txt with a bunch of `CPMDeclarePackage` calls
CPMAddPacakge(
  GITHUB_REPOSITORY rpav/CpmPackages
  VERSION 1.0.0
)

# add a specific package
CPMGetPacakge(cxxopts)

Not exactly sure how good this would scale, but I think CMake should support thousands of CPMDeclarePackage calls without a noticeable performance impact.

If you would be interested in building and maintaining such a repo (or similar solution) I can imagine adding it to the cpm-cmake organisation as an "official" dependency registry!

2. what would be neat is if there was something like: cpm(pkg/cxxopts TAG xyz FOO bar ...)

This may be related to above, but I'm not completely sure I can follow. Could you elaborate / add a specific example?

3. it'd be neat if, alongside source cache, CPM could also share builds.

Would definitely be awesome if we could come up with an internal solution for this, however I'm unsure how to handle the vast possibilities of compiler targets, options, flags etc. without essentially reproducing what ccache is already doing. But if you do have any suggestions or even a working prototype I'd be happy to see if we could integrate it!

4. Some kind of out-of-the-box integration for non-cmake build systems

This would be really useful and also a great addition to the CMake ecosystem besides CPM.cmake. However due to the complex settings of Makefiles, Meson, etc and all possible configurations for CMake flags / toolchains / etc itself it seems to be a very hard problem to get working "out-of-the-box".

[rpav]

Re: 1-2

I'm not sure CPMDeclarePackage and CPMGetPackage do enough to support the "feature" cases. But adding a repository with a command would be slick!

Primarily the reason for splitting definitions up into "package" files is maintenance. If multiple people maintain multiple packages, they don't have to work on modifying an ever-expanding giant file, and they don't have to be concerned about variables etc stepping on each other(*). Commits would be easier to merge, etc etc.

Using packages in a file like the above would be pretty great; I haven't played with the package lock feature yet as most of the things I deal with are repository head and not versioned. (But not all!)

Package definitions however are basically "I'd rather not type this twice or figure it out if someone else has," and also they're getting more complex, to segue into 2.

Some of my libraries have optional dependencies. E.g., I have a "VM-based vector graphics format" that will optionally want to pull one or more backends. It would be nice if the package definition could have something like:

# Let's call this pkg/jga.cmake:

cmake_parse_arguments(jga "" "" BACKENDS 
  ${CPM_PACKAGE_ARGN} # <- cpm function sets this
  )

if("nanovg" IN_LIST BACKENDS)
  cpm(nanovg ...)  # NOTE the optional package dependency
  set(jga_backend_opts "${jga_backend_opts} JGA_BACKEND_NANOVG")
endif()

if("blend2" IN_LIST BACKENDS)
 ...
endif()

CPMAddPackage(
  NAME jga
  ...
  OPTIONS "${jga_backend_opts} ..."
  )

The definition for cpm is trivial:

function(cpm PACKAGE)
  set(CPM_PACKAGE_ARGN ${ARGN})
  include(${PACKAGE})  
endfunction()

((*) Scope separation actually requires the include(..) be in a function. Two birds!)

The usage now gives easy options to users. The package file can pull in relevant dependencies, set the verbose options, do any checks, etc. (This plays into 3 and 4, and some packages/options may not be OPTIONS at all, may be passed to different build systems, download different binaries, what have you.)

# In someone's CMakeLists.txt, or a separate file
# From what I understand this would be used to generate the package lock

cpm(jga BACKENDS nanovg blend2d)

[TheLartians]

Gotcha, that does sound quite interesting and a nice way to CPM-ify badly behaved packages. On second thought though, maintaining a monorepo that contains all optimised dependencies could become increasingly difficult and make it hard for users to upgrade specific dependency versions without touching up others.

A better approach might be to create separate repos for complicated dependencies and list them under the CPM-ified organisation as a namespace. Note that for automatic updating / using a newer version of a dependency we are also currently working out some ideas.

That said, nothing speaks against creating an external project that supplies a command similar to the one described that users could opt-in to. This would have the additional advantage of providing a namespace for dependencies from a specific registry.

Using the new single-argument syntax, usage could be as simple as

CPMAddPackage("gh:rpav/rpav-cpm-registery")
RPAVAddpackage(jga BACKENDS nanovg blend2d)

[rpav]

Re: 3

I'll try and hack up a working prototype. My idea is to cheat mightily and depend on packages (and optionally the user) specifying sufficient disambiguation. For example:

# Using the JGA example above
...

CPMAddPackage(
  NAME jga
  OPTIONS "${jga_backend_opts} ..."
  BUILD_STRING "${jga_backend_opts} ${maybe_other_things}")
...

By default, a build ID might be sufficiently established by the following:

# Default user string; this should be sufficient unless the user knows they're doing something weird
set(CPM_USER_BUILD_STRING "${CMAKE_C_COMPILER_ID}-${CMAKE_BUILD_TYPE}-${CMAKE_SYSTEM_NAME}-${CMAKE_SYSTEM_PROCESSOR}")
string(SHA256 CPM_PACKAGE_BUILD_ID "${CPM_USER_BUILD_STRING}${CPM_PACKAGE_BUILD_STRING}")

# We have something like:

# CPM_USER_BUILD_STRING: Clang-Debug-Linux-x86_64

# nanovg backend ID:
# 70b783e8511fa7da508ede21a97fdbb23bdae20f8ae90bf6c584862c9e2ecbeb

# no option ID:
# 14f6be65836750cadee3047ee5370fe13105a1c91e5731b55d735ccde7829175

# etc

Of course, this should be opt-in just as (but separate from) source cache, and automatically warned/disabled if CMAKE_CROSSCOMPILING is set, or if on Apple, apparently. ;) (At least, until someone adds the additional checks!)

I'll see if I can hack this up sometime in the next few days and provide an example... if it works!

[rpav]

Re: 4

Yeah.

My thought is more "cheating" again and some utilities to potentially help. No massive complicated magic. Much of this would still be on the package to define, but some utilities to help might not be too far out. E.g.,

CPMAddPackage(NAME foo ... DOWNLOAD_ONLY)

if(foo_ADDED)
  cpm_meson_build(
    ${foo_SOURCE_DIR}
    TARGETS libfoo1 somefoobar ...
    ... # various specific options to meson; I'll have to try to hack this up too
  )
  add_library(libfoo1 ...)  # or whatnot
endif()

I'm pretty sure cpm_meson_build could at a minimum transfer CMAKE_C*_COMPILER, CMAKE_C*_FLAGS, etc. Autotools might work similarly, as it has a well-defined interface for those things. Of course, raw make would be very dependent on the package, but this is a "package maintainer" job.

I'll see if this is viable for at least one thing and hack up an example.

[TheLartians]

I definitely love the idea of having a tool to add non-cmake projects into CMake! I would suggest implementing it as a separate project though, that can be easily added using CPM.cmake. This would have the advantage of allowing independent development and versioning of the tool.

[TheLartians]

Moved to discussion for less comment fragmentation 😄

[alandefreitas]

You might be interested in #221

😀

[TheLartians]

#215 as well :)

[rpav]

Sorry I've been hammered the past couple weeks and haven't had a chance to swing back around to this.