package-lock and dealing with conditional dependencies

[itavero]

Currently working on integrating CPM.cmake in one of our projects.
Although I don't really need the package-lock.cmake for development (yet), it would be useful as a key for the GitLab CI cache.

Unfortunately, one of the things I've noticed, is that packages for which CPMAddPackage is no longer called, get removed from the file.
On one hand this makes sense, how else would you clean up the file.
On the other hand, there's conditional dependencies in our projects. When the project is build for the host, the test framework is included (via CPMFindPackage). When the project is build for an embedded target (i.e. ARM Cortex M33), it is not included as the tests can not be ran on that specific target (or at least, we do not have an interface to retrieve the results, so it would not make sense).

I thought about including a unique ID for the different environments to the package-lock.cmake filename (i.e. CPMUsePackageLock(package-lock.${CMAKE_SYSTEM}.cmake)), but then we would just end up with multiple files that should largely have the same dependencies in there. I think this would cause unwanted duplication and might make it harder to maintain.

I'm not yet sure what the right solution would be, but some ideas came to mind that might be feasible:

• Have a function that can be called if the package is not used but should be kept in the package-lock (preferably with just the NAME, as otherwise the version information would have to be duplicated)
• Somehow make CPM.cmake aware of the fact that a package is conditional, by passing something additional in CPMFindPackage or CPMAddPackage

Of course, if there is already a known way of dealing with this, I'd love to hear about that.

[itavero]

The first idea I had, can already be done by adding something like:

 # Helper code to keep CppUTest in the package lock
 if (DEFINED CPM_DECLARATION_CppUTest)
    cpm_add_to_package_lock(CppUTest ${CPM_DECLARATION_CppUTest})
 endif()

[itavero]

On a related note, I think some kind of sorting should probably be applied when cpm-update-package-lock is called, so that the lock file contents doesn't change if the same dependencies are included in a different order.

Since this file should be in version control, it should preferably only change if it actual gets different information in it.
This also makes it an even better candidate for deriving a cache key from it (as I'm planning to do in GitLab CI).

[TheLartians]

Agreed.

[itavero]

Maybe something that should be considered in #221, if not being done already.

[itavero]

One way this could be done is by using something like a regular expression to get all CPMDeclarePackage calls from the file and put them in a list which we can sort using CMake itself.
But we would have to read the existing file every time we want to add something.

I don't see an easy way of dealing with the sorting when the cpm-update-package-lock task is executed.

[TheLartians]

Yeah that's a good point. Without giving it too much thought I think that adding something like a SKIP or INCLUDE_IF flag to CPMAddPackage, which can disable fetching but still updates the package lock would make a lot of sense.
Not sure which would be the better option though.

[itavero]

The question is however, is this also put in the package-lock file somehow?
I think because of the scenario described below, this might be necessary, but I don't know if it would be possible.

I personally don't have a use-case for it (yet?), but what should CPM.cmake do if I have something like

 CPMFindPackage(
    NAME CppUTest
    VERSION 4.0
    GITHUB_REPOSITORY "cpputest/cpputest"
    INCLUDE_IF MYPROJECT_ENABLE_TESTS
 )
 CPMFindPackage(
    NAME CppUTest
    VERSION 3.8
    GITHUB_REPOSITORY "cpputest/cpputest"
    INCLUDE_IF MYPROJECT_ENABLE_LEGACY_TESTS
 )

Of course in this case, the project should ensure that MYPROJECT_ENABLE_TESTS and MYPROJECT_ENABLE_LEGACY_TESTS are not ON at the same time.
We could state this is not supported, but I can already see someone wanting/trying to do something like this.

[OlivierLDff]

Wouldn't it just work like just having multiple CPMAddPackage ? Only VERSION 4.0 would be included.

[itavero]

Wouldn't it just work like just having multiple CPMAddPackage ? Only VERSION 4.0 would be included.

I'm fine with that, but that does mean that the INCLUDE_IF can only be used to switch between including a specific dependency or not including it. Not for including different version of a dependency, based on some variable. When implementing such a feature, that should be made clear in the documentation.

[itavero]

I'll try to add support for the INCLUDE_IF option and create a PR afterwards. Is there any documentation available on how to add tests and run them?

[itavero]

I made a (rough) implementation in #394 (not yet finished). I did a quick test in one of my projects and this at least allowed me to keep the definition in the package lock, even if the package was not actually included (INCLUDE_IF evaluated to false).

[iboB]

The proposed solutions would only work if all packages are listed in a central CMakeLists, and if this was the case, there would not be a point in having a package-lock. This central CMakeLists can play the same role.

But packages are listed in multiple CMakeLists files. My root can have code like:

if(BUILD_SERVER)
    add_subdirectory(server)
endif()
if(BUILD_CLIENT)
    add_subdirectory(client)
endif()

In such case the dependencies of the client won't even be mentioned, if only BUILD_SERVER is true and vice versa. And yes, as @itavero mentioned, different build flavors may require different versions of the same package, leading to an adverse package lock.

Thinking about all edge cases and different scenarios I'm even inclined to propose ditching package-lock altogether. Has anyone made a good use of it?

[TheLartians]

That's a really good point, package-lock definitely isn't compatible with conditional includes. Also to properly update the lock for nested dependencies we would need to somehow query any disabled packages as well making this feature much more difficult to implement than I initially anticipated.

I'm still in favour of keeping the feature, as having a package lock is crucial if we want to support automatic package updates (e.g. #318, #389), but we should make it clear that it doesn't work with conditional includes. As for INCLUDE_IF, it is still possible but would require running a separate temporary CMake process to resolve any nested dependencies when updating the package lock.

Going forward I do think the feature needs a hefty update and be used together with dynamic version tags as suggested in #318.