Bug: GIT_SUBMODULES_RECURSE and GIT_SUBMODULES are ignored

[ethindp]

I have a CMake project that has private submodules. The project's CMake configuration process will acquire the public submodules that I need. Therefore, I've used the following git-related options:

    GIT_SUBMODULES ""
    GIT_SUBMODULES_RECURSE FALSE

According to the CMake documentation, with this set, FetchContent_Declare (and, by extension, ExternalProject_Add) will not clone submodules, as long as CMP0096 is set to NEW. So I did that at the top of my project:

cmake_policy(VERSION 3.25)
cmake_policy(SET CMP0096 NEW)

However, the arguments I set in CPMAddPackage (GIT_SUBMODULES and GIT_SUBMODULES_RECURSE) are ignored, and all submodules are cloned anyway. Is this a bug in CPM? Logically this should work perfectly fine, but it isn't for some reason.

[ethindp]

Okay, so it looks like CPMAddPackage doesn't actually take all possible content options. CPM should probably be updated to take into account all options that ExternalProject/FetchContent accepts.

[TheLartians]

CPM should already forward all additional arguments to FetchContent_Declare, so hard to say what's not working here. Can you confirm that CMake behaves as intended using the same arguments passed directly to FetchContent?

[ethindp]

@TheLartians Yes, can confirm. When I replace the CPMAddPackage line with FetchContent_Declare, it works fine. I do have to set CMP0096 to NEW, but other than that I get the expected behavior.

The reason, as I discovered, is that CPMAddPackage does not in fact forward all arguments to FetchContent_Declare. It only extracts the NAME, FORCE, VERSION, GIT_TAG, DOWNLOAD_ONLY, GITHUB_REPOSITORY, GITLAB_REPOSITORY, BITBUCKET_REPOSITORY, GIT_REPOSITORY, SOURCE_DIR, DOWNLOAD_COMMAND, FIND_PACKAGE_ARGUMENTS, NO_CACHE, SYSTEM, GIT_SHALLOW, EXCLUDE_FROM_ALL, SOURCE_SUBDIR, URL, and OPTIONS arguments, but FetchContent_Declare(and by extension,ExternalProject_Add) supports many more arguments than these. At least, that's my understanding of the code. I'm not positive how cmake_parse_arguments works, so for all I know I could be wrong.

[TheLartians]

In fact from my understanding it should capture all arguments. From the CMake docs:

All remaining arguments are collected in a variable _UNPARSED_ARGUMENTS that will be undefined if all arguments were recognized.

Which we should later forward to FetchContent. So I'm unsure why it isn't working in this case. Can you share a minimal example to reproduce?

[ethindp]

@TheLartians Sure thing. I'm trying to use a project that has some (private) submodules. So I can't just recursively fetch everything; I don't have rights to all of the submodules. The CMake project, therefore, fetches the modules that I actually need, and excludes those that are private, unless I explicitly opt into it doing so. So, I tried this using CPM:

CPMAddPackage(
    NAME nabla
    VERSION 0.4-beta
    GIT_REPOSITORY https://github.com/devsh-graphics-programming/nabla
    GIT_TAG v0.4-beta
    GIT_SUBMODULES ""
    GIT_SUBMODULES_RECURSE FALSE
    GIT_PROGRESS TRUE
    OPTIONS
    "NBL_COMPILE_WITH_CUDA ON"
    "NBL_BUILD_DPL ON"
    "NBL_PCH ON"
    "NBL_FAST_MATH ON"
    "NBL_BUILD_EXAMPLES OFF"
    "NBL_BUILD_TOOLS OFF"
    "NBL_BUILD_MITSUBA_LOADER ON"
    "NBL_BUILD_OPTIX ON"
    "NBL_EMBED_BUILTIN_RESOURCES ON"
)

Doing that still caused Git to fetch all submodules, even those that were private. Which is not my desired behavior. A minimal example for a project might be something like:

cmake_minimum_required(VERSION 3.25)
cmake_policy(VERSION 3.25)
cmake_policy(SET CMP0096 NEW)
cmake_policy(SET CMP0077 NEW)
project(game VERSION 0.0.1 LANGUAGES CXX)
include(CheckPIESupported)
check_pie_supported()
if(NOT CMAKE_MAXIMUM_RECURSION_DEPTH)
  set(CMAKE_MAXIMUM_RECURSION_DEPTH 2000)
endif()
add_executable(game
    main.cpp
)
# Fetch external targets
set(CPM_DOWNLOAD_VERSION 0.38.1)

if(CPM_SOURCE_CACHE)
    set(CPM_DOWNLOAD_LOCATION "${CPM_SOURCE_CACHE}/cpm/CPM_${CPM_DOWNLOAD_VERSION}.cmake")
elseif(DEFINED ENV{CPM_SOURCE_CACHE})
    set(CPM_DOWNLOAD_LOCATION "$ENV{CPM_SOURCE_CACHE}/cpm/CPM_${CPM_DOWNLOAD_VERSION}.cmake")
else()
    set(CPM_DOWNLOAD_LOCATION "${CMAKE_BINARY_DIR}/cmake/CPM_${CPM_DOWNLOAD_VERSION}.cmake")
endif()

if(NOT (EXISTS ${CPM_DOWNLOAD_LOCATION}))
    message(STATUS "Downloading CPM.cmake to ${CPM_DOWNLOAD_LOCATION}")
    file(DOWNLOAD
        https://github.com/cpm-cmake/CPM.cmake/releases/download/v${CPM_DOWNLOAD_VERSION}/CPM.cmake
        ${CPM_DOWNLOAD_LOCATION}
    )
endif()

include(${CPM_DOWNLOAD_LOCATION})

CPMAddPackage(
    NAME nabla
    VERSION 0.4-beta
    GIT_REPOSITORY https://github.com/devsh-graphics-programming/nabla
    GIT_TAG v0.4-beta
    GIT_SUBMODULES ""
    GIT_SUBMODULES_RECURSE FALSE
    GIT_PROGRESS TRUE
    OPTIONS
    "NBL_COMPILE_WITH_CUDA ON"
    "NBL_BUILD_DPL ON"
    "NBL_PCH ON"
    "NBL_FAST_MATH ON"
    "NBL_BUILD_EXAMPLES OFF"
    "NBL_BUILD_TOOLS OFF"
    "NBL_BUILD_MITSUBA_LOADER ON"
    "NBL_BUILD_OPTIX ON"
    "NBL_EMBED_BUILTIN_RESOURCES ON"
)
include(${CMAKE_CURRENT_BINARY_DIR}/_deps/nabla-src/cmake/build/AddNablaModule.cmake)
addNablaModule(game "${CMAKE_CURRENT_BINARY_DIR}/_deps/nabla-build/install")
target_compile_features(game PUBLIC cxx_std_20)
set_target_properties(game PROPERTIES
    CXX_EXTENSIONS OFF
    POSITION_INDEPENDENT_CODE TRUE
    C_STANDARD 18
    C_STANDARD_REQUIRED 18
    CXX_SCAN_FOR_MODULES ON
    CXX_STANDARD 20
    CXX_STANDARD_REQUIRED 20
    INTERPROCEDURAL_OPTIMIZATION ON
   PCH_INSTANTIATE_TEMPLATES ON
    PCH_WARN_INVALID ON
    UNITY_BUILD ON
    UNITY_BUILD_UNIQUE_ID "GAME"
)

(I might've made some errors, I hope not.) The source code in main.cpp can be anything, since CMake fails the configure stage.

[ethindp]

@TheLartians I should also note that you'll need the Cuda toolkit and NVIDIA OptiX for this to work. You can turn that off if you don't have them or you don't want to go get them just for this. The configure stage will take quite a while anyway, though.

[TheLartians]

Super, thanks for the reproducer!
Now that I see it I believe this is related an issue currently being worked on where empty string parameters (eg the "" in GIT_SUBMODULES "") get lost in the CMake parsing toolchain.

Hopefully the fix will be released soon, but in the meanwhile could you try replacing the contents of CPM.cmake with this development version and see if that solves your issue?

[CraigHutchinson]

@ethindp I feel this bug was prematurely closed, the Fix PR hasn't made progress so this bug should be open so its visible to others.