build: migrate from Conan + vendored deps to vcpkg manifest mode#7487
build: migrate from Conan + vendored deps to vcpkg manifest mode#7487tete17 wants to merge 9 commits intoProject-OSRM:masterfrom
Conversation
tete17
force-pushed
the
build/migrate-to-vcpkg
branch
4 times, most recently
from
4131e76 to
43d6379
Compare
|
can only say kudos :) I've been on/off working with vcpkg over quite some time as well and in 2026 it's definitely worth migrating. so much better than conan for most scenarios. I didn't look: it's still optional right and fall back to system dependencies? in case the docker optimizations are more scattered around, maybe it'd make sense to do that as a separate PR? |
tete17
force-pushed
the
build/migrate-to-vcpkg
branch
5 times, most recently
from
c1aaf8d to
2b67c3f
Compare
|
does the switch have an impact on static linking of the binaries? |
|
Yes, mostly in a good direction. Linux / macOS ( Boost, TBB, zlib, bz2, expat, lzma, zstd, lua, libxml2, osmium, protozero, flatbuffers runtime, abseil — all statically linked into the binary. This is actually more static than before: pre-vcpkg, TBB came from the system as Windows ( |
|
Fully static binaries would be preferred. Tbb is a bit of a borderline case which could be static or dynamic. |
tete17
force-pushed
the
build/migrate-to-vcpkg
branch
from
2b67c3f to
077e7bc
Compare
|
Switched Windows to TBB stays static (vcpkg builds it static on all our triplets now). Force-pushed, CI will tell us if anything breaks on Windows with the static linkage. |
|
Gave the PR a first round of scrutiny. Generally speaking I think this looks good. I will post a couple of comments later on a number of smaller touch ups and changes. Looks like we should be able to get this merged before the next monthly release. |
Sorry @nilsnolde I let claude code anwser and your second paragraph slipped through. Yes indeed the vcpkg integration is always optional. By default it will try to pick up the system packages. You may need to tweak a bit the cmake presets but it should be no problem |
Introduce vcpkg manifest mode as the single dependency management strategy. The manifest pins all dependencies with a baseline commit and version overrides for sol2 (3.3.1), lua (5.4.8), and flatbuffers (25.9.23) to maintain compatibility with existing code. CMakePresets.json provides cross-platform build presets (release, debug, asan, CI variants for Linux/macOS/Windows) that wire up the vcpkg toolchain file automatically. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Replace the dual ENABLE_CONAN/system-packages dependency resolution with a single vcpkg-based path. All find_package() calls now consume vcpkg CONFIG-mode packages or standard CMake modules. Key changes: - Remove ENABLE_CONAN option and entire if/else dependency block - Remove flatbuffers add_subdirectory (now a vcpkg package) - Remove vendored include_directories for libs moved to vcpkg - Add cmake/FindOsmium.cmake (extracted from third_party/libosmium) with patched protozero detection for vcpkg's header-only port - Wire up modern imported targets (Boost::*, TBB::tbb, etc.) - Harvest INTERFACE_INCLUDE_DIRECTORIES from header-only targets for legacy code that reads Boost/TBB include path variables - Update Node.js CMakeLists.txt to use TBB::tbb imported target Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Delete third_party/{flatbuffers,fmt,libosmium,protozero,rapidjson,sol2}
— all are now consumed as vcpkg packages. The remaining third_party/
contents (microtar, vtzero) stay vendored as they have no vcpkg port.
Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Delete conanfile.py — Conan is fully replaced by vcpkg manifest mode. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
- Replace all Conan install steps with lukka/run-vcpkg@v11 - Remove manual Boost, TBB, and system library install steps - Add vcpkg binary cache via GitHub Actions cache backend - Rename matrix entries from conan-* to vcpkg-* - Add vcpkg-smoke.yml for non-gating manifest resolution check - Update windows-build.bat to use cmake --preset ci-windows - Update UBSan suppressions for vcpkg include paths Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Replace apt-get/apk system dependency installation with vcpkg bootstrap in the builder stage. Key optimizations: - Split vcpkg install into its own layer (cached when manifest unchanged, source-only changes skip the expensive dep build) - BuildKit cache mounts for vcpkg archives, downloads, and buildtrees persist across docker build invocations on the host - Switch from Make to Ninja for lower scheduling overhead - Add ccache with a persistent cache mount for C++ compilation, dramatically speeding up rebuilds with small source changes - Keep VCPKG_ROOT=/vcpkg (outside /opt) so the runstage COPY doesn't drag the ~5 GB vcpkg tree into the final image Image size remains ~331 MB (debian) with only TBB shared libs copied to the runstage. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Replace Conan and system-package build instructions with vcpkg workflow in README and Windows dependency documentation. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Two CI fixes: 1. sol2 3.3.1 has noexcept specifiers on call() templates whose addresses are taken as lua_CFunction (int (*)(lua_State*)). On ARM64 clang-18 this causes "address of overloaded function does not match required type" errors. Add a vcpkg overlay port that patches out the noexcept and fixes broken operator() calls that referenced the template without explicit arguments. 2. file(GENERATE) for libosrm.pc fails on multi-config generators (MSVC) because $<TARGET_LINKER_FILE:...> produces different values per config. Skip pkgconfig generation for multi-config generators since it is not used on Windows. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
Two separate caching gaps were making every CI run rebuild all 113 ports from scratch: 1. The x-gha binary caching backend has been removed in the vcpkg version we pin (c3867e714). VCPKG_BINARY_SOURCES=clear;x-gha,readwrite was a no-op, vcpkg printed a warning and built everything from source. Switch to the files backend pointing at a workspace directory, and persist that directory across runs with actions/cache@v5. Cache key includes hashes of vcpkg.json, vcpkg-configuration.json, and the overlay ports tree so port changes invalidate. 2. Docker builds used BuildKit --mount=type=cache, which only persists on the same runner — on ephemeral GHA runners that means no cache hits. Replace raw 'docker build' with docker/build-push-action@v6 + buildx configured with cache-to/cache-from=type=gha. Scoped per base image so debian and alpine don't collide. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
tete17
force-pushed
the
build/migrate-to-vcpkg
branch
from
077e7bc to
06a10df
Compare
DennisOSRM
left a comment
DennisOSRM
left a comment
There was a problem hiding this comment.
left a first round of comments. I think this is already pretty close to being mergeable. It seems the major remaining piece is back porting the lua5.5 compatibility patch. Pinning old versions of sol2 and lua seems subpar. The other comments should be pretty easy to fix.
All in all, good work! We are close.
| CCACHE_COMPRESS: 1 | ||
| CASHER_TIME_OUT: 599 # one second less than 10m to avoid 10m timeout error: https://github.com/Project-OSRM/osrm-backend/issues/2742 | ||
| CMAKE_VERSION: 3.21.2 | ||
| CMAKE_VERSION: 3.25.0 |
There was a problem hiding this comment.
consider bumping the version to 3.31 which comes with Debian trixie
| } | ||
|
|
||
| if (option_variables.contains("version")) | ||
| if ((option_variables.count("version") > 0)) |
There was a problem hiding this comment.
the changes in this file should be reverted. This seems unrelated to the vcpkg migration
| return return_code::fail; | ||
| } | ||
|
|
||
| if (option_variables.contains("version")) |
There was a problem hiding this comment.
the changes in this file should be reverted. This seems unrelated to the vcpkg migration
| } | ||
|
|
||
| if (option_variables.contains("version")) | ||
| if ((option_variables.count("version") > 0)) |
There was a problem hiding this comment.
the changes in this file should be reverted. This seems unrelated to the vcpkg migration
| } | ||
|
|
||
| if (option_variables.contains("version")) | ||
| if ((option_variables.count("version") > 0)) |
There was a problem hiding this comment.
the changes in this file should be reverted. This seems unrelated to the vcpkg migration
| } | ||
|
|
||
| if (option_variables.contains("version")) | ||
| if ((option_variables.count("version") > 0)) |
There was a problem hiding this comment.
the changes in this file should be reverted. This seems unrelated to the vcpkg migration
| @@ -0,0 +1,122 @@ | |||
| --- a/include/sol/function_types_stateless.hpp | |||
There was a problem hiding this comment.
checkout the stopgap patch for lua5.5 compatibility that we used before: #7404. Using this patch should allow use to use the most recent release of sol2 and lua.
| message(STATUS "Using Lua ${LUA_VERSION_STRING}") | ||
| # All dependencies below come from vcpkg (vcpkg.json). The vcpkg toolchain | ||
| # file exposes them via standard CMake find_package() / imported targets. | ||
| find_package(Boost 1.70 REQUIRED CONFIG COMPONENTS ${BOOST_COMPONENTS}) |
There was a problem hiding this comment.
we should bump the minimum boost version to 1.83 (trixie)
| "description": "Open Source Routing Machine - high performance routing engine", | ||
| "homepage": "https://github.com/Project-OSRM/osrm-backend", | ||
| "dependencies": [ | ||
| "boost-algorithm", |
There was a problem hiding this comment.
can we narrow this list down? The used libs are:
set(Boost_DATE_TIME_LIBRARY Boost::date_time)
set(Boost_IOSTREAMS_LIBRARY Boost::iostreams)
set(Boost_THREAD_LIBRARY Boost::thread)
set(Boost_REGEX_LIBRARY Boost::regex)
set(Boost_PROGRAM_OPTIONS_LIBRARY Boost::program_options)
set(Boost_UNIT_TEST_FRAMEWORK_LIBRARY Boost::unit_test_framework)
| "flatbuffers", | ||
| "libosmium" | ||
| ], | ||
| "overrides": [ |
There was a problem hiding this comment.
see the patch for sol2 above. this should let us get rid of the overrides.
3 participants
Summary
Migrate osrm-backend's dependency management from the current three-path system (Conan 2.x + vendored
third_party/sources + system packages) to a single vcpkg manifest-mode setup. One dependency path on every platform, vendored copies deleted, CMakeLists simplified.Why vcpkg over the status quo
The current setup has three parallel dependency paths that have diverged over time:
ENABLE_CONANgating a separate CMakeLists branchapt-get,apk) for Linux CI and Docker, with custom Find modulesthird_party/(flatbuffers, fmt, sol2, rapidjson, protozero, libosmium) as a fallbackThis creates maintenance burden: dependency version bumps require touching 3+ places, the
if(ENABLE_CONAN)/else()block in CMakeLists.txt has subtle differences (e.g., Lua 5.2 vs 5.4), and the vendored copies are perpetually stale.vcpkg manifest mode collapses this into:
vcpkg.json— single source of truth for all dependency versionsvcpkg-configuration.json— pinned baseline for reproducibilityCMakePresets.json— cross-platform build configurationsfind_package()path in CMakeLists.txt — no moreENABLE_CONANbranchingWhy vcpkg specifically
x-gha,readwrite) — CI doesn't rebuild boost from source on every runfind_package(), no wrapper scripts neededWhat changes
build: add vcpkg manifest...vcpkg.json+vcpkg-configuration.json+CMakePresets.jsonbuild: rewrite CMakeLists.txt...ENABLE_CONAN, addcmake/FindOsmium.cmakebuild: remove vendored third_party...third_party/{flatbuffers,fmt,libosmium,protozero,rapidjson,sol2}(−360k lines)build: remove Conanconanfile.pyci: migrate CI workflows...lukka/run-vcpkg@v11, addvcpkg-smoke.ymlbuild(docker): rewrite Dockerfiles...docs: update build instructionsVersion pins (overrides in vcpkg.json)
usertype_containerwithunordered_map<string,bool>LUA_ERRGCMM, incompatible with sol2 3.3.1generated/are pinned to this versionDocker build optimizations
COPY vcpkg.json→vcpkg install→COPY . /srcso source-only changes skip the expensive dep layerWhat stays vendored
third_party/microtar/— single .c OBJECT library, no vcpkg portthird_party/vtzero/— not in the vcpkg baseline usedTest plan
--helpsmoke test)osrm-routed --helpexits 0)npm install && npm test)🤖 Generated with Claude Code