Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bazel: use a sysroot for compilation #24669

Open
wants to merge 5 commits into
base: dev
Choose a base branch
from
Open

bazel: use a sysroot for compilation #24669

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
75ef8d0
bazel: support packaging as docker image
rockwotj Dec 5, 2024
71f8c7a
bazel: add a sysroot for a more deterministic build
rockwotj Jan 2, 2025
dd47d5b
bazel/rpk: build with purego
rockwotj Jan 2, 2025
4430087
bazel: support cross compiling rust
rockwotj Jan 3, 2025
7536982
bazel: default to purego (no CGO)
rockwotj Jan 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions MODULE.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,8 @@ use_repo(non_module_dependencies, "seastar")
use_repo(non_module_dependencies, "unordered_dense")
use_repo(non_module_dependencies, "wasmtime")
use_repo(non_module_dependencies, "xxhash")
use_repo(non_module_dependencies, "x86_64_sysroot")
use_repo(non_module_dependencies, "aarch64_sysroot")

bazel_dep(name = "toolchains_llvm", version = "1.1.2")
single_version_override(
Expand Down Expand Up @@ -131,6 +133,16 @@ llvm.toolchain(
"linux-x86_64": ["http://redpanda-core-toolchain.s3-website-us-east-1.amazonaws.com/llvm-19.1.1-ubuntu-22.04-x86_64.tar.gz"],
},
)
llvm.sysroot(
name = "llvm_19_toolchain",
label = "@x86_64_sysroot//:sysroot",
targets = ["linux-x86_64"],
)
Comment on lines +136 to +140
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dang this is very cool. does the sysroot provide isolation, or is it merely providing precedence over the default system root (e.g. to pick up other bits installed on the system from bazel/install-deps.sh).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've not researched this in depth. It's at the end of the day passing --sysroot to clang, but anecdotally I've believe it provides isolation because the first pass I did the build failed to find valgrind and xfs headers even tho I have those locally (I was only pulling in glibc and libgcc_s)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh, i see--the other deps aren't for the compiler--stuff like bison, ragel, etc..

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well considering my build works locally but fails in CI I think there is still some leaking from the local system happening...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although the missing header is in the sysroot...

Screenshot 2025-01-03 at 3 34 20 PM

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eek

llvm.sysroot(
name = "llvm_19_toolchain",
label = "@aarch64_sysroot//:sysroot",
targets = ["linux-aarch64"],
)
use_repo(llvm, "llvm_19_toolchain", "llvm_19_toolchain_llvm")

register_toolchains("@llvm_19_toolchain//:all")
Expand All @@ -148,6 +160,16 @@ llvm.toolchain(
"linux-x86_64": ["http://redpanda-core-toolchain.s3-website-us-east-1.amazonaws.com/pgo/llvm-18.1.8-ubuntu-22.04-x86_64.tar.gz"],
},
)
llvm.sysroot(
name = "llvm_18_toolchain",
label = "@x86_64_sysroot//:sysroot",
targets = ["linux-x86_64"],
)
llvm.sysroot(
name = "llvm_18_toolchain",
label = "@aarch64_sysroot//:sysroot",
targets = ["linux-aarch64"],
)
use_repo(llvm, "llvm_18_toolchain", "llvm_18_toolchain_llvm")

register_toolchains("@llvm_18_toolchain//:all")
Expand Down Expand Up @@ -247,6 +269,10 @@ use_repo(
rust = use_extension("@rules_rust//rust:extensions.bzl", "rust")
rust.toolchain(
edition = "2021",
extra_target_triples = [
"aarch64-unknown-linux-gnu",
"x86_64-unknown-linux-gnu",
],
versions = ["1.80.0"],
)
use_repo(rust, "rust_toolchains")
Expand All @@ -270,6 +296,38 @@ crate.annotation(
gen_build_script = "off",
)

# ====================================
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bazel: support packaging as docker image
Big picture:

just to clarify, this is the proposal for the docker image packaging that'll eventually be uploaded into docker hub and used by the community/docker-compose/etc...?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be a breaking change to use this exact configuration. I think we could offer this specific version as a redpanda-slim variant, but I think we'll stick with the existing packaging story for the medium term. Eventually, we should move our existing setup into Bazel, but I don't see that happening for awhile. I actually wanted to do this docker thing so that I could run/update the Wasm tests to not use the nightly build but use the actual docker image at HEAD, and we can orchestrate everything with Bazel.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so that I could run/update the Wasm tests to not use the nightly build but use the actual docker image at HEAD, and we can orchestrate everything with Bazel.

got it. yeh i wasn't sure how it was going to be used

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I do think we should move to something slimmed down like this eventually. But I should be more clear about the intent of this for sure

# OCI Base Images
# ====================================
bazel_dep(name = "rules_oci", version = "2.0.1", dev_dependency = True)

oci = use_extension("@rules_oci//oci:extensions.bzl", "oci")

# Use a distroless docker image that only comes with the following:
#
# ca-certificates
# /etc/passwd entry for a root user
# /tmp directory
# tzdata
# glibc
# libgcc1
oci.pull(
name = "distroless_cc_debian12",
# This is the latest nonroot tag
digest = "sha256:594b5200fd1f06d17a877ebee16d4af84a9a7ab83c898632a2d5609c0593cbab",
image = "gcr.io/distroless/cc-debian12",
platforms = [
"linux/amd64",
"linux/arm64/v8",
],
)
use_repo(
oci,
"distroless_cc_debian12",
"distroless_cc_debian12_linux_amd64",
"distroless_cc_debian12_linux_arm64_v8",
)

# ====================================
# clang-tidy
# ====================================
Expand Down
1,281 changes: 909 additions & 372 deletions MODULE.bazel.lock
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 0 additions & 4 deletions bazel/install-deps.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,6 @@ deb_deps=(
make
pkgconf
ragel
valgrind
xfslibs-dev
sudo
)

Expand All @@ -48,8 +46,6 @@ fedora_deps=(
libtool
perl
ragel
valgrind-devel
xfsprogs-devel
xorg-x11-util-macros
sudo
)
Expand Down
95 changes: 85 additions & 10 deletions bazel/packaging/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
load("@bazel_skylib//rules:expand_template.bzl", "expand_template")
load("@rules_oci//oci:defs.bzl", "oci_image", "oci_load")
load("@rules_pkg//pkg:mappings.bzl", "pkg_attributes", "pkg_files", "strip_prefix")
load("@rules_pkg//pkg:tar.bzl", "pkg_tar")

pkg_files(
name = "redpanda_binary_with_runfiles",
srcs = ["//src/v/redpanda"],
attributes = pkg_attributes(
mode = "0755",
),
# Including runfiles has the effect of copying all of the shared libraries needed to run the binary.
# Although they always retain the directory structure that is found in bazel build output, meaning
# that `strip_prefix` has no effect on runfiles here (so we strip_prefix in the other targets).
Expand All @@ -24,24 +27,32 @@ pkg_files(

pkg_files(
name = "redpanda_libexec",
srcs = ["//src/v/redpanda"],
srcs = [
"//src/go/rpk/cmd/rpk",
"//src/v/redpanda",
],
attributes = pkg_attributes(
mode = "0755",
),
prefix = "libexec",
# This flattens the files so that everything is directly in the `libexec` folder.
strip_prefix = strip_prefix.files_only(),
)

expand_template(
name = "redpanda_binary_wrapper",
out = "redpanda",
substitutions = {
"%%BINARY%%": "redpanda",
},
template = "exec.tmpl.sh",
pkg_files(
name = "redpanda_conf",
srcs = ["//conf:redpanda.yaml"],
prefix = "etc/redpanda",
# This flattens the files so that everything is directly in the `etc/redpanda` folder.
strip_prefix = strip_prefix.files_only(),
)

pkg_files(
name = "redpanda_bin",
srcs = [":redpanda_binary_wrapper"],
srcs = [
"//bazel/packaging/bin_wrapper:redpanda",
"//bazel/packaging/bin_wrapper:rpk",
],
attributes = pkg_attributes(
mode = "0755",
),
Expand All @@ -63,3 +74,67 @@ pkg_tar(
extension = ".tar.gz",
package_file_name = "redpanda.tar.gz",
)

pkg_tar(
name = "redpanda_image_broker_files",
srcs = [
":redpanda_bin",
":redpanda_lib",
":redpanda_libexec",
],
package_dir = "/opt/redpanda",
)

# This is the ID for the nonroot user in the distroless containers.
NONROOT_USER = "65532"

pkg_tar(
name = "redpanda_image_config_files",
srcs = [
":redpanda_conf",
],
mode = "0o555",
owner = NONROOT_USER + "." + NONROOT_USER,
package_dir = "/etc/redpanda",
)

pkg_tar(
name = "redpanda_image_writable_dirs",
empty_dirs = [
"/etc/redpanda",
"/var/lib/redpanda/data",
],
mode = "0o755",
owner = NONROOT_USER + "." + NONROOT_USER,
)

oci_image(
name = "image",
base = "@distroless_cc_debian12",
cmd = [
"redpanda",
"start",
"--overprovisioned",
],
entrypoint = ["/opt/redpanda/bin/rpk"],
exposed_ports = [
"8081",
"8082",
"9092",
"9644",
],
labels = {"org.opencontainers.image.authors": "Redpanda Data <[email protected]>"},
tars = [
":redpanda_image_broker_files",
":redpanda_image_config_files",
":redpanda_image_writable_dirs",
],
user = "nonroot",
volumes = ["/var/lib/redpanda/data"],
)

oci_load(
name = "image_load",
image = ":image",
repo_tags = ["redpandadata/redpanda-dev:latest"],
)
24 changes: 24 additions & 0 deletions bazel/packaging/bin_wrapper/BUILD
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
load("@rules_go//go:def.bzl", "go_binary")

# gazelle:ignore

go_binary(
name = "rpk",
srcs = ["main.go"],
pure = "on",
visibility = ["//bazel/packaging:__pkg__"],
x_defs = {
"BinaryPath": "/opt/redpanda/libexec/rpk",
},
)

go_binary(
name = "redpanda",
srcs = ["main.go"],
pure = "on",
visibility = ["//bazel/packaging:__pkg__"],
x_defs = {
"BinaryPath": "/opt/redpanda/libexec/redpanda",
"LDLibraryPath": "/opt/redpanda/lib",
},
)
38 changes: 38 additions & 0 deletions bazel/packaging/bin_wrapper/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
/*
* Copyright 2024 Redpanda Data, Inc.
*
* Use of this software is governed by the Business Source License
* included in the file licenses/BSL.md
*
* As of the Change Date specified in that file, in accordance with
* the Business Source License, use of this software will be governed
* by the Apache License, Version 2.0
*/

package main

import (
"os"
"slices"
"strings"
"syscall"
)

var (
LDLibraryPath = ""
BinaryPath = ""
)

func main() {
path := "PATH=/opt/redpanda/bin"
existingPath, ok := os.LookupEnv("PATH")
if ok {
path += string(os.PathListSeparator) + existingPath
}
env := slices.DeleteFunc(os.Environ(), func(envvar string) bool { return strings.HasPrefix(envvar, "PATH=") })
env = append(env, path)
if LDLibraryPath != "" {
env = append(env, "LD_LIBRARY_PATH="+LDLibraryPath)
}
syscall.Exec(BinaryPath, os.Args, env)
}
5 changes: 0 additions & 5 deletions bazel/packaging/exec.tmpl.sh
View file
Open in desktop

This file was deleted.

20 changes: 20 additions & 0 deletions bazel/repositories.bzl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,3 +197,23 @@ def data_dependency():
strip_prefix = "xxHash-bbb27a5efb85b92a0486cf361a8635715a53f6ba",
url = "https://github.com/Cyan4973/xxHash/archive/bbb27a5efb85b92a0486cf361a8635715a53f6ba.tar.gz",
)

sysroot_build_file = """
filegroup(
name = "sysroot",
srcs = glob(["*/**"]),
visibility = ["//visibility:public"],
)"""
http_archive(
name = "x86_64_sysroot",
build_file_content = sysroot_build_file,
sha256 = "b7544f6931531eababde47d8723362bc9b3c9de8ef7d9a5d1d62d2e614904d5b",
urls = ["http://redpanda-core-toolchain.s3-website-us-east-1.amazonaws.com/sysroot-ubuntu-22.04-x86_64-2025-01-03.tar.gz"],
)

http_archive(
name = "aarch64_sysroot",
build_file_content = sysroot_build_file,
sha256 = "a2d8c3ce82c70f884281db846e756f5877f2c77964cbfb1ba6b954e1f759e2ed",
urls = ["http://redpanda-core-toolchain.s3-website-us-east-1.amazonaws.com/sysroot-ubuntu-22.04-aarch64-2025-01-03.tar.gz"],
)
2 changes: 1 addition & 1 deletion bazel/toolchain/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
llvm-*.tar.gz
*.tar.gz
2 changes: 1 addition & 1 deletion bazel/toolchain/Dockerfile.llvm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ ADD --chmod=755 https://github.com/kadwanev/retry/raw/20997c7712a4/retry /usr/bi

WORKDIR /opt

ARG LLVM_VERSION=17.0.6
ARG LLVM_VERSION=19.1.6

RUN git clone --branch llvmorg-${LLVM_VERSION} --single-branch https://github.com/llvm/llvm-project.git

Expand Down
37 changes: 37 additions & 0 deletions bazel/toolchain/Dockerfile.sysroot
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
FROM ubuntu:jammy AS pkgs

ENV DEBIAN_FRONTEND=noninteractive
SHELL ["/bin/bash", "-c"]

RUN apt-get update && apt-get install -y \
libc6 \
libc6-dev \
libc6-dbg \
libgcc-12-dev \
valgrind \
xfslibs-dev \
&& apt-get clean && rm -rf /var/lib/apt/lists/*
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

curious why we don't put the rest of the deps into this list?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which deps? The ones from bazel/install-deps.sh? I believe the rest of those are for tools that we need bazel to invoke (ragel, etc) are there others that are compile time?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, i realized this when you mentioned above that sysroot was just passed to clang. i was thinking sysroot was a bazel thing, not a compiler toolchain thing.


# Copy required glibc files
RUN mkdir -p /output/pkgs/{usr/,}lib/$(uname -m)-linux-gnu /output/pkgs/usr/include /output/pkgs/lib64 \
&& cp -r /usr/lib/*-linux-gnu/ld-linux-*.so* /output/pkgs/lib64/ \
&& cp -r /usr/lib/*-linux-gnu/libc.{a,so}* /output/pkgs/lib/$(uname -m)-linux-gnu/ \
&& cp -r /usr/lib/*-linux-gnu/libc_nonshared.a* /output/pkgs/usr/lib/$(uname -m)-linux-gnu/ \
&& cp -r /usr/lib/*-linux-gnu/libm.{a,so}* /output/pkgs/lib/$(uname -m)-linux-gnu/ \
&& (if test $(uname -m) = "x86_64"; then cp -r /usr/lib/x86_64-linux-gnu/libmvec.{a,so}* /output/pkgs/lib/x86_64-linux-gnu/; fi) \
&& cp -r /usr/lib/*-linux-gnu/libdl.{a,so}* /output/pkgs/usr/lib/$(uname -m)-linux-gnu/ \
&& cp -r /usr/lib/*-linux-gnu/librt.{a,so}* /output/pkgs/usr/lib/$(uname -m)-linux-gnu/ \
&& cp -r /usr/lib/*-linux-gnu/libutil.{a,so}* /output/pkgs/usr/lib/$(uname -m)-linux-gnu/ \
&& cp -r /usr/lib/*-linux-gnu/libpthread.{a,so}* /output/pkgs/usr/lib/$(uname -m)-linux-gnu/ \
&& cp -r /usr/lib/*-linux-gnu/libgcc_s.so* /output/pkgs/usr/lib/$(uname -m)-linux-gnu/ \
&& cp -r /usr/lib/*-linux-gnu/*crt*.o /output/pkgs/usr/lib/$(uname -m)-linux-gnu/ \
&& cp -r /usr/lib/*-linux-gnu/libatomic.so* /output/pkgs/usr/lib/$(uname -m)-linux-gnu/ \
&& cp -r /usr/lib/gcc/*-linux-gnu/12/*.o /output/pkgs/usr/lib/$(uname -m)-linux-gnu/ \
&& cp -r /usr/lib/gcc/*-linux-gnu/12/libgcc*.{a,so}* /output/pkgs/usr/lib/$(uname -m)-linux-gnu/ \
&& cp -r /usr/include/* /output/pkgs/usr/include/

FROM scratch

COPY --from=pkgs /output/pkgs /


Loading
Loading