Skip to content

Commit

Permalink
Trying out Pixi for image building
Browse files Browse the repository at this point in the history
I'm trying out using Pixi for managing our Conda and Pypi environment in place of Mamba.

pixi-kernel replaces nb-conda-kernels to allow us to support per-project environments. 

I'm using features to capture each tutorial, then adding the tutorial feature to the default environment. The `[dependencies]` table captures our base dependencies, though we might want to refactor them out into features as well ('jupyterlab support', 'cli', 'storage', 'fast xarray',...).

You can test the lab locally with either `pixi run lab` or `docker compose up py`. 

There is one little oddity I know of right now in the pixi workflow. If you create a new directory ((like a project), `pixi init`, then following pixi commands don't target that manifest as normal. 

```
$ pixi init
Initialized project in /home/jovyan/test/.
$ pixi add fastapi
 WARN Using manifest /home/jovyan/pixi.toml from `PIXI_PROJECT_MANIFEST` rather than local /home/jovyan/test/pixi.toml
```

This is because the image has already set a `pixi shell` (well `pixi shell-hook`) to force pixi to look for a specific manifest and recorded that in an environment variable. Pixi will warn that it's using a different manifest, but its easy to screw that up.

Running `pixi shell --manifest-path ./pixi.toml` will set the pixi scope to the current environment.

```
$ pixi shell --manifest-path ./pixi.toml
 . "/tmp/pixi_env_WOz.sh"
jovyan@e5cb7d16120a:~/test$  . "/tmp/pixi_env_WOz.sh"
(test) jovyan@e5cb7d16120a:~/test$ pixi add fastapi
```
  • Loading branch information
abkfenris committed Jul 31, 2024
1 parent c3d1f4b commit fff2b7d
Show file tree
Hide file tree
Showing 9 changed files with 18,640 additions and 847 deletions.
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -102,3 +102,6 @@ venv.bak/

# mypy
.mypy_cache/

# Mac
**/.DS_Store
3 changes: 3 additions & 0 deletions py-base/.gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# GitHub syntax highlighting
pixi.lock linguist-language=YAML

4 changes: 4 additions & 0 deletions py-base/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# pixi environments
.pixi
*.egg-info

2 changes: 0 additions & 2 deletions py-base/CONDARC

This file was deleted.

31 changes: 10 additions & 21 deletions py-base/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#syntax=docker/dockerfile:1.3
FROM continuumio/miniconda3:4.12.0@sha256:58b1c7df8d69655ffec017ede784a075e3c2e9feff0fc50ef65300fc75aa45ae
#syntax=docker/dockerfile:1.7
FROM ghcr.io/prefix-dev/pixi:0.26.1-jammy@sha256:45d86bb788aaa24d215eff57712f250329486ca8f442b91959bc9ce7ce6e053c

ENV NB_USER jovyan
ENV NB_UID 1000
Expand All @@ -10,32 +10,21 @@ ENV CONDA_ENV base

# Output logging faster
ENV PYTHONUNBUFFERED 1
# Don't write bytecode
ENV PYTHONDONTWRITEBYTECODE 1

USER root
RUN adduser --disabled-password --gecos "Default Jupyter user" ${NB_USER} \
&& echo ". ${CONDA_DIR}/etc/profile.d/conda.sh ; conda activate ${CONDA_ENV}" > /etc/profile.d/init_conda.sh \
# && echo ". ${CONDA_DIR}/etc/profile.d/conda.sh ; conda activate ${CONDA_ENV}" > /etc/profile.d/init_conda.sh \
&& chown -R ${NB_USER}:${NB_USER} /srv

WORKDIR ${HOME}
USER ${USER}

COPY ./conda-linux-64.lock /tmp/
USER ${NB_USER}

RUN --mount=type=cache,id=ohw_py,target=/opt/conda/pkgs,uid=${NB_UID},gid=${NB_UID} \
conda install --name ${CONDA_ENV} --file /tmp/conda-linux-64.lock && \
# micromamba install --name ${CONDA_ENV} --file environment.yml && \
find -name '*.a' -delete && \
# rm -rf /opt/conda/conda-meta && \
rm -rf ${CONDA_DIR}/include && \
find -name '__pycache__' -type d -exec rm -rf '{}' '+'
COPY ./pixi.toml ./pixi.lock ${HOME}

COPY CONDARC ./.condarc
COPY --chown=${NB_USER} entrypoint.sh /opt/entrypoint.sh
RUN --mount=type=cache,id=ohw_py,target=/home/jovyan/.cache/rattler/cache,uid=${NB_UID},gid=${NB_UID} \
pixi install --frozen -e default

# USER root
# RUN chown -R jovyan ${CONDA_DIR}
USER ${NB_USER}
RUN pixi shell-hook --frozen -e default > /srv/shell-hook.sh \
&& echo 'exec "$@"' >> /srv/shell-hook.sh

ENTRYPOINT [ "/opt/entrypoint.sh" ]
ENTRYPOINT ["/bin/bash", "/srv/shell-hook.sh"]
682 changes: 0 additions & 682 deletions py-base/conda-linux-64.lock

This file was deleted.

142 changes: 0 additions & 142 deletions py-base/environment.yml

This file was deleted.

Loading

0 comments on commit fff2b7d

Please sign in to comment.