podman 5.3.1 unable to mount volume with an idmap.

[james-lawrence]

Issue Description

Im unable to run a container with a volume mounted w/ the idmap option. My end goal is to have the volume be owned by the ubuntu user in the container instead of root.

I feel like this is pebcac. but I'm completely baffled atm.

Steps to reproduce the issue

in the container if i omit the --volume idmap.

$ ls -lha /opt
total 8.0K
drwxr-xr-x 2 root root 4.0K Nov 20 02:03 .
dr-xr-xr-x 18 root root 4.0K Dec 31 03:43 ..

$ id -u test
994

on the host

id -u other
1003
id -u me
1000

ls -lha test
total 16K
drwx------ 4 me me 4.0K Dec 30 17:41 .
drwx------ 4 me me 4.0K Dec 30 17:36 ..
-rw-r--r-- 1 me me 0 Dec 30 22:45 derp.txt

podman run --replace --name test --detach --volume "test:/opt/test:idmap=uids=0-1003-1#994-1000-1;gids=0-1003-1#994-1000-1" ubuntu-24-10 /usr/sbin/init

Describe the results you received

Error: crun: mount_setattr /opt/test: Operation not permitted: OCI permission denied

Describe the results you expected

/opt/test to contain the contents of the test directory and be owned by user 994.

podman info output

host:
  arch: amd64
  buildahVersion: 1.38.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-1:2.1.12-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: e8896631295ccb0bfdda4284f1751be19b483264'
  cpuUtilization:
    idlePercent: 97.46
    systemPercent: 1.12
    userPercent: 1.42
  cpus: 16
  databaseBackend: sqlite
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2045
  hostname: mambli
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 655360
  kernel: 6.6.67-1-lts
  linkmode: dynamic
  logDriver: journald
  memFree: 37651767296
  memTotal: 66791415808
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.13.1-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.13.1
    package: netavark-1.13.1-1
    path: /usr/lib/podman/netavark
    version: netavark 1.13.1
  ociRuntime:
    name: crun
    package: crun-1.19.1-2
    path: /usr/bin/crun
    version: |-
      crun version 1.19.1
      commit: 3e32a70c93f5aa5fea69b50256cca7fd4aa23c80
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-2024_11_27.c0fbc7e-1
    version: |
      pasta 2024_11_27.c0fbc7e
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-1
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.8.0
      SLIRP_CONFIG_VERSION_MAX: 5
      libseccomp: 2.5.5
  swapFree: 4293652480
  swapTotal: 4294963200
  uptime: 12h 21m 16.00s (Approximately 0.50 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.ignore_chown_errors: "true"
  graphRoot: /home/user/.local/share/containers/storage
  graphRootAllocated: 426093133824
  graphRootUsed: 155489542144
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 35
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 5.3.1
  Built: 1732225906
  BuiltTime: Thu Nov 21 16:51:46 2024
  GitCommit: 4cbdfde5d862dcdbe450c0f1d76ad75360f67a3c
  GoVersion: go1.23.3
  Os: linux
  OsArch: linux/amd64
  Version: 5.3.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

laptop =)

Additional information

--cap-add SYS_ADMIN, --privileged, sudo had no effect.

[eriksjolund]

I think the idmap option is only supported when running rootful, that is you would need

sudo podman run ...

Side note: Some problems can also be solved by using:

--userns keep-id:uid=${uid},gid=${gid}

which is pretty similar to

 --uidmap=+${uid}:@$(id -u)

I compared the two in this reddit comment https://www.reddit.com/r/podman/comments/1dcj84b/comment/l7zuu25/

[james-lawrence]

as i mentioned in additional information sudo doesnt change anything. keep-id:uid=${uid}... isnt going to solve this.

first off I want the user mapped to root as normal due to launching systemd. I just want the volume translated differently. I've also tried various forms of --userns and --uidmap/--gidmap nothing works. Im suspecting kernel misconfig or something atm

[eriksjolund]

OK, here is another idea:
The ownership of the volume can be specifed with an option to podman volume create

$ podman volume create -o=o=uid=1050,gid=1050 test
test
$ podman run --rm -v test:/src docker.io/library/alpine ls -ld /src
drwxr-xr-x    2 1050     1050             6 Dec 29 13:45 /src
$ 

[james-lawrence]

Alright, so using the volume create generates new directories with specific uid/gids. which is fine (and works, see below). but doesn't address the idmap option not working. correct me if i'm wrong. the idmap option is suppose to allow remapping of uid/gids of the volume on the host to specific uid/gids within the container. i.e. idmap=uids=994-1000-1 should present any file/directory owned by uid 1000 on the host as 994 within the container. this is exactly what I'm looking for but looks like there is a bug in podman handling this mapping.

± podman volume create -o=o=uid=1050,gid=1050 derp
derp
 ± podman run --rm -v derp:/src docker.io/library/archlinux ls -ld /src
drwxr-xr-x 2 1050 1050 4096 Dec 31 14:44 /src
± podman run --rm -v derp:/src docker.io/library/ubuntu ls -ld /src
drwxr-xr-x 2 1050 1050 4096 Dec 31 14:44 /src
± podman run --rm -v derp:/opt/test2 eg ls -ld /opt/test2
drwxr-xr-x 2 1050 1050 4096 Dec 31 14:44 /opt/test2

± podman volume create -o=o=uid=994,gid=994 derp2
derp2
podman run --replace --name eg-test --detach -v derp2:/opt/test2 eg /usr/sbin/init
fad5bdd030c8b8a139cf22676ca0007728774d1c0e4962368cefaa67beb211e6
± podman exec -it eg-test /bin/bash
root@fad5bdd030c8:/# ls -lha /opt/
total 16K
drwxr-xr-x  4 root root 4.0K Dec 31 14:47 .
dr-xr-xr-x 18 root root 4.0K Dec 31 14:47 ..
drwxr-xr-x  2 eg   eg   4.0K Dec 31 04:00 eg
drwxr-xr-x  2 eg   eg   4.0K Dec 31 14:47 test2

ls -lha /home/user/.local/share/containers/storage/volumes/
total 16K
drwx------  4 user user 4.0K Dec 31 09:47 .
drwx------ 12 user user 4.0K Dec 31 09:47 ..
drwx------  3 101049 101049 4.0K Dec 31 09:44 derp
drwx------  3 100993 100993 4.0K Dec 31 09:47 derp2

documentation for idmap:

Idmapped mount

If idmap is specified, create an idmapped mount to the target user namespace in the container. The idmap option supports a custom mapping that can be different than the user namespace used by the container. The mapping can be specified after the idmap option like: idmap=uids=0-1-10#10-11-10;gids=0-100-10. For each triplet, the first value is the start of the backing file system IDs that are mapped to the second value on the host. The length of this mapping is given in the third value. Multiple ranges are separated with #.

[james-lawrence]

spent some time playing with mount inside the container, feel like I'm clearly missing something here =)

root@9947a9e97430:/# ls -lha /opt/test/
total 8.0K
drwxrwxrwx 2 root root 4.0K Dec 31 13:56 .
drwxr-xr-x 4 root root 4.0K Dec 31 15:08 ..
-rw-r--r-- 1 root root    0 Dec 31 13:13 derp.txt
-rw-r--r-- 1 root root    0 Dec 31 13:44 derp2
-rw-r--r-- 1 root root    0 Dec 31 13:56 derp3
root@9947a9e97430:/# mount --map-users 0:994:1 /opt/test/ /mnt
mount: /mnt: permission denied.
       dmesg(1) may have more information after failed mount system call.
root@9947a9e97430:/# mkdir /opt/foo 
root@9947a9e97430:/# touch /opt/foo/example.txt
root@9947a9e97430:/# mount --map-users 0:994:1 /opt/foo/ /mnt
mount: /mnt: permission denied.
       dmesg(1) may have more information after failed mount system call.
root@9947a9e97430:/# mount --map-users 0:994:1 /opt/test/ /mnt
mount: /mnt: permission denied.
       dmesg(1) may have more information after failed mount system call.

[james-lawrence]

on my computer all the overlay permissions look correct:

sudo ls -lha /home/me/.local/share/containers/storage/overlay/0a07f40e4a36a31ead39ce441b0b5c25a5204de4a5c2e5946f926f312c575311/diff/opt/
total 12K
drwxr-xr-x 3 me me 4.0K Dec 31 10:27 .
dr-xr-xr-x 6 me me 4.0K Dec 31 10:29 ..
drwxr-xr-t 2 me me 4.0K Dec 31 10:27 test

looks correct, 1000999 maps to uid 1000 which is ubuntu:

sudo ls -lha /home/me/.local/share/containers/storage/overlay/0a07f40e4a36a31ead39ce441b0b5c25a5204de4a5c2e5946f926f312c575311/diff/home/
total 12K
drwxr-xr-x 3 me me 4.0K Dec 31 10:29 .
dr-xr-xr-x 6 me me 4.0K Dec 31 10:29 ..
drwxr-x--- 2 100999 100999 4.0K Dec 31 10:29 ubuntu 

I touched a file as root in ubuntu's home and on my host it shows up correctly.

sudo ls -lha /home/me/.local/share/containers/storage/overlay/0a07f40e4a36a31ead39ce441b0b5c25a5204de4a5c2e5946f926f312c575311/diff/home/**
total 8.0K
drwxr-x--- 2 100999 100999 4.0K Dec 31 10:29 .
drwxr-xr-x 3 me me 4.0K Dec 31 10:29 ..
-rw-r--r-- 1 me me    0 Dec 31 10:29 root.txt

[james-lawrence]

simplest reproduction i can make of the issue.

rm -rf test && mkdir test && podman run --rm -it -v "$(pwd)/test:/opt/test:rw,idmap=uids=1000-1000-1;gids=1000-1000-1" --userns host docker.io/library/ubuntu:oracular /usr/bin/bash
Error: crun: mount_setattr `/opt/test`: Operation not permitted: OCI permission denied
± podman run --rm -it --userns host docker.io/library/ubuntu:oracular id -u ubuntu
1000

[james-lawrence]

alright finally got something working, though it fails in interesting ways...:

rm -rf test && mkdir test && sudo podman run -it -v "$(pwd)/test:/opt/test:rw,idmap=uids=1000-1000-1;gids=1000-1000-1" --userns host docker.io/library/ubuntu:oracular /usr/bin/bash

root@adb6cee2fb6e:/# ls -lha /opt/test/
total 8.0K
drwxr-xr-x 2 ubuntu ubuntu 4.0K Dec 31 15:59 .
drwxr-xr-x 1 root   root   4.0K Dec 31 15:59 ..

# looks like a possible mismatch in syscall.Errnos?
root@adb6cee2fb6e:/# touch /opt/test/root.txt
touch: cannot touch '/opt/test/root.txt': Value too large for defined data type 

root@2cbfb664d060:/# chmod o+w /opt/test
root@2cbfb664d060:/# touch /opt/test/root.txt
touch: cannot touch '/opt/test/root.txt': Value too large for defined data type

root@2cbfb664d060:/# su ubuntu
ubuntu@2cbfb664d060:/# touch /opt/test/ubuntu.txt
ubuntu@adb6cee2fb6e:/opt/test$ ls -lha .
total 8.0K
drwxr-xr-x 2 ubuntu ubuntu 4.0K Dec 31 16:00 .
drwxr-xr-x 1 root   root   4.0K Dec 31 15:59 ..
-rw-rw-r-- 1 ubuntu ubuntu    0 Dec 31 16:00 ubuntu.txt

definitely a bug somewhere in the mounting stack. there are actually a couple based on this:

• first idmap doesn't work for rootless. which is either undocumented limitation, or its just broken.
• there seems to be a bug in the permissions mapping based on the fact root couldn't touch a file in the directory. even with o+w set. but this could just be a fallout of the subuid mappings on the host. i.e. root runs as 1000, and the directory is owned by 100999.
• downgrading to 5.2.5 presents same issue.

[eriksjolund]

In the example you used idmap=uids=1000-1000-1;gids=1000-1000-1

1 means a range of size 1 so it will only be possible to use UID 1000 and GID 1000.

In the following example I also get the error message Value too large for defined data type without using Podman.

$ id -u
1000
$ id -g
1000
$ mkdir test
$ mkdir mnt
$ sudo  mount -B --map-users 1000:1000:1 --map-groups 1000:1000:1 ./test ./mnt
$ cd mnt
$ ls -ldn .
drwxr-xr-x. 2 1000 1000 22 Jan  1 13:01 .
$ sudo touch root.txt
touch: cannot touch 'root.txt': Value too large for defined data type
$ touch file.txt
$ ls -ln
total 0
-rw-r--r--. 1 1000 1000 0 Jan  1 13:02 file.txt

[james-lawrence]

fair. im less concerned about the value too large error, which is essentially a 'no idea what to do with root uid/gid' it was just interesting edge case. the permission denied errors when trying to mount the volume on a rootless container is the problem.

mounting shouldn't fail in rootful vs rootless context because of the idmap options. using the minimal example:

podman run -it -v "$(pwd)/test:/opt/test:rw" --userns host ubuntu:latest bash -c 'cat /proc/self/uid_map /proc/self/gid_map && ls -lha /opt/test'
         0       1000          1
         1     100000      65536
         0       1000          1
         1     100000      65536
total 8.0K
drwxr-xr-x 2 root root 4.0K Dec 31 18:29 .
drwxr-xr-x 3 root root 4.0K Jan  1 17:08 ..

mounting the volume is clearly possible in the rootless context.

sudo podman run -it -v "$(pwd)/test:/opt/test:rw,idmap=uids=1000-1000-1;gids=1000-1000-1" --userns host ubuntu:latest bash -c 'cat /proc/self/uid_map /proc/self/gid_map && ls -lha /opt/test'
         0          0 4294967295
         0          0 4294967295
total 8.0K
drwxr-xr-x 2 ubuntu ubuntu 4.0K Dec 31 18:29 .
drwxr-xr-x 1 root   root   4.0K Jan  1 17:05 ..

the idmap clearly is workable.

podman run -it -v "$(pwd)/test:/opt/test:rw,idmap=uids=1000-1000-1;gids=1000-1000-1" --userns host ubuntu:latest bash -c 'cat /proc/self/uid_map /proc/self/gid_map && ls -lha /opt/test'
Error: crun: mount_setattr `/opt/test`: Operation not permitted: OCI permission denied

either the volume can be mounted or it cant. the mapping is either valid or it isnt. adding a idmap shouldn't suddenly choke.

taking a look at mount_setattr there are 4 reasons for mount_setattr to return ENOPERM:

One of the mounts had at least one of MOUNT_ATTR_NOATIME, MOUNT_ATTR_NODEV, MOUNT_ATTR_NODIRATIME, MOUNT_ATTR_NOEXEC, MOUNT_ATTR_NOSUID, or MOUNT_ATTR_RDONLY set and the flag is locked. Mount attributes becomes locked on a mount if: ...

MOUNT_ATTR_NOATIME - I'm assuming this is unset.
MOUNT_ATTR_NODIRATIME - I'm assuming this is unset.
MOUNT_ATTR_NOEXEC - docs be default this is not set.
MOUNT_ATTR_NOSUID - docs by default this is set.
MOUNT_ATTR_RDONLY - docs by default volumes are mounted RW.

Defaults mean setting the idmap should fail this by default. due to NOSUID being set by default. but since it doesn't in the sudo case I'm guessing the sudo case prevents the flag from getting locked.

A valid file descriptor value was specified in userns_fd, but the file descriptor refers to the initial user namespace.

Im hoping this isnt the case.

An attempt was made to add an ID mapping to a mount that is already ID mapped.

also possible, but I'm guessing not the case due to sudo working.

The caller does not have CAP_SYS_ADMIN in the initial user namespace.

this is possible and root would resolve this problem.

Taking what we've learned:

# ensuring my current user has the sys_admin capability.
± capsh --print | grep -i current
# output
#  Current: cap_sys_admin=i
#  Current IAB: cap_sys_admin

# ensuring podman has the sys_admin capability (purely for debugging this) 
sudo setcap -r /usr/bin/podman
sudo setcap 'cap_sys_admin=ep' /usr/bin/podman
getcap /usr/bin/podman
# output: /usr/bin/podman cap_sys_admin=ep

# ensure both noexec, nosuid, and rdonly are not set and give the container the sys_admin cap just in case.
± podman run -it -v "$(pwd)/test:/opt/test:rw,exec,suid,idmap=uids=1000-1000-1;gids=1000-1000-1" --userns host --cap-add=sys_admin ubuntu:latest bash -c 'cat /proc/self/uid_map /proc/self/gid_map && ls -lha /opt/test'
# output:  Error: crun: mount_setattr `/opt/test`: Operation not permitted: OCI permission denied

tears think i'm going to jump over to crun and follow up there with this.