Exited container not get release from Jenkins

[saibalak80]

The containers not getting cleaned. any suggest whats going wrong?

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
3002f37077b9 jenkins-slave:1.7 "/bin/sh -cxe cat ..." 21 seconds ago Starting sad_fermi
f15178ca9657 jenkins-slave:1.7 "/bin/sh -cxe cat ..." 16 minutes ago Exited (0) 10 minutes ago wonderful_mclean
257727b1aa9a jenkins-slave:1.7 "/bin/sh -cxe cat ..." 16 minutes ago Exited (0) 10 minutes ago

[samrocketman]

Need more system information.

• Linux distro and version.
• Docker daemon version.

[saibalak80]

I'm running it on vsphere .

Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 11
Server Version: v1.3.1-16055-afdab46
Storage Driver: vSphere Integrated Containers v1.3.1-16055-afdab46 Backend Engine
VolumeStores: default swap
vSphere Integrated Containers v1.3.1-16055-afdab46 Backend Engine: RUNNING
VCH CPU limit: 39600 MHz
VCH memory limit: 64 GiB
VCH CPU usage: 4529 MHz
VCH memory usage: 18.79 GiB
VMware Product: VMware vCenter Server
VMware OS: linux-x64
VMware OS version: 6.0.0
Registry Whitelist Mode: enabled
Whitelisted Registries: x18gvichub01a.prd.cloudnow.dbs.com,x18gvichub01b.prd.cloudnow.dbs.com
Plugins:
Volume: vsphere
Network: bridge
Swarm: inactive
Operating System: linux-x64
OSType: linux-x64
Architecture: x86_64
CPUs: 39600
Total Memory: 64 GiB
ID: vSphere Integrated Containers
Docker Root Dir:
Debug Mode (client): false
Debug Mode (server): false
Registry: registry.hub.docker.com
Experimental: false
Live Restore Enabled: false

[saibalak80]

Another one issue on below flow.

1. No containers templates configured
2. 10 Jobs pending in Queue and waiting for containers.
3. Configured Template with docker once retention strategy
4. 10 containers spin up and 8 finished job and die.
5. 2 were running as its long-running process.
6. one more job added in the queue and it's waiting for long time to spin a new one.
7. when the 2 current running jobs completed then only the new containers coming up.

is this expected one,? there should be new slave spin up when job added to the queue right?

[samrocketman]

https://vmware.github.io/vic-product/assets/files/html/1.1/vic_app_dev/container_operations.html vsphere does not appear to be a full implementation of the remote docker API. If you manually delete a single offline agent does the container get removed on the backend? What version of yet another docker plugin are you using?

[saibalak80]

Yes it does, the version im using is "0.1.0-rc46".

[samrocketman]

@saibalak80 as a temporary workaround you can create a regularly scheduled freestyle job which cleans up dead agents. You would need the groovy plugin installed and run the following system groovy script. https://github.com/samrocketman/jenkins-script-console-scripts/blob/master/cleanup-offline-agents.groovy

I would personally modify it so that it takes into account the age of the agent (i.e. don't delete agents which were just created and make sure the dead agent is at least 5 minutes old) and run it on an infrequent schedule like no less than every 30 minutes. You don't even necessarily need to account for the age of the agent because the docker plugin will just provision new agents after they were deleted.

It's not a perfect workaround but would address infinite resource starvation.

[KostyaSha]

AFAIR i did cleanup somewhere. In retentionStrtategy?...

[samrocketman]

I don't experience this when using Docker daemon so it might be a difference with how vsphere containers work compared to Docker.