Memory leak

[phlmn]

Hi, I'm experiencing extensive memory usage of the Tartelet process. Tartelet runs for less than one day and then does not process jobs anymore, due to the system being out of memory.

I've configured Tartelet to run two VMs.

System Info

Hardware Overview:
Model Name: Mac mini
Model Identifier: Mac14,3
Model Number: Z16K000EYD/A
Chip: Apple M2
Total Number of Cores: 8 (4 performance and 4 efficiency)
Memory: 16 GB
System Firmware Version: 8422.121.1
OS Loader Version: 8422.121.1
Activation Lock Status: Disabled

System Software Overview:
System Version: macOS 13.4 (22F66)
Kernel Version: Darwin 22.5.0
Boot Volume: Macintosh HD
Boot Mode: Normal
Secure Virtual Memory: Enabled
System Integrity Protection: Enabled

[simonbs]

@phlmn Are you running the latest version of Tartelet?

I've just checked our runners and I'm not seeing nearly as high a memory usage. See below for a screenshot of Activity Monitor from one of the runners.

I also ran the Leaks profiling tool on Tartelet and it revealed no leaks when runners were restarted a bunch of times.

[slashmo]

I saw the same memory usage spike happening on our Mac Mini M1 (16GB RAM) with Tartelet 0.7.0. We also hit ~ 120GB of memory usage and the runners weren't properly registering anymore, therefore stalling our CI over night. The host runs the latest version of macOS Ventura (13.6 (22G120)).

[phlmn]

Sorry for the late answer. The funny thing is the issue seems to have disappeared on our system… The only thing I know we have changed is too run only one VM. But I believe we also had memory leaks after the change.

[xavierLowmiller]

We're also seeing this on version 0.7.0:

[xavierLowmiller]

I also ran the Leaks profiling tool on Tartelet and it revealed no leaks when runners were restarted a bunch of times.

Maybe this can be debugged in Xcode 15.1, the release notes mention that the Leaks instrument has had a bug.

[simonbs]

I'll make a note to look into this once I get the time but that might not happen until in a few weeks. If anyone else want to take a swing at it in the mean time that would be amazing 🤗

[weakfl]

I'm experiencing the same issue with v0.7.1, at least with two VMs.

[hisaac]

Throwing my name in the ring here, we're also seeing this issue, running the most recent version of Tartelet. It takes a day or so before it crops up.

[hisaac]

So we've got multiple machines running Tartelet. The issue seems to crop up randomly.

One thing I've noticed is that, when I VNC into a host machine that is currently experiencing the problem, I'm presented with a login screen. Once logging in, there are no active Tart VM sessions running, but they immediately start upon logging in.

On machines that aren't currently experiencing the issue, when I VNC into them, I'm brought right to the desktop without going to the login window first, and Tart VMs are currently running.

It appears as if the machines are going to sleep, even though as far as I can tell, all the machines are configured the same, and have the sleep functionality disabled.

[simonbs]

I've just checked our three machines which have been maintaining two runners for 42 consecutive days and Tartelet is using around 85 MB of memory on all of them.

I'd love to address this issue but right now I'm unsure where to begin as I'm unable to reproduce it.

[hisaac]

Understandable @simonbs. I'd like to keep documenting findings here in case anyone affected has ideas or solutions.

[hisaac]

Not sure if either of these will be useful to anyone, but I took a process sample and spindump of Tartelet when I saw it was using high memory on one of our hosts.

• Process Sample.txt
• Spindump.txt

Nothing obvious sticks out to me, but I'm not very experienced reading these.

[simonbs]

I wonder if this might be fixed by #97. It ssems like the sort of thing that could cause a memory leak like this.

[kuhnroyal]

Yeah, was thinking the same. But haven't seen any big memory usage even before that.