Multi-Tenancy Support in Solid Queue

[hubertjakubiak]

Are there any plans to add support for multi-tenancy in Solid Queue? Specifically, is there a roadmap or intention to allow each tenant to store background jobs in their dedicated database?

[rosa]

@hubertjakubiak yes! Via shards: #353

[hubertjakubiak]

Hi @rosa,

Just to check, should a single worker process handle background jobs from multiple databases?

If so, I guess the Solid Queue tables would need to move from queue_schema.rb to separate schema files for each tenant, like schema.rb, tenant_1_schema.rb, and tenant_2_schema.rb. Does that sound right?

[rosa]

In my plans, a worker would handle jobs from a single DB (a single shard) so that we can easily join workers with their claimed executions. This would allow deregistering a worker and releasing claimed executions within a transaction (all in the same DB). So you'd have at least as many workers as shards.

About the schema, yes, that's exactly right, it'd need to use multiple schema files, all identical, one per shard. Unfortunately, that's how it works. I'd prefer if Rails supported having a single schema file for all shards since they're identical, but it's not a big deal.

[hubertjakubiak]

I would expect one worker to handle background jobs for all tenants. Having separate workers for each tenant goes against the idea of multi-tenancy, where resources are shared. Just like I have one web worker for the site, I would expect one worker to handle jobs for all tenants.

[rosa]

Then, Solid Queue is not going to support multi-tenancy as you expect it. It'll support sharding (with the main goal of horizontal scaling), which you could use for multi-tenancy but with a worker per shard.

[rosa]

Going to close this one as we already have an issue for sharding, #353.

[hubertjakubiak]

@rosa Thank you for clarifying how the feature will work. I appreciate it.