Support for custom StorageConfig implementations

[empperi]

Right now Keystone 6 supports two StorageConfig implementations: kind: "s3" and kind: "local". This is very limiting and Keystone should allow users to write their own implementations if these do not suffice.

Adding support for this should be rather simple. One needs to change the StorageConfig type declaration to be an interface out of which specific implementations are inherited and these then may add new properties. This would be a backwards compatible change in typing.

Also at the code level it seems like these places are affecting this change:

keystone/packages/core/src/lib/assets/createImagesContext.ts

Lines 33 to 39 in b3fcc3e

	imageAssetsAPIs.set(
	storageKey,
	storageConfig.kind === 'local'
	? localImageAssetsAPI(storageConfig)
	: s3ImageAssetsAPI(storageConfig)
	);
	}

and

keystone/packages/core/src/lib/assets/createFilesContext.ts

Lines 39 to 47 in b3fcc3e

	if (storageConfig.type === 'file') {
	adaptersMap.set(
	storageKey,
	storageConfig.kind === 'local'
	? localFileAssetsAPI(storageConfig)
	: s3FileAssetsAPI(storageConfig)
	);
	}
	}

In both cases these just do a crude if-else based logic on kind property and then initialize the appropriate implementation. To retain backwards compatibility such if-else logic should be kept but there should be a third check that if the storageConfig is already an instance of StorageConfig then it would just simply use that.

This would provide ultimate flexibility without sacrificing backwards compatibility and it is a relatively simple and minor change to the codebase.

[dcousens]

This is very limiting and Keystone should allow users to write their own implementations if these do not suffice.

Thanks for the feature request @empperi, do you have any examples of where this flexibility would have helped you?
Maybe something concrete we could use as a basis for what a custom StorageConfig might be trying to achieve?

[empperi]

My initial need was to support Google Cloud Storage. That can be handled via S3 adapter but then it seemed to flow the authentication keys within the URI to the outside world. I kinda get it since the bucket was not public but this was frightening. So, I wanted to Keystone to act as a proxy in between but this is not supported by the S3 provider. I could hack around this by extending the express app and then adding specific exceptions to isAccessAllowed but this fragments the functionality all over the place. But with custom StorageConfig I could keep it all neatly in one place where it should be.

In the future I also need to show images from a 3rd party API but in a way that they are automatically bound to the objects within Keystone. Which means custom UI elements etc. Now, if there would be a proper StorageConfig implementation that I could write it would fit nicely to the whole architecture. Again, no need to expose 3rd party API to the public internet, authentication keys would be in Keystone and so on.

Sure, CDNs are also in play and should be taken into consideration but that is a different story and can be mostly handled by generateUrl.

[hpaul]

Researching keystonejs as the backend for our next project I had found out that it's missing the same feature as requested by @empperi. And since our project is somewhat dependent on Google Cloud Storage sadly I was going to say that Keystone isn't a fit, even if it answer all other requirements and seems like a fun to work with.

But, and @empperi pointed out, I found out that actually Google Cloud Storage is "compatible" with S3, and you could actually use AWS library to store and get things out of GCS. Further reading for anyone else wondering here: https://cloud.google.com/storage/docs/aws-simple-migration.

Certainly I wouldn't expect it to be a walk in the park, there will be of course a few issues, but I'll have them written here, and the way to overcome them.

[empperi]

We ended up using the S3 compatibility layer in Cloud Storage and it has worked quite ok in production for half a year now. So based on that experience I would say you can proceed with that approach. There were few minor caveats one had to work around but unfortunately I can't remember those anymore. I'm not working on that project anymore and don't have access to the source code or other resources so I could double check. But in any case, there was nothing major for us to prevent using Cloud Storage.

[hpaul]

Great, I'm glad it works like that out of the box, since we're talking about two companies which are competing with each other.

Once I get up with things I'll have a feedback posted here. Thanks for your prompt answer @empperi !