Crazy Ideas: OpenDAL File System

[Xuanwo]

Can we build a file system with OpenDAL? Ideally, no. OpenDAL and its underlying storage services do not support random writes, which are crucial for file system use cases. To enable this feature, something like JuiceFS with a metadata service is needed.

However, there is still a need for a file system that:

• Supports both sequential and random reads
• Only supports sequential writes
• Provides other basic file system operations such as listing and stat (which can be easily cached)

There are some projects that fulfill these requirements:

• s3fs-fuse: https://github.com/s3fs-fuse/s3fs-fuse
• mountpoint-s3: https://github.com/awslabs/mountpoint-s3
• gcsfuse: https://github.com/GoogleCloudPlatform/gcsfuse

Perhaps we can implement a fuse via OpenDAL so that users can locally mount storage services and use them as local file systems for specific use cases.

Similiar ideas

• feat: oay fuse frontend support #2625

[Xuanwo]

I have been researching about virtio/virtiofs/vDPA/VDUSE recently. Maybe we can provide both virtio and fuse support for ofs?

For virtio

Users can mount ofs to container or vm directly. All control message send via a char device and all data send via virtio which is much faster than fuse.

This solution is linux only.

For fuse

Users can mount ofs to host directly via fuse. All control message and data are sent via fuse device.

Nearly all platform supported.

---

We can build ofs-core and provide ofs-virtiofs and ofs-fuse for users.

[oowl]

I'm having a bit of trouble grasping this concept. openDAL provides a set of library APIs for users. Therefore, we suggest users should utilize some bindings or the core rust API for their purpose. However, the term 'ofs' seems perplexing to me. Could it be more akin to the 'oay' frontend service?

[Xuanwo]

Yes. More like part of oay.

[Xuanwo]

I'm also thinking of splitting oay into different binaries to make it easier to use.

[Xuanwo]

A PoC written by @Inokinoki

https://github.com/Inokinoki/DalFs