Question on usage of preopens in WASM component

[shihanng]

I have a Go code that writes to a temporary directory like the following. I want to compile the code into a WebAssembly component and call it from a Python code.

package main

import (
	"os"

	gen "github.com/shihanng/play-wasm/wit/gen"
)

func init() {
	a := HostImpl{}
	gen.SetConvert(a)
}

type HostImpl struct{}

func (e HostImpl) Exec(input []uint8) []uint8 {
	res := []byte(input)
	res = append(res, "hello from exec"...)

	gen.ConvertPrint(string(res))

	tmpFile, err := os.CreateTemp("", "test_")
	if err != nil {
		// Print for debug
		return []byte(err.Error() + ": create temp")
	}

	defer tmpFile.Close()
	defer os.Remove(tmpFile.Name())

	if _, err := tmpFile.Write(res); err != nil {
		// Print for debug
		return []byte(err.Error() + ": write")
	}

	return []byte(res)
}

//go:generate wit-bindgen tiny-go . --out-dir=gen
func main() {}

I can compile the code and use python -m wasmtime.bindgen to generate the Python code. The following is my entry point to call the WebAssembly code:

import json

import wasmtime.bindgen
from pyconvert import Root, RootImports, imports
from wasmtime import Store


class Host(imports.Host):
    def print(self, s: str):
        print(s + "test")


class HostEnvironment:
    def get_environment(self):
        return [("TMPDIR", "/tmp2/")]


class HostPreopens:
    def get_directories(self):
        return []


def main():
    store = Store()
    demo = Root(
        store,
        RootImports(
            Host(),
            None,
            None,
            None,
            None,
            HostPreopens(),
            HostEnvironment(),
            None,
            None,
            None,
            None,
            None,
            None,
            None,
            None,
            None,
        ),
    )
    data = {"key": "value"}
    json_str = json.dumps(data)
    json_bytes = json_str.encode("utf-8")
    res = demo.exec(store, json_bytes)
    print(res.decode("utf-8"))


if __name__ == "__main__":
    main()

When trying to execute the main.py, I get an error indicating that I don't have access to the temporary directory. I am guessing that I need to implement the HostPreopens; however, I could not find documentation showing how to do so.

python main.py
{"key": "value"}hello from exectest
open /tmp2/test_0: errno 76: create temp

Is Preopens the right approach to solve the issue above? Where can I find documentation or example code regarding providing WASM component access to the file system?

Note: I put all the codes above, including the WIT-file, in this repo https://github.com/shihanng/play-wasm/tree/fs/wit for reproducibility.

Thank you.

[alexcrichton]

Thanks for the report! It looks like you're using the component support which I should give a warning is a little underbaked at the moment. For example as I'm sure you've found here you're left to implement WASI yourself which can be a bit of an undertaking. This is likely where much of the complexity lies, and building WASI from scratch will mean you're opening yourself up to a number of interesting failure modes to debug.

I'm not super familiar with TinyGo here, so I can't say for certain, but my guess is that the get_directories method you've implemented returns nothing but the guest is trying to open a file in /tmp2 via the environment variable you've configured. This means that the guest doesn't have access to open a directory which is probably where the error is coming from.

[shihanng]

@alexcrichton thank you for the reply and the insight. Do you know where I can find references or examples of how to implement the get_directories method properly? I realize that I do not see the message printed on the console when I include a print statement inside the method. This led me to think that the get_directories is not being called during execution.

class HostPreopens:
    def get_directories(self):
        print("in get dir")
        return []

[alexcrichton]

Sorry as I mentioned before this is all not a well-trodden path so I'm not aware of example or documentation to assist you. If that's not getting printed then I'm probably wrong about the source of the error. I can't access the repository link you posted above, so I don't know what the error is here.

[shihanng]

I can't access the repository link you posted above, so I don't know what the error is here.

@alexcrichton I am sorry 🙏 . I didn't realize that the repo I shared was private. I made it a public repo: https://github.com/shihanng/play-wasm

The code reproduces the abovementioned issue is in the wip/ directory.

I am using the following to create the WASM component and create the bindgen in the pyconvert/ directory.

cd wit
wasm-tools component embed --world convert . main.wasm -o main.embed.wasm
wasm-tools component new main.embed.wasm --adapt wasi_snapshot_preview1.reactor.wasm -o main.component.wasm
python -m wasmtime.bindgen main.component.wasm --out-dir pyconvert

The code that uses the WASM component is in wit/main.py. Running the code gives me the error open /tmp2/test_0: errno 76.

cd wit
python main.py
{"key": "value"}hello from exectest
open /tmp2/test_0: errno 76: create temp

[alexcrichton]

Ok I believe you're running into an issue with TinyGo where it doesn't fully support the "reactor" pattern today. I may be wrong, though, and my knowledge of TinyGo isn't great so it could be dated too. But I believe that historically the TinyGo runtime requires initialization through the _start function (probably not that function specifically but some other function too) which is not supported in custom entry points, such as the one for exec in the WIT that was generated.

I believe that because the runtime isn't fully initialized that various constructors aren't run which means that files don't work. I recall seeing a similar issue in a different repository, but I unfortunately can't remember where at this time.