[Proposal] Introduce a transpilable C# language/.NET runtime subset specialized for native library wrappers

[ceztko]

Summary:
A subset of the C# language and the .NET runtime is described and discussed. The purpose of this subset is to seamlessly transpile C# library wrappers that make use of .NET unmanaged interop to different languages/frameworks. The proof of concept tool CodeBinder is introduced together with examples of its current transpilation capabilities.

👉 Disclaimer:
I created this issue using the blank template as this is not a simple/self-contained API proposal, but a more complex one involving the runtime and also the language. I dare to CC @jkotas as the engineer that I noticed as the most involved in every aspect of .NET core/interop internals. Please feel free to convert this issue to a discussion and/or remove the CC if you find anything inappropriate. This proposal doesn't want to promote/advice usage of the CodeBinder tool which will continue to be developed privately, unless endorsed by a major organization/entity.

Background:
C# and the .NET runtime natively provide exceptional support for interop with native libraries through the C ABI, which is often used to create library wrappers. Other languages/frameworks provide solutions to call native code but unless they support it transparently (eg. ObjectiveC) it may be hard to create efficient trampolines to native code (eg. JNI for JRE, NAPI for NodeJS). Furthermore doing this for a big matrix of languages/runtimes may be time consuming and repetitive task. To ease and accelerate this process, it may be convenient to write the library wrappers in one environment (C#/.NET) and transpile it to other languages/frameworks (eg. Java JDK/Android, TypeScript on NodeJS, ObjectiveC/Swift, ...), implementing a write once-consume everywhere approach. Moreover, the interop entry points (DllImport or LibraryImport) can be used to generate definition templates in C/C++ or NativeAOT. This approach could also answer some open questions as what can/should be exposed in NativeAOT libraries.

The proposal:
To be able to successfully transpile C# code to other languages, a subset of the complex C# syntax should be picked. Among the many C# features the following ones would represent a solid and expressive subset:

• Basic flow control
• Arithmetic
• Classes/Structs/Interfaces with accessibility modifiers
• Enums
• Properties
• Finalization
• Non reified generics
• Native pass-by-ref parameters
• Enumeration
• Native Interop

As per the Runtime support, some basic structures/types (from simplest ones as IntPtr to List/Dictionary) would be handy to provide expressive interfaces, plus some commonly available methods such as Object.ToString().

Proof of Concept:
In the past 4 years, I worked on CodeBinder C# transpilation tool. The tool implements most ideas of the current proposal and targets the following languages/runtimes:

• Java/JNI (JDK/Android)
• ObjectiveC (can be also consumed from Swift)
• TypeScript (commonjs/ESModule, using NodeJS)

The following languages are supported to create implementation templates for native entry points:

• C/C++
• NativeAOT (experimental)

CodeBinder takes advantage of the partial types/methods features to selectively transpile fragments that are meant to be run only a particular framework/runtime. A sample library (a document format like parsing library) that shows most important transpile capabilities can be found here (a few class examples here, here and here). The result of the transpilation of this library can be found here. Here it follows few excerpts:

Java:

public class Document extends HandledObject<Document>
{
    PageCollection _pages;
    Metadata _metadata;

    static
    {
        Common.initLibrary();
    }

    public Document()
    {
        this(SLCreateDocument());
    }

    public static Document create(
            String filename,
            String password
        )
    {
        long ptr = SLPdfLoadFile(filename, password);
        Document doc = new Document(ptr);
        return doc;
    }

    // ...
}

TypeScript:

export class Document extends HandledObject<Document>
{
    private _pages: PageCollection = null!;
    private _metadata: Metadata = null!;

    static
    {
        Common.initLibrary();
    }

    static new()
    {
        return new Document(napi.SLCreateDocument());
    }

    static createWithFile(
        filename: string,
        password?: string | null
    ): Document
    {
        password = password ?? null;
        let ptr: number = napi.SLPdfLoadFile(filename, password);
        let doc: Document | null = new Document(ptr);
        return doc;
    }

    // ...
}

ObjectiveC (public header):

OCSAMPLELIBRARY_API @interface OCDocument : CBHandledObject
{
    @private OCPageCollection * _pages;
    @private OCMetadata * _metadata;
}

    - (id)init;

    + (OCDocument *)createWithFile:(NSString *)filename;

    // ...
}

These excerpt were generated starting from the following annotated C# class:

[Module("Document")]
public class Document : HandledObject<Document>
{
    PageCollection _pages;
    Metadata _metadata;

    static Document()
    {
        Common.InitLibrary();
    }

    [OverloadBinding("New", OverloadFeature.ParameterArity)]
    public Document()
        : this(SLCreateDocument())
    {
    }

    [OverloadBinding("WithFile", OverloadFeature.TypeMatch)]
    public static Document Create(string filename, string? password = null)
    {
        var ptr = SLPdfLoadFile(filename, password);
        var doc = new Document(ptr);
        return doc;
    }

    // ...
}

Limitations:
CodeBinder provides solutions to overcome limitations in the target languages, such as missing support for method overload and general limitations in finalization. Similarly to many complex projects done by a single individual, some initial design choices of CodeBinder didn't quite meet the initial expectations and would require polishing/rework. Also some features are not available (but still planned) in all languages/frameworks (eg. enumeration and delegates interop).

Possible integration with the .NET ecosystem:
In a vision of a possible integration of this proposal with the .NET ecosystem, the .NET SDK would feature a special project that can use only the subset of the C# language/.net runtime described above. The publish target would generate the native glue code (JNI/NAPI...) and the language wrappers in the target frameworks (Java/TypeScript...).

Alternative solutions:
A solution (node-api-dotnet) specifically suited to consume .NET/NativeAOT libraries in TypeScript was recently introduced and officially endorsed by Microsoft. The aim of this proposal and CodeBinder are similar but wider in scope compared to node-api-dotnet, as it specifically covers targeting more languages/frameworks. I'm currently not aware of other similar proposals to the one I described in this issue.

[ghost]

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Summary:
A subset of the C# language and the .NET runtime is described and discussed. The purpose of this subset is to seamlessly transpile C# library wrappers that make use of .NET unmanaged interop to different languages/frameworks. The proof of concept tool CodeBinder is introduced together with examples of its current transpilation capabilities.

👉 Disclaimer:
I created this issue using the blank template as this is not a simple/self-contained API proposal, but a more complex one involving the runtime and also the language. I dare to CC @jkotas as the engineer that I noticed as the most involved in every aspect of .NET core/interop internals. Please feel free to convert this issue to a discussion and/or remove the CC if you find anything inappropriate. This proposal doesn't want to promote/advice usage of the CodeBinder tool which will continue to be developed privately, unless endorsed by a major organization/entity.

Background:
C# and the .NET runtime natively provide exceptional support for interop with native libraries through the C ABI, which is often used to create library wrappers. Other languages/frameworks provide solutions to call native code but unless they support it transparently (eg. ObjectiveC) it may be hard to create efficient trampolines to native code (eg. JNI for JRE, NAPI for NodeJS). Furthermore doing this for a big matrix of languages/runtimes may be time consuming and repetitive task. To ease and accelerate this process, it may be convenient to write the library wrappers in one environment (C#/.NET) and transpile it to other languages/frameworks (eg. Java JDK/Android, TypeScript on NodeJS, ObjectiveC/Swift, ...), implementing a write once-consume everywhere approach. Moreover, the interop entry points (DllImport or LibraryImport) can be used to generate definition templates in C/C++ or NativeAOT. This approach could also answer some open questions as what can/should be exposed in NativeAOT libraries.

The proposal:
To be able to successfully transpile C# code to other languages, a subset of the complex C# syntax should be picked. Among the many C# features the following ones would represent a solid and expressive subset:

• Basic flow control
• Arithmetic
• Classes/Structs/Interfaces with accessibility modifiers
• Enums
• Properties
• Finalization
• Non reified generics
• Native pass-by-ref parameters
• Enumeration
• Native Interop

As per the Runtime support, some basic structures/types (from simplest ones as IntPtr to List/Dictionary) would be handy to provide expressive interfaces, plus some commonly available methods such as Object.ToString().

Proof of Concept:
In the past 4 years, I worked on CodeBinder C# transpilation tool. The tool implements most ideas of the current proposal and targets the following languages/runtimes:

• Java/JNI (JDK/Android)
• ObjectiveC (can be also consumed from Swift)
• TypeScript (commonjs/ESModule, using NodeJS)

The following languages are supported to create implementation templates for native entry points:

• C/C++
• NativeAOT (experimental)

CodeBinder takes advantage of the partial types/methods features to selectively transpile fragments that are meant to be run only a particular framework/runtime. A sample library (a document format like parsing library) that shows most important transpile capabilities can be found here (a few class examples here, here and here). The result of the transpilation of this library can be found here. Here it follows few excerpts:

Java:

public class Document extends HandledObject<Document>
{
    PageCollection _pages;
    Metadata _metadata;

    static
    {
        Common.initLibrary();
    }

    public Document()
    {
        this(SLCreateDocument());
    }

    public static Document create(
            String filename,
            String password
        )
    {
        long ptr = SLPdfLoadFile(filename, password);
        Document doc = new Document(ptr);
        return doc;
    }

    // ...
}

TypeScript:

export class Document extends HandledObject<Document>
{
    private _pages: PageCollection = null!;
    private _metadata: Metadata = null!;

    static
    {
        Common.initLibrary();
    }

    static new()
    {
        return new Document(napi.SLCreateDocument());
    }

    static createWithFile(
        filename: string,
        password?: string | null
    ): Document
    {
        password = password ?? null;
        let ptr: number = napi.SLPdfLoadFile(filename, password);
        let doc: Document | null = new Document(ptr);
        return doc;
    }

    // ...
}

ObjectiveC (public header):

OCSAMPLELIBRARY_API @interface OCDocument : CBHandledObject
{
    @private OCPageCollection * _pages;
    @private OCMetadata * _metadata;
}

    - (id)init;

    + (OCDocument *)createWithFile:(NSString *)filename;

    // ...
}

These excerpt were generated starting from the following annotated C# class:

[Module("Document")]
public class Document : HandledObject<Document>
{
    PageCollection _pages;
    Metadata _metadata;

    static Document()
    {
        Common.InitLibrary();
    }

    [OverloadBinding("New", OverloadFeature.ParameterArity)]
    public Document()
        : this(SLCreateDocument())
    {
    }

    [OverloadBinding("WithFile", OverloadFeature.TypeMatch)]
    public static Document Create(string filename, string? password = null)
    {
        var ptr = SLPdfLoadFile(filename, password);
        var doc = new Document(ptr);
        return doc;
    }

    // ...
}

Limitations:
CodeBinder provides solutions to overcome limitations in the target languages, such as missing support for method overload and general limitations in finalization. Similarly to many complex projects done by a single individual, some initial design choices of CodeBinder didn't quite meet the initial expectations and would require polishing/rework. Also some features are not available (but still planned) in all languages/frameworks (eg. enumeration and delegates interop).

Possible integration with the .NET ecosystem:
In a vision of a possible integration of this proposal with the .NET ecosystem, the .NET SDK would feature a special project that can use only the subset of the C# language/.net runtime described above. The publish target would generate the native glue code (JNI/NAPI...) and the language wrappers in the target frameworks (Java/TypeScript...).

Alternative solutions:
A solution (node-api-dotnet) specifically suited to consume .NET/NativeAOT libraries in TypeScript was recently introduced and officially endorsed by Microsoft. The aim of this proposal and CodeBinder are similar but wider in scope compared to node-api-dotnet, as it specifically covers targeting more languages/frameworks. I'm currently not aware of other similar proposals to the one I described in this issue.

Author:	ceztko
Assignees:	-
Labels:	area-Meta, untriaged
Milestone:	-

[jkotas]

There are many ways to do interop between .NET and other languages, there is no one right way to do interop. We like the variety of interop solutions that are available in .NET. At the same time, it is not feasible to have all solutions to be built into .NET SDK. Our strategy has been to enable the interop solution to be published as optional independent NuGet packages. I think an independent optional NuGet package would be a good way to expose the interop solution that you are proposing.

I do not see us defining a transpilable C# language subset. In the limit, all C# features can be transpiled to other language with enough effort. The transpiler implementation and the transpilable subset are tightly coupled together. You should be able to enforce the transpillable subset that you have defined via a Roslyn analyzer that comes as part of your package. It is very common pattern for interop generators to come with associated analyzer.

Thank you for sharing your proposal. I am going to convert this issue to a discussion since that feels more appropriate.

[ceztko]

Thank you Jan for the prompt answer.

there is no one right way to do interop [...] In the limit, all C# features can be transpiled to other language with enough effort

Fair enough, but since this proposal is about class libraries, the syntax of the different languages will pose a limit to the API surface that can be effectively exposed by the transpilation. Within the API implementation boundary, pretty much everything could be theoretically transpiled, agreed.

It is very common pattern for interop generators to come with associated analyzer.

I would be curious if you, or anybody else, can point me to one/few examples of such generators that got enough wide scope to target several languages/frameworks.

You should be able to enforce the transpillable subset that you have defined via a Roslyn analyzer that comes as part of your package

Yes, this is something I considered and in fact I have a validation step done with Roslyn analysis, but it's not integrated in the regular C# compilation. Also, I believe that for best results the C# compilation should happen against a reduced runtime/BCL, but here I am not aware if this could be implemented transparently by just plugging a simple nuget dependency or if this would require a customized C#/.NET project.

In conclusion, everything is subjected to resources available to be invested in some work. As a contractor, I reached the point that investing more in my technology would be inconvenient for my client in the cost/benefits balance. Unless this technology turns into something bigger, I may just end adding few features and then stall the development, or maybe replace my work with some other technology later (which, granted, it wouldn't be end of the world). I would be curious to know what made the cut for Microsoft to endorse node-api-dotnet and what would make the cut to endorse a project with a scope similar to my proposal.