Friendly schema/spec for code generator (breaking change)

[davidB]

Actual Context

SDKs provide

• data model (struct, class, ...)
• serializer: data model -> json
• deserializer: json -> data model
• validation (on json or data model or type constraint to never create invalid data model)
• tests (example):
  • property: random data model equal deserialize(serialize(random data model))
  • property: serialize(random data model) match schema
  • json examples match schema
  • deserialize(json examples) generate valid data models

SDK Language: Java, Go, Python, Rust

Some issues preventing to use generic code generator for SDK:

• multiple json schemas (generator often takes one schema to generate a set of types and a single serializer/deserializer)
• duplicate types between schemas (context, url, purl, ...), but we don't want to generate them several times in SDK
• use of custom "primitive" types not supported by default in programming languages: purl, non-empty string, URI & URI-Reference, timestamp (RFC 3339 format start to be supported by generators)

Flow

flowchart LR
    A["Schema"]
    B["SDK language X"]
    C["Documentation (markdown)"]
    E["JSON Example"]
    T["Test for SDK language X"]
    C -- manual --> B
    A -- full custom --> B
    C -- manual --> A
    C -- manual --> E
    A -- full custom --> T
    B -- full custom --> T
    E -- full custom --> T

Loading

Proposals

Possible changes in the approach

• Using tagged union/polymorphism and having
  • a single top level event
  • a single definition of context (like in the markdown spec)
• Make SDK validations pluggable - e.g. hooks AKA handlers

Maybe a better flow & requirements

flowchart LR
    A["Schema"]
    B["SDK language X"]
    C["Documentation (markdown)"]
    E["JSON Example"]
    T["Test for SDK language X"]
    A -- semi custom --> C
    A -- semi custom --> B
    A -- semi custom --> E
    A -- full custom --> T
    B -- full custom --> T
    E -- full custom --> T

Loading

Data (event) should:

• wire friendly
• convertible into json (for display,...)

Schema language should:

• expressive for all the constraints & rules
  • allow documentation (field, structure)
  • allow example
  • "typed string" (eg: uri, datetime, pattern)
  • range
  • enumeration (simple)
  • composition / hierarchy / (tagged) union
• generators for programing language (maybe not allow / support all constraints)
  • target lang: java / kotlin, python, go, rust, typescript, swift
  • structure
  • decoder (wire format to programming language structure): deserializer, unmarshaller
  • encoder (programming language structure to wire): serializer, marshaller
  • validation
  • NICE: sample generation
  • NICE: code completion
• cli to work on the wire format
  • compatibility check (backward/forward compatibility check, also for versioning suggestion)
  • validation of schema, and data against schema
  • sample generation
  • documentation generation
  • NICE: formatter
• versioning
  • allow to flag "deprecated" fields, structure
  • allow to detect compatibility issue?

Options for schema

• json-schema (today)
• model from openapi (json-schema + polymorphism + ...)
• message from asyncapi
• JSON TypeDef: Schemas for JSON optimized for code generation
• protobuf IDL
• smithy
• apache avro
• apache parquet
• apache arrow
• apache orc
• cuelang
• pkl
• hcl
• dhall

Matrix to compare options