1.0.0-alpha.0 (zero-copy)

This is the first released version of chumsky's 'zero-copy' rewrite.

This release has no precise changelog, although one will be added when 1.0.0 is eventually released in full form. For now, things are still fluctuating enough that a full changelog would inevitably be out of date in a few weeks.

Thanks

This release has been over a year in development and represents the work of a lot of people. In particular:

• @CraftSpider, who effectively co-developed the rewrite with me and came up with large chunks of the core API
• @wackbyte, who ported many combinators over to the new codebase as well as adding no_std support
• @bew, who reworked many combinators around changes to the core API
• @Zij-IT, who ported all of the text combinators across, as well as the (yet to be merged) pratt parser combinator created by @alvra
• Many other contributors who worked on smaller items

How you can help

This is the first alpha release. Do not expect a finished product: many minor API features are still incomplete, missing, or subject to change. Some documentation is incomplete, or still refers to concepts from past versions of the crate. In particular, the tutorial has not yet been updated. You may experience bugs, API footguns, and more issues besides. That said, we're releasing this version because we believe the core of the rewrite is ready to be exposed to users and we want to find out what problems there are and catch them before a full release.

We'd like folks to open issues if they find:

• Bugs
• API oddities (things that don't look/feel right, or could be expressed more neatly)
• Things that feel like they should work, but don't (lifetime issues, unnecessary cloning, etc.)
• Missing features

If you're an even awesome-er sort of person and you feel like contributing to the crate, there's still a lot of work that needs doing in the following areas:

• Documentation
• Writing/updating examples
• Filling API 'holes'
• Porting old APIs over
• Small improvements to existing combinators
• Writing tests
• API design: there's still work to be done on the context-sensitivity, recovery, and iterable parser APIs

All this aside, you'd be helping us out a bunch just by using this alpha release (especially porting existing chumsky parsers over to it) and telling us how you got on: what worked, what didn't work, what things you got stuck on or confused by, etc. If you'd like to give a more casual report like this, feel free to start a discussion.

What's new?

Needless to say, the crate has received a substantial upgrade, overhauling virtually every aspect of its API. It's substantially more capable than it ever was, and now supports the following:

• Zero-copy parsing: parser outputs can hold references to the input
• Nested parsing: parsers can handle nested data structures like token trees
• Stateful parsing: parsers can be parameterised by state, allowing for the natural integration of arena allocators, string interners, etc.
• Memoisation: parsers can opt into memoisation, allowing you to quickly parse awkward grammars that would normally produce exponential behaviour in a traditional recursive descent parser
• Left recursion: the aforementioned memoisation feature can also properly handle left recursive grammars elegantly
• Context-sensitive parsing: parsers can use built-in context sensitivity to carefully parameterise future parsers, allowing you to parse things like Rust-style raw strings, Pythonic indentation, and other context-sensitive syntax that context-free parsers traditionally struggle with
• Iterable parsers: parsers that produce multiple outputs can now be turned into iterators, similar to logos lexers

Performance

On top of all of that, we've worked really hard to push performance as far as we can using an innovative use of Generic Associated Types (GATs) internally that allows chumsky to automatically detect when an output is never used (such as with .then_ignore(...)) and avoid generating it all in the first place. You can find some technical details about this approach in Niko Matsakis' blog where they discuss chumsky.

Our work on performance has paid off: chumsky's JSON benchmark is now extremely competitive, beating out nom and others, and even banging on the door of more traditional hand-written JSON parsers.

In general, you can probably expect this new release to be several times faster than older releases for similar parsers. The JSON benchmark is about 12x faster.

Conclusion

Pushing zero-copy to the point of a release was always going to be a very long road to walk, but we're finally approaching the end. Thanks for using chumsky, and - if you're fortunate enough to have the resources and kind enough to consider donating them - please support the other contributors I listed at the top of this release!