Skip to content
This repository has been archived by the owner on May 28, 2024. It is now read-only.

v0.3.0
Compare
Choose a tag to compare
@Yard1 Yard1 released this 02 Oct 23:25
· 70 commits to master since this release
v0.3.0
470a5e2
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Please note that API stability is not expected until 1.0 release. This update introduces breaking changes.

This release introduces a new vLLM backend and removes the dependency on TGI. This is because TGI is not Apache 2.0 licensed anymore, and the new license is too restrictive for most organizations to run in production. On the other hand, vllm is Apache 2.0 licensed and is a better foundation to build on top of. There are some breaking changes to model configuration YAMLs related to the new vLLM backend.

Refer to the updated ray-llm/models/README.md file for details on the updated configuration file format.

What's changed?

  • Documentation

    • Updated readme and documentation

  • API & SDK

    • Updated the format of model configuration yamls.

  • Backend

    • Completely replaced the text-generation-inference based backend with vLLM based backend. This means RayLLM now supports all models vLLM supports.
    • Improved observability and metrics.
    • Improved testing.

In order to use RayLLM, ensure you are using the official Docker image anyscale/aviary:latest.

Assets 2
Loading