Tool Calling with mlx-community/Meta-Llama-3.1-8B-Instruct-bf16

[Pentar0o]

Hey, I've succeeded to use tool calling with Ollama in python and wanted to use MLX instead but it seems that MLX don't support tools calling, is there any documentation available that could help ?

[madroidmaq]

@Pentar0o
Perhaps you can try the https://github.com/madroidmaq/mlx-omni-server project, which supports the use of OpenAI's sdk. The code is roughly as follows:

from openai import OpenAI

# Configure client to use local server
client = OpenAI(
    base_url="http://localhost:10240/v1",  # default port to mlx omni server
    api_key="mlx-omni-server",  # API key is not required for local server
)


def chat(model: str = "mlx-community/Llama-3.2-3B-Instruct-4bit"):
    tools = [
        {
            "type": "function",
            "function": {
                "name": "get_current_weather",
                "description": "Get the current weather in a given location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {
                            "type": "string",
                            "description": "The city and state, e.g. San Francisco, CA",
                        },
                        "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
                    },
                    "required": ["location"],
                },
            },
        }
    ]

    messages = [{"role": "user", "content": "What's the weather like in Boston today?"}]

    return client.chat.completions.create(
        model=model, messages=messages, tools=tools, tool_choice="auto"
    )


if __name__ == "__main__":
    completion = chat(model="mlx-community/Llama-3.2-3B-Instruct-4bit")
    message = completion.choices[0].message
    print(f"message: {message}")
    if message.tool_calls:
        print("success")
    else:
        print("failed")