Better documentation for *using* Jan

[OmegaLambda1998]

Hi, really appreciate this software! I've messed around with a few different ways of getting local LLM's up and running and this is by far the least painful. The purpose of this discussion is to suggest some ways in which the post-installation user experience can be improved.

As background on myself, I'd consider myself a technically compotent general-user, with minimal familiarity or experience with LLMs. As such, general computing tasks (building software from source, reading source code, managing configuration files, etc...) are well within my capabilities, but LLM specific tasks (picking a model, optimising inference / model / engine parameters, prompt engineering, etc...) are all new to me. Keeping in mind my undersampled data-point of one, here are some things I think would reduce the learning curve of Jan and better reach it's stated goal:

Our goal is to make it easy for a layperson1 to download and run LLMs and use AI with full control and privacy.

--- What Jan does well ---

First I want to start with what I think Jan does very well, and that is getting a general local LLM up and running with minimal effort. Contributing to this is:

1. Simple, cross-platform installation options
2. Painless gpu integration. Seriously, this is for the biggest highlight of Jan
3. "one-click" model installation, with reasonable recommended models

The end result of this is that to get a general LLM running on my laptop took about 30 minutes, most of which was waiting for the model to download.

-- What Jan could do better ---

The biggest challenge I've personally had is what to do after the general LLM is up and running. In other words, it is currently not obvious how to run LLMs and use AI with full controll. The source of this problem is that the documentation does very well explaining how to do something but not what to do and why to do it. To illustrate the issue, I want to walk through trying to get Jan to work as a coding assistant. Working through the Quickstart guide, the very first point of decision making is choosing and downloading a model.

Following the guide I goes to the hub and am presented with:

1. Recommended models they can download
2. Models I can use (not immediately obvious why these can be used immediately, trial and error revealed they are cloud models requiring an API key)
3. More models I can download
4. Some models I can download but am told I don't have enough VRAM
5. A way to import my own models, either locally or through huggingface

What I feel is missing from this page is what differenciates each model, and how and why to pick a model. To rectify this I think the docs could use some information on basic LLM nomenclature, in particular:

1. Mistral vs Llama vs Gemma vs Claude, etc... Who made these? How are they different? Why would I want one over the other?
2. What do the B Q submodel qualifiers actually mean?

In addition, I think the following would be useful, though possibly outside the scope of Jan / "not Jan's problem"

1. Given my hardware, what general advice is there for picking a model?
2. How do I test how well a model works?
3. What limitations do different models have that a user might not be aware of?
4. How well do different models perform at specific tasks? (maybe point towards the various leaderboards and describe briefly how to use them)

So, after doing my own research, I pick a model that others have said does well for code, and should run on my hardware. Note that model configuration does a pretty good job of describing what each parameter does, and generically why I'd want to set them to any specific value.

The next step in the quickstart is to customise the assistants instructions. Once again the guide does very well describe how to do this, but not what and why.

Here I think a (possibly user contributed) list of example instruction sets could be useful. Even just pointing the user to something like The Big Prompt Library, and providing two or three examples of how the assistant instructions change the experience could be enough.

As a side point, I think it's unclear how the assistant instructions are used. Would I have an equivalent experience if I just copy-pasted the instruction as the first prompt in a thread?

Finally I can start using Jan as a code assistant. If I find that it is not performing as well as I want, what do I do? The source of the issue could be:

1. The choice of model is ill-suited to my hardware
2. The choice of model (or underlying training data) is ill-suited to my stated goal
3. The model configuration is ill-suited to either my hardware or stated goal
4. The assistant instructions are ill-suited to my stated goal
5. Something else..?

Obviously LLM's are huge complex beasts so automating the troubleshooting of "I want it to code better" is not feasible, however providing some ideas / starting point for how to troubleshoot would be useful.

My final comment is that in addition to the quick start & basic usage sections, it would be could to provide step-by-step guides for getting Jan optimised for specific workloads. Going through how to get Jan working as a coding assistant, or a grammer checker & prose improver, or a storytelling and roleplay engine. Importantly these guides should include not only how to do this, but why decision were made for this usecase, which would allow a user to adapt the guide for their specific goal.

I hope this doesn't come of as harsh, I have been using Jan for weeks now and it's steadily replacing ChatGPT as my go to LLM. I wrote this up because I think local, private AI like Jan offers is incredibely important, and making it easier for a user (especially a layman as defined in the docs) to get Jan doing what they want, as well as it can is an important step in wider adoption

[thinwoodsman]

What if there was a button on the left, similar to the Hub one, which contained Task-specific prompts for all installed models?

Implementationwise, this would be built from a directory with per-Task prompts that models copy into, or by looking for prompt files (.pmt?) in each subdirectory of models ("jan/models/mammoth-7b.Q4_K_M/MathProblem.pmt).

Each entry could provide the prompt as well as examples for Chat and API usage, and there could be a button to start a new thread with that model and prompt.

This could be pretty useful, expecially when dealing with stuff like Chain-of-Thought reasoning, where each model might have a slightly different trigger phrase. Code generation is similar, sometimes you have to append "Let's write a program" to the query instead of starting with "Write me a program that..."

[tenten71]

I always vote for better/more documentation in the world. There's never enough especially in a more niche area where the "experts" assume everyone knows what to do. I wouldn't ask the developers to rewrite documentation of general AI terms but perhaps provide links of good source material to read or start up discussions her in Github or Discord for the community to sort of help fill in those questions for all new comers. (forgive me if they already exist but I just haven't gotten to them yet)