diff --git a/img/blog/langchain4j-example-chat-room.png b/img/blog/langchain4j-example-chat-room.png new file mode 100644 index 000000000..e5647d1f0 Binary files /dev/null and b/img/blog/langchain4j-example-chat-room.png differ diff --git a/img/blog/langchain4j.png b/img/blog/langchain4j.png new file mode 100644 index 000000000..4891938a9 Binary files /dev/null and b/img/blog/langchain4j.png differ diff --git a/posts/2024-03-28-open-liberty-with-langchain4j-example.adoc b/posts/2024-03-28-open-liberty-with-langchain4j-example.adoc new file mode 100644 index 000000000..439a4eeaa --- /dev/null +++ b/posts/2024-03-28-open-liberty-with-langchain4j-example.adoc @@ -0,0 +1,357 @@ +--- +layout: post +title: "Run LangChain4j in Jakarta EE and MicroProfile application on Open Liberty" +# Do NOT change the categories section +categories: blog +author_picture: https://avatars3.githubusercontent.com/gkwan-ibm +author_github: https://github.com/gkwan-ibm +seo-title: Run LangChain4j in Jakarta EE and MicroProfile application on Open Liberty +seo-description: Experience AI technology in an Jakarta EE and MicroProfile application running on Open Liberty by using LangChain4j APIs. +blog_description: "Experience AI technology in an Jakarta EE and MicroProfile application running on Open Liberty by using LangChain4j APIs." +open-graph-image: https://openliberty.io/img/twitter_card.jpg +open-graph-image-alt: Open Liberty Logo +--- += Run LangChain4j in Jakarta EE and MicroProfile application on Open Liberty +Gilbert Kwan +:imagesdir: / +:url-prefix: +:url-about: / + +:example-location: https://github.com/langchain4j/langchain4j-examples/tree/main/jakartaee-microprofile-example + + +Artificial Intelligence (AI) is an exciting and disruptive field that is already transforming businesses, and even entire industries, by enabling automation, improving decision-making and unlocking new insights from data. With the rise in large language models, or LLMs, such as ChatGPT, over the last few years there has been a significant shift in the performance of AI and its potential to drive enterprise value. So, how will this impact on software development and the creation of cloud-native Java applications for enterprises? + +* <> +* <> +* <> +* <> + - <> + - <> + - <> +* <> + - <> + - <> + - <> + - <> +* <> +* <> + +[#what_Are_LLMs] +== What are LLMs? + +A language model is a model of natural language based on probabilities. They are able to generate probabilities of a series of words together. https://www.ibm.com/topics/large-language-models[Large language models (LLMs)] are simply langauge models that are categorized by their large size.They are trained on immense amounts of data, possibly billions of paramters, using self-supervised and semi-supervised learning techniques. This makes them capable of understanding and generating natural language and other types of content to perform a wide range of tasks, including: + +* *Text generation*: language generation abilities, such as writing emails, blog posts or other mid-to-long form content in response to prompts that can be refined and polished. An excellent example is retrieval-augmented generation (RAG). + +* *Content summarization*: summarize long articles, news stories, research reports, corporate documentation and even customer history into thorough texts tailored in length to the output format. + +* *AI assistants*: chatbots that answer customer queries, perform backend tasks and provide detailed information in natural language as a part of an integrated, self-serve customer care solution. + +* *Code generation*: assists developers in building applications, finding errors in code and uncovering security issues in multiple programming languages, even “translating” between them. + +* *Sentiment analysis*: analyze text to determine the customer’s tone in order understand customer feedback at scale and aid in brand reputation management. + +* *Language translation*: provides wider coverage to organizations across languages and geographies with fluent translations and multilingual capabilities. + + +See the introductory video below for more information on LLMs, including what they are, how they work and business applications. + + + +LLMs can be found in the service offerings of almost all of the major cloud service providers, e.g. IBM offers models through it's https://www.ibm.com/watsonx[watsonx] services, https://azure.microsoft.com/en-us/solutions/ai[Microsoft Azure] offers LLMs like Llama 2 and OpenAI GPT-4, and https://aws.amazon.com/bedrock/[Amazon Bedrock] offers models from a range of AI companies. + +[#Java_LLMs] +== How can we take advantage of LLMs in Java applications? + +Developing applications powered by large language models isn't always an easy task. Integrating AI/LLM capabilities into an application can be challenging. To make this easier, LangChain, an open source framework was developed in 2022 to help streamline the process of creating generative AI apps. + +LangChain provides tools and abstractions to improve the customization, accuracy, and relevancy of the information the models generate. For example, developers can use LangChain components to build new prompt chains or customize existing templates. LangChain also includes components that allow LLMs to access new data sets without retraining and organizes the large quantities of data these models require so that they can be accessed with ease. + +Although LangChain is primarily available in Python and JavaScript/TypeScript versions, there are options available to use LangChain in Java applications through community projects like https://github.com/langchain4j/langchain4j[LangChain4j]. LangChain4j APIs provide the ability to intergate LLMs into your Java application for different AI platforms, such as OpenAI and Hugging Face, etc. + +image::/img/blog/langchain4j.png[langchain4j GitHub README image,width=50%,align="center"] + +[#using_Langchain4j] +== How to use LangChain4j in a Jakarta EE and MicroProfile application + +Langchain4j has a very useful open source https://github.com/langchain4j/langchain4j-examples[langchain4j-examples] GitHub repositry where it stores example applications. However, although this repo provided useful examples of using LangChain4j in general Java apps, we could not find any examples showcasing how you could experience these AI technologies in a Jakarta EE/MicroProfile based application. So, we decided to build one ourselves called `jakartaee-microprofile-example` which can now be found in this https://github.com/langchain4j/langchain4j-examples[langchain4j-examples] GitHub repository. This demo application highlights how to use LangChain4j APIs in an application using Jakarta EE and MicroProfile on Open Liberty. + + +[#tryout] +== Try out the jakartaee-microprofile-example application + +To see how you could apply LangChain4j to your own Jakarta EE and/or MicroProfile application, check out this example project for yourself. + +[#pre-reqs] +=== Pre-requisites +Before you clone the application to your machine, install JDK 17 and ensure that your `JAVA_HOME` environment variable is set. You can make use of the https://developer.ibm.com/languages/java/semeru-runtimes/downloads[IBM Semeru Runtime] as your chosen Java runtime. This runtime offers performance benefits from deep technology investment in projects such as Eclipse OpenJ9 and is available across a wide variety of hardware and software platforms. To find out more about IBM Semeru Runtime, see https://openliberty.io/blog/2022/08/19/open-liberty-semeru-performance.html[Open Liberty and Semeru Runtimes, cloud-native performance that matters]. + +The application uses Hugging Face. You need to get a Hugging Face API Key: + +* Sign up and login to https://huggingface.co +* Go to Access Tokens by https://huggingface.co/settings/tokens +* Create a new access token with "read" role + +To access the repository remotely you'll need to install https://git-scm.com/book/en/v2/Getting-Started-Installing-Git[Git] if you haven't already. You can clone the `langchain4j-examples` GitHub repository to your local machine by running: + +[source] +---- +git clone https://github.com/langchain4j/langchain4j-examples.git +---- + +[#env_setup] +=== Environment Set Up + +To run the application, navigate to the `jakartaee-microprofile-example` directory: + +[source] +---- +cd langchain4j-examples/jakartaee-microprofile-example +---- + +and set the environment variables: + +[source] +---- +export JAVA_HOME= +export HUGGING_FACE_API_KEY= +---- + +[#start_app] +=== Start the application + +To start the application, use the provided Maven wrapper to run the https://openliberty.io/docs/latest/development-mode.html[Liberty dev mode]: + + +[source] +---- +./mvnw liberty:dev +---- + +After you see the following message, the application is ready: + +---- +************************************************************************ +* Liberty is running in dev mode. +* Automatic generation of features: [ Off ] +* h - see the help menu for available actions, type 'h' and press Enter. +* q - stop the server and quit dev mode, press Ctrl-C or type 'q' and press Enter. +* +* Liberty server port information: +* Liberty server HTTP port: [ 9080 ] +* Liberty server HTTPS port: [ 9443 ] +* Liberty debug port: [ 7777 ] +************************************************************************ +---- + +To ensure the application has started successfully, you can run the tests by pressing the `enter/return` key from the command-line session. If the tests pass, you should see a similar output to the following example: + +---- +[INFO] ------------------------------------------------------- +[INFO] T E S T S +[INFO] ------------------------------------------------------- +[INFO] Running it.dev.langchan4j.example.ChatServiceIT +[INFO] ... +[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.439 s... +[INFO] ... +[INFO] Running it.dev.langchan4j.example.ModelResourceIT +[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.733 s... +[INFO] +[INFO] Results: +[INFO] +[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 +---- + +[#access_app] +=== Access the application + +Once the application is running, you can access it through a browser of your choice at http://localhost:9080/ and start experimenting with it. + +image::/img/blog/langchain4j-example-chat-room.png[Chat Room of LangChain4j Jakarta EE and MicroProfile example,width=70%,align="center"] + +You can type in any text that you want to chat with the AI agent. Here are some suggested messages: + +* `What is MicroProfile?` +* `Which top 10 companies contribute MicroProfile?` +* `any documentation?` + + +[#how_app_work] +== How does the application work? + +The application demostrates how to use the LangChain4j APIs, https://openliberty.io/docs/ref/feature/#cdi-4.0.html[Jakarta Contexts and Dependency Injection], https://openliberty.io/docs/latest/reference/feature/websocket-2.1.html[Jakarta WebSocket], https://openliberty.io/docs/ref/feature/#mpConfig-3.0.html[MicroProfile Config], and https://openliberty.io/docs/latest/reference/feature/mpMetrics-5.1.html[MicroProfile Metrics] features. + +[#create_service] +=== Creating the LangChain4j AI service + +The application uses the `HuggingFaceChatModel` class to provide the model for building the AI service. + +See the {example-location}/src/main/java/dev/langchain4j/example/chat/ChatAgent.java[`src/main/java/dev/langchain4j/example/chat/ChatAgent.java`] file. +[source, java, role="no_copy"] +---- + public Assistant getAssistant() { + ... + HuggingFaceChatModel model = HuggingFaceChatModel.builder() + .accessToken(HUGGING_FACE_API_KEY) + .modelId(CHAT_MODEL_ID) + .timeout(ofSeconds(TIMEOUT)) + .temperature(TEMPERATURE) + .maxNewTokens(MAX_NEW_TOKEN) + .waitForModel(true) + .build(); + assistant = AiServices.builder(Assistant.class) + .chatLanguageModel(model) + .chatMemoryProvider( + sessionId -> MessageWindowChatMemory.withMaxMessages(MAX_MESSAGES)) + .build(); + ... + } +---- + +Through the customized {example-location}/src/main/java/dev/langchain4j/example/chat/ChatAgent.java[`Assistant`] interface, the application can send messages to the LLM by its `chat()` method. + +---- + interface Assistant { + String chat(@MemoryId String sessionId, @UserMessage String userMessage); + } +---- + +[#external_config] +=== Externalizing the configuration + +As the above code, to access the model, an API key is required. For security practice, it is not hard-coded in the code. The application externalizes the API key and the LangChain4j model properties with the MicroProfile Config feature that helps the application to run in different environments without code changes. You can learn more from the https://openliberty.io/docs/latest/external-configuration.html[External configuration of microservices] document. + +See the {example-location}/src/main/java/dev/langchain4j/example/chat/ChatAgent.java[`src/main/java/dev/langchain4j/example/chat/ChatAgent.java`] file. +[source, java, role="no_copy"] +---- + @Inject + @ConfigProperty(name = "hugging.face.api.key") + private String HUGGING_FACE_API_KEY; + + @Inject + @ConfigProperty(name = "chat.model.id") + private String CHAT_MODEL_ID; + + @Inject + @ConfigProperty(name = "chat.model.timeout") + private Integer TIMEOUT; + + @Inject + @ConfigProperty(name = "chat.model.max.token") + private Integer MAX_NEW_TOKEN; + + @Inject + @ConfigProperty(name = "chat.model.temperature") + private Double TEMPERATURE; + + @Inject + @ConfigProperty(name = "chat.memory.max.messages") + private Integer MAX_MESSAGES; +---- + +To fine tune the LangChain4j model or even try out another LLM, you simply update the values in the {example-location}/src/main/resources/META-INF/microprofile-config.properties[`langchain4j-examples/jakartaee-microprofile-example/src/main/resources/META-INF/microprofile-config.properties`] file or provide them through the enviroment variables. + +---- +hugging.face.api.key=set it by env variable +chat.model.id=NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO +chat.model.timeout=120 +chat.model.max.token=200 +chat.model.temperature=1.0 +chat.memory.max.messages=20 +---- + +[#communicate] +=== Communicating between the client and LLM + +The application provides the interactive UI client for users to communicate with the LLM. Jakarta WebSocket enables two-way communication between the client and the `ChatService` service. Each client makes an HTTP connection to the service and send out the messages by the `send()` method. + +See the {example-location}/src/main/webapp/chatroom.js[`src/main/webapp/chatroom.js`] file. +[source, java, role="no_copy"] +---- + const webSocket = new WebSocket('ws://localhost:9080/chat'); + ... + function sendMessage() { + ... + var myMessage = document.getElementById('myMessage').value; + ... + webSocket.send(myMessage); + ... + } +---- + +The service recieves the user messeages through the WebSocket `onMessage()` method, forward them to the LLM by calling the `ChatAgent.chat()` method, and then boardcast the LLM responsed answers back to the client session through the `sendObect()` method. + +See the {example-location}/src/main/java/dev/langchain4j/example/chat/ChatService.java[`src/main/java/dev/langchain4j/example/chat/ChatService.java`] file. +[source, java, role="no_copy"] +---- + @OnMessage + public void onMessage(String message, Session session) { + ... + try { + ... + answer = agent.chat(sessionId, message); + } catch (Exception e) { + ... + } + + try { + session.getBasicRemote().sendObject(answer); + } catch (Exception e) { + e.printStackTrace(); + } + + } +---- + +[#enable_metrics] +=== Enabling metrics + +To determine the performance and health of the application, it uses the MicroProfile Metrics feature to collect how much processing time is needed for a chat by applying the `@Timed` annotation to the `onMessage()` method. + +See the {example-location}/src/main/java/dev/langchain4j/example/chat/ChatService.java[`src/main/java/dev/langchain4j/example/chat/ChatService.java`] file. +[source, java, role="no_copy"] +---- + @OnMessage + @Timed(name = "chatProcessingTime", + absolute = true, + description = "Time needed chatting to the agent.") + public void onMessage(String message, Session session) { + ... +---- + +And, collect how many agents are created by applying the `@Counted` annotation to the `AgentManager.createAgent()` method. + +Visit the url http://localhost:9080/metrics?scope=application to check out the metrics. +---- +# HELP chatProcessingTime_seconds Time needed chatting to the agent. +# TYPE chatProcessingTime_seconds summary +chatProcessingTime_seconds{mp_scope="application",quantile="0.5",} 0.0 +chatProcessingTime_seconds{mp_scope="application",quantile="0.75",} 0.0 +chatProcessingTime_seconds{mp_scope="application",quantile="0.95",} 0.0 +chatProcessingTime_seconds{mp_scope="application",quantile="0.98",} 0.0 +chatProcessingTime_seconds{mp_scope="application",quantile="0.99",} 0.0 +chatProcessingTime_seconds{mp_scope="application",quantile="0.999",} 0.0 +chatProcessingTime_seconds_count{mp_scope="application",} 6.0 +chatProcessingTime_seconds_sum{mp_scope="application",} 31.674357666 +# HELP chatProcessingTime_seconds_max Time needed chatting to the agent. +# TYPE chatProcessingTime_seconds_max gauge +chatProcessingTime_seconds_max{mp_scope="application",} 13.191547042 +---- + +If you are interested in other ways to use the LangChain4j APIs, you can study the REST APIs that are provided by the {example-location}/src/main/java/dev/langchain4j/example/rest/ModelResource.java[`src/main/java/dev/langchain4j/example/rest/ModelResource.java`] file. + + +[#where_to_next] +== Where to next? + +Check out the https://openliberty.io/guides/[Open Liberty guides] for more information and interactive tutorials that walk you through using more Jakarta EE and MicroProfile APIs with Open Liberty. + +[#help_links] +== Helpful links +* link:https://github.com/langchain4j[LangChain4j] +* link:https://huggingface.co/models[Hugging Face LLMs] +* link:https://openliberty.io/guides/jakarta-websocket.html[Bidirectional communication between services using Jakarta WebSocket] +* link:https://openliberty.io/guides/cdi-intro.html[Injecting dependencies into microservices] +* link:https://openliberty.io/guides/microprofile-config.html[Configuring microservices] +* link:https://openliberty.io/guides/microprofile-metrics.html[Providing metrics from a microservice]