Interacting with Ollama with Langchain (Python)

In my previous blog post I installed Ollama locally so that I could play around with Large Language Models (LLMs). I used Python with requests to do a test of the LLM, but also wrote that there are Ollama Python libraries available. In this blog post I’m taking a look at a Python LLM library called Langchain.

Note: If you want to delve straight into creating a LLM chatbot then I recommend Real Python’s tutorial.

Model Change: tinyllama to gemma

I’ve changed from the tinyllama model to gemma2, specifically the gemma2:2b model. The gemma2 model is a Google model and the 2b variant has roughly 2 billion parameters. These extra parameters do come at a cost though as gemma2 is 1.6GB compared to tinyllama’s 640MB.

Environment Variables

I’m using a .env file to store the LLM model name and LLM URL location, with Python’s dotenv and os grabbing the values. The .env file currently has two lines in:

LLM_URL="http://IP-OF-LLM:PORT"
LLM_MODEL="gemma2:2b"

Storing these in an environment (.env) file allow me to quickly update settings in one place and will allow me to exclude the .env from future Git actions, e.g. to stop the values from being stored in one of my Git repositories.

Langchain

As I’m using Ollama as my local LLM and not one of the public based LLMs (e.g. ChatGPT) I made some changes to the Real Python based guide. The following Python libraries are used and can be installed (preferably in a virtual environment) using a pip install:

pip install python-dotenv
pip install langchain
pip install langchain_ollama

I have then used the base_url option of the OllamaLLM to let it know where to find Ollama.

chat_model = OllamaLLM(model=os.getenv('LLM_MODEL'),
                       base_url=os.getenv('LLM_URL'))

After some interaction via the Python REPL I altered the code so that the Python file could handle interaction when run rather than having to be imported.

import dotenv
import os
from langchain_ollama import OllamaLLM

dotenv.load_dotenv()

chat_model = OllamaLLM(model=os.getenv('LLM_MODEL'),
                       base_url=os.getenv('LLM_URL'))

Human_Question = input("What do you want to ask Ollama? ")

SystemMessage = "You are a assistant knowledgable in space. 
                Only answer questions related to space. 
                If the question is not related to space 
                then reply with 'I don't know'."

messages = [
    SystemMessage,
    Human_Question,
]

print(chat_model.invoke(messages))

This gave the LLM a restriction around what it can and cannot answer questions on, although this would need a lot more testing to see if it held up to hacking attempts.

Langchain has a templates to help with prompts and I’ll be looking at them next. Again, if you are after creating an LLM chatbot in one go, then head over to Real Python.