Table of Contents

Have you ever wanted to run your own AI bot for personal use? With the available llama.cpp model developed by Facebook, it is possible to bind using go-llama.cpp.

go-llama.cpp is a binding module written in golang. According to their documentation, the binding are high level - coded in C/C++ for higher response. It is possible to run AI bot using CPU and GPU. But the recommended setup would be to have a dedicated GPU with a CPU with higher frequency machines.

This makes self hosting AI almost impossible because data models require very high RAM memory and CPU power. However, there are some data models that can run on comparitively low RAM >=8GB (at least).

Full credits to thedmdim for coding this in golang.

Prerequisites:

  • RAM >=8GB
  • CPU would max at 100%
  • Docker
  • Telegram bot
  • Data models

Installation:

Make sure you have the latest version of docker and docker compose.

We need data model .bin files which have the dataset for the bot to process. You can download them from HuggingFace. For this demonstration, I would be using wizardLM-7B.ggmlv3.q4_1.bin which runs perfectly for low RAM machines.

Download the file and move it to models folder. wget https://huggingface.co/TheBloke/wizardLM-7B-GGML/resolve/main/wizardLM-7B.ggmlv3.q4_1.bin

Create a telegram bot using Botfather and obtain bot token.

Clone the repo,

git clone https://github.com/thedmdim/llama-telegram-bot

cp .env.example .env and edit .env

TG_TOKEN=REPLACE_WITH_BOT_TOKEN_FROM_BOTFATHER
# Replace the exact path of the model file
MODEL_PATH=/models/wizardLM-7B.ggmlv3.q4_1.bin

## optional, test as required.
# N_TOKENS=1024
# Q_SIZE=1000
# N_CPU=8

Edit docker-compose.yml file as you see fit, here is mine.

version: '3.9'

services:
  bot:
    image: thedmdim/llama-telegram-bot:latest
    privileged: true
    container_name: llama-telegram-bot
    restart: unless-stopped
    volumes:
      - ${MODEL_PATH}:${MODEL_PATH}
    env_file:
      - .env

Now run, docker compose up -d. Done, now your telegram bot should be up and running.

Type anything you like, the bot should respond back to you.

Here is a sample summary of an article,

Here is a sample summary of an article

Reference:

https://github.com/thedmdim/llama-telegram-bot

https://t.me/telellamabot – a demo instance of this bot