Automating Knowledge Base Creation for Custom LLM Chatbots

Introduction

In today's fast-paced world, automation plays a crucial role in simplifying complex tasks and improving efficiency. One such area where automation has made significant strides is in the creation of knowledge bases for custom Language Model (LLM) chatbots. This blog explores the general idea behind automating the process of creating a knowledge base for a hotel reservation chatbot and highlights the benefits of using a website to streamline this process.

The Concept

The concept revolves around a webpage that allows developers to input information about hotels. This information is then collected and used to generate text-based descriptions of the hotels. These descriptions serve as the foundation for creating an OpenAPI embedding, which is a standardized format for representing data. The OpenAPI embeddings are then utilized by a customized chatbot to facilitate hotel reservations/inquiries etc

Hotel KB Creator

The details Collector webpage

In this, we collect information from a new hotel, details about its room and amenities. This data is stored in my MongoDB Collection.

 router.route("/add").post((req,res)=>{
     // console.log(req.body);
     const name = req.body.hotel_name;
     const address = req.body.hotel_address;
     const phone = req.body.hotel_phone;
     const email = req.body.hotel_email;
     const website = req.body.hotel_website;
     const description = req.body.hotel_description;
     const rooms = JSON.parse(req.body.roomsData);
     const amenities = req.body.amenities;
     const nearbyAttractions = req.body.attractions;
     async function addHotel(){
         try {
             // Insert rooms into the database
             const insertedRooms = await Room.insertMany(rooms);

             // Extract the ObjectIds of the inserted rooms
             const roomIds = insertedRooms.map(room => room._id);

             // Create a new hotel with room ObjectIds
             const newHotel = new Hotel({
                 name,
                 address,
                 phone,
                 email,
                 website,
                 description,
                 rooms: roomIds, // Assign the array of room ObjectIds
                 amenities,
                 nearbyAttractions
             });

             // Save the new hotel to the database
             await newHotel.save();

             console.log('Hotel and rooms created successfully');
         } catch (error) {
             console.error('Error creating hotel:', error);
         }
     }
     addHotel();
 })

Based on this data, I created a blog of the hotel in a template file and expose it in a route '/blogs/'

 router.route("/blogs").get(async (req, res) => {
     try {
         // Fetch all hotels and wait for the result
         const hotels = await Hotel.find();

         let data = [];

         // Process each hotel
         for (const hotel of hotels) {
             let rooms = [];

             // Fetch rooms for the current hotel
             for (const room of hotel.rooms) {
                 const roomData = await Room.findById(room);

                 if (roomData) {
                     rooms.push(roomData);
                 }
             }

             const template = `
                 The name of the hotel is ${hotel.name}.
                 The hotel is located at ${hotel.address}.
                 You can contact the hotel at ${hotel.phone}.
                 Send inquiries to ${hotel.email}.
                 Visit the hotel website at ${hotel.website}.
                 Description: ${hotel.description}.
                 The rooms available are: ${rooms.map(room => room ? room.name : 'N/A').join(', ')}.
             `;

             data.push(template);
         }

         res.json(data);
     } catch (error) {
         console.error("Error fetching data:", error);
         res.status(500).json({ error: "Internal server error" });
     }
 });

Creating OpenApi Embeddings from the blog data.

 import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
 import { createClient } from "@supabase/supabase-js";
 import { SupabaseVectorStore } from "langchain/vectorstores/supabase";
 import { OpenAIEmbeddings } from "langchain/embeddings/openai";
 const dotenv = require("dotenv");
 dotenv.config();
 async function ingestData() {
     let text_hotel_data = ""
     axios.get("hotels/blogs").then((res)=>{
     text_hotel_data = res.body;
     }).catch((err)=>{
     console.log(err)
     })
     const client = createClient(
         process.env.SUPABASE_URL,
         process.env.SUPABASE_PRIVATE_KEY,
     );

     const splitter = new RecursiveCharacterTextSplitter({
         chunkSize: 100,
         chunkOverlap: 10,
     });

     const splitDocuments = await splitter.createDocuments([text_hotel_data]);

     const embeddings = new OpenAIEmbeddings({
         modelName: "text-embedding-ada-002",
         apiKey: process.env.OPENAI_API_KEY,
     });

     const vectorstore = await SupabaseVectorStore.fromDocuments(
         splitDocuments,
         embeddings,
         {
             client,
             tableName: "documents",
             queryName: "match_documents",
         }
     );

     return { ok: true };
 }

 ingestData();

Streamlining the Process

By leveraging automation, the process of creating a knowledge base for a custom LLM chatbot becomes more efficient and less time-consuming. The website acts as a centralized platform where developers or hotel owners can easily input hotel information. This streamlining of the process ensures that the chatbot has access to up-to-date and accurate information, enhancing the overall user experience.

Benefits of Automation

Time-saving: Automating the knowledge base creation process reduces the time required to gather and organize hotel information. This allows businesses to focus on other critical aspects of their operations.
Accuracy: Manual data entry is prone to errors, which can lead to incorrect or outdated information. Automation minimizes the risk of human error, ensuring that the chatbot provides accurate and reliable information to users.
Scalability: As the number of hotels increases, manually managing the knowledge base becomes increasingly challenging. Automation enables seamless scalability, allowing businesses to effortlessly handle a growing number of hotels without compromising efficiency.
Consistency: By automating the process, the generated text-based descriptions and OpenAPI embeddings maintain a consistent format, ensuring a cohesive user experience across different hotels.

Conclusion

Automating the creation of a knowledge base for a custom LLM chatbot brings numerous benefits to the hotel reservation process. By utilizing a website to streamline the collection and organization of hotel information, businesses can save time, improve accuracy, and enhance the overall user experience. This automation paves the way for efficient and scalable chatbot solutions, revolutionizing the way hotel reservations are made.