How to chat with your documents

System overview

UI & Frontend

const { getRootProps, getInputProps, open } = useDropzone({
   noClick: true,
   noKeyboard: true,
   onDrop: (acceptedFiles: File[]) => {
     setSelectedFiles(acceptedFiles);
   },
   multiple: true,
});

Document Parser

• Collection of utilities to read from different sources and extract raw text

• Ideally we should be able to parse multiple type of documents including, PDFs, MD, TXT, DOC, DOCX, etc.

Embedding Knowledge

1. Fine tuning: Re-train the original model to add more information
• This is usually requires more work and computing power since it involves almost a full flow of model training, including general ETL (Extraction, Transformation, Loading) of your data, hyper-parameter tuning, and enough iterations to be happy with the result.

2. Aggregate Context: Add more information as “context” to your prompt or question
• This approach give us less flexibility on re-shaping our model, but it’s cheaper and faster. It involves parsing our original text data into a vector representation (aka embeddings). This vector representation of our text is sent as special context for the model to take in consideration for reacting to your prompt or question.
• The main limitation of this approach is the max size of the “context” that you’re allow to inject through embeddings. The state of the art models today allow you to embed only a few pages of content at a time. This will likely be less of a limitation as models evolve.

Embedding flow

1. Chunk raw text: embeddings are generated by parsing a discrete size of text so we need to chunk our whole body of text first.

• Calculate Tokens (cost of embedding): Most services will charge your by “token” a discrete unite of measurement that usually maps amount of computation needed to run a task. We want to pre calculate the cost in order to avoid nasty surprises.

• Generate embedding: finally batch process all your chunks of text and generate the vector representations of your data.

• Storing your context: use a vector-based database to save your documents. By storing the vector representation of your data you only need to spend the parsing computation once and you can re-use it every time you ask a question to your chatbot.

// Load PDF files from the specified directory
const directoryLoader = new DirectoryLoader(filePath, {
  '.pdf': (path) => new PDFLoader(path),
  '.docx': (path) => new DocxLoader(path),
  '.txt': (path) => new TextLoader(path),
});

const rawDocs = await directoryLoader.load();

// Split the PDF documents into smaller chunks
const textSplitter = new RecursiveCharacterTextSplitter({
  chunkSize: Number(chunkSize),
  chunkOverlap: Number(overlapSize),
});

const docs = await textSplitter.splitDocuments(rawDocs);

// OpenAI embeddings for the document chunks
const embeddings = new OpenAIEmbeddings({
  openAIApiKey: openAIapiKey as string,
});

// Get the Pinecone index with the given name
const index = pinecone.Index(targetIndex);

// Store the document chunks in Pinecone with their embeddings
await PineconeStore.fromDocuments(docs, embeddings, {
  pineconeIndex: index,
  namespace: namespaceName as string,
  textKey: 'text',
});

Query Enhancing

Options for data ingestion

• Having an independent ingestion pipeline that regularly gets triggered with new data being dropped in a directory and produce the corresponding embeddings storing them in a database or file.

• Having a document parsing feature in-app for each user to drop their own files or URLs
• Generate embeddings every time document is loaded and store them locally
• Generate embeddings only once and store them in a vector database

The App

• Drop PDF section

• Store embeddings in Pinecone vector database

• Only generate new embeddings for new files

• Check for same file name & embeddings similarity

• Q&A Section
• Perform cosine distance calculations in the database to retrieve only relevant bits of content that can fit under the current limitations of context size

• Answers include source reference pointing to the original document, so that the user can always refer back to the original material

• Aggregate chat history to answer further questions in the same session

Acknowledgements