add gpt4 generated queries on test

clean rag training

Added fine-tuning python notebook for rag

add train rag file

Notebook for augmented dataset

Fix bug

Second attempt of dynamic rag

adding latest test dataset run

Adding test set generation python notebook, and correspond train and test dataset tsv files

Updated Augmentation Dataset with 250 verified queries

Notebooks for GPT evaluation

Fix typo in rag

Add second RAG notebook

Add changes to save new rag dadataframe

Add a rag helper notebook

Add a pre rag prompt

Added testing on small database to test_rag notebook, removed excess print statements

Updated compare_result function to allow passing the cursor, also re-ran test_pretrained and test_rag with updated loss function

Tweaked compare_result to work with proper directory, updated rag python notebook prompt and vector stores to increase accuracy

Updating compare result function in src directory, fixing small issues in test_rag prompt, adding test_rag notebook

fix cuda

Fix colab bugs

Prepare fine-tune for colab

updated rag evaluation function to match that from baseline testing

Update test_rag.py

Merge branch 'higher_rank' into main

finished finetuning with 16 rank lora

Fix bug on function evaluation

Upload data_augmentation.tsv file (still need to go through queries and check if they are valid)

Modify path to relative path

Add relative path functionality for colab

colab support part two

Add support for colab

Add a source folder

Move paht to a variable

Create demo_rag.py

Create rag_metadata.py

Create test_rag.py

Add column evaluator

added fine-tune testing on validation set with short queries only

Adding validation set testing of newly fine-tuned model

switch fine-tuned evaluation script to use 8 rank model

trained 8 rank LoRA finetuned model, saving train and validation datasets to disk

first cut at fine-tuning with masked training, working but very very slow for inference

fixed bug in test_finetuned notebook to read from proper directory

Storing 16 and 32 rank LoRA finetuned models

Increasing LoRA rank to 32 and updating loss function

first cut at adding data augmentation python notebook

added python notebook for testing finetuned model

First decent attempt, seems to generate valid queries after pre-training. Needs evaluation still