second commit

answers/answer_what_is_the_nature_of_consciou.json

@@ -0,0 +1,31 @@
+{
+  "question": "What is the nature of consciousness?",
+  "answer_passages": [
+    {
+      "passage_number": 1,
+      "book": "Come Follow To You - Vol 2 - Osho.pdf",
+      "text": "consciousness\tis\tjust\ta\twave\ton\tthe\tocean.\tThe\tunconscious\tis\tvast.\"\tThen\tthe\nego\tdisappears.\tEgo\tis\tthe\tpart,\tthinking\titself\tto\tbe\tthe\twhole.\tNon-ego\tis\tthe\npart,\tbecoming\taware\tof\tthe\twhole.\tThen\tthe\tego\tdisappears.\nHow\tto\tdefine\tthe\tnature\tof\tconsciousness?\tIt\thas\tnever\tbeen\tdefined,\tit\tnever\nwill\tbe\tdefined.\tWho\twill\tdefine\tit?\tTo\tdefine\tit\tyou\thave\tto\tbe\taway\tfrom\tit.\tTo\ndefine\tanything\tyou\thave\tto\tstand\tout\tof\tit,\tyou\tneed\ta\tdistance.\tPerspective\twill\nnot\tbe\tpossible\tif\tthe\tdistance\tis\tnot\tthere.\nYou\tare\tconsciousness,\tyou\tARE\tunconsciousness.\tThere\tis\tnobody\twho\tcan\nstand\toutside\tand\tdefine\tit.\tYou\tcan\tknow\tit,\tbut\tyou\tcannot\tdefine\tit.\tThat's\twhy\nall\treligion\tis\tmysterious,\tmystical,\tvague,\tcloudy\t--\tbecause\tno\tterm\twhich\tis\nvery\tbasic\tto\treligion\tcan\tbe\tdefined.\nThe\tsubject\tcannot\tbe\tmade\tan\tobject.\tI\tcannot\tput\tmyself\tin\tfront\tof\tme,\tso\tI\ncannot\tdefine.\tNeither\thas\tBuddha\tdefined,\tnor\tJesus.\tDefinition\tas\tsuch\tis\ndebarred\tby\tthe\tvery\tnature\tof\tthe\tphenomenon."
+    },
+    {
+      "passage_number": 2,
+      "book": "Come Follow To You - Vol 2 - Osho.pdf",
+      "text": "singing;\tcarry\tthe\tcross\tbut\tcarry\tit\twith\ta\tdeep\tcelebration\twithin.\tThen\tyou\tlive\nboth:\tyou\tlive\tlife,\tyou\tlive\tdeath.\tAnd\tyou\tlive\tboth\tof\tthem\tdeeply\tand\nintensely.\tWhen\tyou\tcan\tlive\tboth\tintensely,\tthey\tbecome\tone.\tThen\tyou\tknow\nthat\tlife\tand\tdeath\tare\ttwo\taspects\tof\tthe\tsame\tthing,\tof\tthe\tsame\tenergy.\tLife\tis\nexpression,\tmanifestation.\tDeath\tis\ta\treturning.\nQuestion\t5\nWOULD\tYOU\tDEFINE\tAND\tDISCUSS\tTHE\tNATURE\tOF\nCONSCIOUSNESS?\nHOW\tDOES\tCONSCIOUSNESS\tRELATE\tTO\tEGO?\tIS\tCONSCIOUSNESS\nTHE\nCREATIVE\tPRINCIPLE?\t(THAT\tIS,\tCOULD\tYOU\tSAY\tEQUALLY,\t\"IN\tTHE\nBEGINNING\tWAS\tCONSCIOUSNESS\"\tINSTEAD\tOF\t\"IN\tTHE\tBEGINNING\nWAS\tTHE\tWORD\tOR\tTHE\tLOGOS\"?)\tHOW\tDOES\tCONSCIOUSNESS\nRELATE\nTO\tGOD?\nIn\tthe\tBEGINNING\tWAS\tTHE\tWORD,\tor\tthe\tlogos.\tThe\tsame\tcannot\tbe\tsaid\nabout\tconsciousness,\tbecause\tin\tthe\tbeginning\tunconsciousness\twas\talso\tthere.\nConsciousness\tis\tjust\ta\tpart\tof\tyour\treality,\tthe\treality\tof\tthe\twithin.\nUnconsciousness\tis\talso\tthere.\tSo\tjust\tconsciousness\twas\tnot\tthere\tin\tthe"
+    },
+    {
+      "passage_number": 3,
+      "book": "A Cup of Tea.pdf",
+      "text": "washes them out constantly with every movement,\nand when the whole body is alive\nonly then you begin to feel the cosmic\nconsciousness all around you.\nHow can a frozen consciousness,\nand that too surrounded by a dead body,\nA Cup of Tea 219 Osho\nCHAPTER 1.\nfeel the cosmic?\n228\nNow man knows more about man than ever\nand yet no problem is solved.\nIt seems that something is basically wrong\nwith our so-called knowledge itself.\nThis whole knowledge is derived from analysis\nand analysis is incapable of penetrating\ninto the depths of consciousness.\nThe analytical method is all right with matter or with things\nbecause there is no inside to them,\nbut consciousness is insideness,\nand to use the analytical method with consciousness\nis to treat it as an object\nwhile it is not an object at all.\nAnd it cannot be made an object;\nits very nature is subjectivity,\nits being is subjectivity,\nso it must not be approached from outside\nbecause then whatsoever is known about it is not about it."
+    },
+    {
+      "passage_number": 4,
+      "book": "Blessed Are the Ignorant.pdf",
+      "text": "the conscious is just the tip of the iceberg.\nNow a second discovery is coming closer every day. It is not good to call it a discovery – it\nis a rediscovery, because yoga has known it always. Just as below the conscious there is the\n137\nCHAPTER 22. THERE IS NO NEED TO SEEK HAPPINESS – JUST START LIVING IT\nunconscious, above the conscious there is the superconscious. Just as the unconscious is the dark\nnight, the superconscious is pure light. And all the experiences of the mystics who talk about god\nas pure light, are nothing but the experiences of the superconscious.\nY ou are just in the middle – everybody is just in the middle. The conscious is the link, the bridge,\nbetween the unconscious and the superconscious.\nThe unconscious is the whole of nature, and the superconscious is god. In between the two is\nthe man – just a wavering, a continuous wavering to be this or to be that, to be or not to be... a"
+    },
+    {
+      "passage_number": 5,
+      "book": "Come Follow To You - Vol 2 - Osho.pdf",
+      "text": "is\tjust\ton\tthe\tsurface;\tdeeply\thidden\tare\tlayers\tand\tlayers\tof\tunconsciousness.\nOne\thas\tto\ttranscend\tboth\tto\tknow\tthat\twhich\twas\tin\tthe\tbeginning\t--\twhich\tis\nGod.\n\"Would\tyou\tdefine\tand\tdiscuss\tthe\tnature\tof\tconsciousness?\tHow\tdoes\nconsciousness\trelate\tto\tego?\t\"\tOne\tpart\tof\tyou\tis\tconscious,\tone-tenth.\tNine-\ntenths\tof\tyou\tis\tunconscious.\tIf\tthe\tconscious\tpart\tthinks\titself\tto\tbe\tthe\twhole,\tit\nbecomes\tthe\tego.\tThen\tit\tforgets\tabout\tthe\tunconscious;\tthen\tthe\tpart\timagines\nitself\tto\tbe\tthe\twhole.\tThen\tit\tis\tthe\tego.\nIf\tthe\tconscious\tbecomes\taware\tof\tthe\tunconscious\talso....\tThat\tis\tthe\twhole\neffort\tof\treligion,\tthat\tis\tthe\twhole\teffort\tof\tmeditation.\tIf\tthe\tconscious\tturns\nback,\tlooks\tback,\tand\tbecomes\taware\tof\tthe\tunconscious\talso\t--\tthe\tdark\tnight\nwithin\t--\nthen\tthe\tconscious\tknows\tthat\t\"I\tam\tconscious,\tI\tam\tunconscious\talso.\tMy\nconsciousness\tis\tjust\ta\twave\ton\tthe\tocean.\tThe\tunconscious\tis\tvast.\"\tThen\tthe\nego\tdisappears.\tEgo\tis\tthe\tpart,\tthinking\titself\tto\tbe\tthe\twhole.\tNon-ego\tis\tthe"
+    }
+  ],
+  "total_passages": 5
+}

create_vector_db.py

@@ -5,6 +5,7 @@ from langchain.text_splitter import RecursiveCharacterTextSplitter
 import chromadb
 from chromadb.utils import embedding_functions
 from tqdm import tqdm
+import torch
 
 class PDFVectorizer:
     def __init__(self, pdf_dir: str, db_dir: str):
@@ -17,8 +18,12 @@ class PDFVectorizer:
         )
         # Initialize ChromaDB with sentence-transformers embeddings
         self.client = chromadb.PersistentClient(path=db_dir)
+        # Check if GPU is available
+        self.device = 'cuda' if torch.cuda.is_available() else 'cpu'
+        print(f"Using device: {self.device}")
         self.embedding_function = embedding_functions.SentenceTransformerEmbeddingFunction(
-            model_name="all-MiniLM-L6-v2"
+            model_name="all-MiniLM-L6-v2",
+            device=self.device
         )
         self.collection = self.client.create_collection(
             name="osho_books",

osho_qa_service.py

@@ -0,0 +1,142 @@
+# Suppress warnings - must be before any imports
+import os
+os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
+os.environ['TOKENIZERS_PARALLELISM'] = 'false'
+os.environ['TF_ENABLE_ONEDNN_OPTS'] = '0'
+
+import warnings
+import logging
+# Suppress all warnings
+warnings.filterwarnings('ignore')
+# Specific suppressions
+warnings.filterwarnings('ignore', category=UserWarning)
+warnings.filterwarnings('ignore', category=DeprecationWarning)
+warnings.filterwarnings('ignore', category=FutureWarning)
+warnings.filterwarnings('ignore', message='.*benefit from vacuuming.*')
+warnings.filterwarnings('ignore', message='.*sparse_softmax_cross_entropy.*')
+
+# Suppress all logging
+logging.getLogger().setLevel(logging.ERROR)
+# Suppress TensorFlow logging
+logging.getLogger('tensorflow').setLevel(logging.ERROR)
+os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
+
+import json
+from typing import Dict, List
+import chromadb
+from chromadb.utils import embedding_functions
+
+def clean_text(text: str) -> str:
+    """Clean the text by removing extra spaces and formatting."""
+    # Remove multiple spaces
+    text = ' '.join(text.split())
+    # Remove unnecessary line breaks
+    text = text.replace('\n', ' ')
+    
+    # Remove text before first complete sentence
+    if '.' in text:
+        # Split by period and remove any incomplete sentence at start
+        sentences = text.split('.')
+        # Remove first part if it seems like a partial sentence
+        if len(sentences) > 1:  # Only if there are multiple sentences
+            sentences = sentences[1:]  # Remove first part
+        text = '.'.join(sentences)
+        text = text.strip()  # Remove leading/trailing whitespace
+        if text:  # Add period back if text is not empty
+            text += '.'
+    
+    return text
+
+def get_answer_from_osho(question: str, n_results: int = 5) -> Dict:
+    """
+    Get answer from Osho's books based on the question.
+    
+    Args:
+        question (str): The question to ask
+        n_results (int): Number of relevant passages to return
+        
+    Returns:
+        Dict: A dictionary containing the question and formatted answer with sources
+    """
+    # Initialize ChromaDB client
+    db_dir = os.path.join(os.getcwd(), "vector_db")
+    client = chromadb.PersistentClient(path=db_dir)
+    
+    # Initialize embedding function
+    embedding_function = embedding_functions.SentenceTransformerEmbeddingFunction(
+        model_name="all-MiniLM-L6-v2"
+    )
+    
+    # Get the collection
+    collection = client.get_collection(
+        name="osho_books",
+        embedding_function=embedding_function
+    )
+    
+    # Query the collection
+    results = collection.query(
+        query_texts=[question],
+        n_results=n_results
+    )
+    
+    # Format the answer
+    answer_parts = []
+    for i, (doc, metadata) in enumerate(zip(results['documents'][0], results['metadatas'][0])):
+        answer_part = {
+            "passage_number": i + 1,
+            "book": metadata['book'],
+            "text": clean_text(doc.strip())
+        }
+        answer_parts.append(answer_part)
+    
+    # Create the response
+    response = {
+        "question": question,
+        "answer_passages": answer_parts,
+        "total_passages": len(answer_parts)
+    }
+    
+    return response
+
+def save_qa_to_file(qa_response: Dict, output_file: str = None):
+    """
+    Save the Q&A response to a JSON file.
+    
+    Args:
+        qa_response (Dict): The Q&A response to save
+        output_file (str): Optional output file path. If None, generates a filename
+    """
+    if output_file is None:
+        # Create answers directory if it doesn't exist
+        answers_dir = os.path.join(os.getcwd(), "answers")
+        os.makedirs(answers_dir, exist_ok=True)
+        
+        # Generate filename from question
+        filename = f"answer_{qa_response['question'][:30].lower().replace(' ', '_')}.json"
+        output_file = os.path.join(answers_dir, filename)
+    
+    # Save to file
+    with open(output_file, 'w', encoding='utf-8') as f:
+        json.dump(qa_response, f, ensure_ascii=False, indent=2)
+    
+    return output_file
+
+if __name__ == "__main__":
+    # Example usage
+    question = "What is the nature of consciousness?"
+    
+    # Get answer
+    response = get_answer_from_osho(question)
+    
+    # Save to file
+    output_file = save_qa_to_file(response)
+    
+    # Print the response
+    print(f"\nQuestion: {response['question']}\n")
+    for passage in response['answer_passages']:
+        print(f"\nPassage {passage['passage_number']}:")
+        print(f"Book: {passage['book']}")
+        print(f"Text: {passage['text'][:200]}...")
+        print("-" * 80)
+    
+    print(f"\nResponse saved to: {output_file}")

query_vector_db.py

@@ -1,41 +1,44 @@
+# Suppress warnings - must be before any imports
 import os
-import chromadb
-from chromadb.utils import embedding_functions
+os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
+os.environ['TOKENIZERS_PARALLELISM'] = 'false'
+os.environ['TF_ENABLE_ONEDNN_OPTS'] = '0'
 
-def query_vector_db(query: str, n_results: int = 5):
-    """Query the vector database and return relevant passages."""
-    # Initialize ChromaDB client
-    db_dir = os.path.join(os.getcwd(), "vector_db")
-    client = chromadb.PersistentClient(path=db_dir)
+import warnings
+import logging
+# Suppress all warnings
+warnings.filterwarnings('ignore')
+# Specific suppressions
+warnings.filterwarnings('ignore', category=UserWarning)
+warnings.filterwarnings('ignore', category=DeprecationWarning)
+warnings.filterwarnings('ignore', category=FutureWarning)
+warnings.filterwarnings('ignore', message='.*benefit from vacuuming.*')
+warnings.filterwarnings('ignore', message='.*sparse_softmax_cross_entropy.*')
+
+# Suppress all logging
+logging.getLogger().setLevel(logging.ERROR)
+# Suppress TensorFlow logging
+logging.getLogger('tensorflow').setLevel(logging.ERROR)
+os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
+
+from osho_qa_service import get_answer_from_osho
+
+if __name__ == "__main__":
+    # Example query
+    query = "What is the relationship between breath and consciousness?"
     
-    # Initialize embedding function (same as used in creation)
-    embedding_function = embedding_functions.SentenceTransformerEmbeddingFunction(
-        model_name="all-MiniLM-L6-v2"
-    )
+    # Get answer using the service
+    response = get_answer_from_osho(query)
     
-    # Get the collection
-    collection = client.get_collection(
-        name="osho_books",
-        embedding_function=embedding_function
-    )
+    # Display formatted answer
+    print(f"\nQuery: {response['question']}\n")
     
-    # Query the collection
-    results = collection.query(
-        query_texts=[query],
-        n_results=n_results
-    )
+    # Get the first result as main answer
+    main_passage = response['answer_passages'][0]
+    print(f"I have discussed this idea in book '{main_passage['book']}':")
+    print(f"{main_passage['text']}\n")
     
-    # Print results
-    print(f"\nQuery: {query}\n")
-    for i, (doc, metadata) in enumerate(zip(results['documents'][0], results['metadatas'][0])):
-        print(f"\nResult {i+1}:")
-        print(f"Book: {metadata['book']}")
-        print(f"Passage: {doc[:200]}...")  # Show first 200 characters
-        print("-" * 80)
-
-if __name__ == "__main__":
-    while True:
-        query = input("\nEnter your query (or 'quit' to exit): ")
-        if query.lower() == 'quit':
-            break
-        query_vector_db(query)
+    # List other books as references
+    other_books = [p['book'] for p in response['answer_passages'][1:]]
+    if other_books:
+        print("You can refer to other books:", ", ".join(other_books))

requirements.txt

@@ -4,3 +4,4 @@ chromadb==0.4.20
 sentence-transformers==2.2.2
 tqdm==4.66.1
 huggingface-hub==0.19.4
+streamlit==1.29.0

streamlit_app.py

@@ -0,0 +1,108 @@
+import streamlit as st
+import os
+os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
+os.environ['TOKENIZERS_PARALLELISM'] = 'false'
+os.environ['TF_ENABLE_ONEDNN_OPTS'] = '0'
+
+import warnings
+import logging
+# Suppress all warnings
+warnings.filterwarnings('ignore')
+logging.getLogger().setLevel(logging.ERROR)
+
+from osho_qa_service import get_answer_from_osho
+
+# Set page config
+st.set_page_config(
+    page_title="Ask Osho",
+    page_icon="🧘‍♂️",
+    layout="centered",
+    initial_sidebar_state="collapsed"
+)
+
+# Title and description
+st.title("🧘‍♂️ Ask Osho")
+st.markdown("""
+This application allows you to ask questions and receive answers from Osho's wisdom.
+Choose from example questions or ask your own question.
+""")
+
+# Example questions
+example_questions = [
+    "What is the relationship between breath and consciousness?",
+    "How can meditation help in daily life?",
+    "What is the difference between mind and consciousness?",
+    "What is the nature of love?",
+    "How can one find inner peace?"
+]
+
+# Initialize session state for question if not exists
+if 'question' not in st.session_state:
+    st.session_state.question = ""
+
+# Create columns for example questions
+st.subheader("Example Questions")
+cols = st.columns(3)
+
+# Function to update question
+def set_question(q):
+    st.session_state.question = q
+
+# Create buttons for example questions
+for i, question in enumerate(example_questions):
+    col_idx = i % 3
+    if cols[col_idx].button(f"Q{i+1}: {question[:30]}...", key=f"q{i}", 
+                           help=question):  # Show full question on hover
+        set_question(question)
+
+# Or ask your own question
+st.subheader("Or Ask Your Own Question")
+question = st.text_input("Type your question here:", 
+                        value=st.session_state.question,
+                        key="question_input",
+                        on_change=lambda: set_question(st.session_state.question_input))
+
+# Answer button
+if st.button("Please Answer Osho", type="primary", key="answer_button"):
+    if question:
+        with st.spinner("Seeking wisdom in Osho's teachings..."):
+            try:
+                # Get answer using the service
+                response = get_answer_from_osho(question)
+                
+                # Display the answer in a nice box
+                st.markdown("---")
+                st.subheader("Osho's Answer")
+                
+                # Main answer
+                main_passage = response['answer_passages'][0]
+                st.info(f"**From the book**: _{main_passage['book']}_")
+                st.markdown(main_passage['text'])
+                
+                # Other references
+                other_books = [p['book'] for p in response['answer_passages'][1:]]
+                if other_books:
+                    st.markdown("---")
+                    st.success("**You can also explore these books:**")
+                    for book in other_books:
+                        st.markdown(f"- {book}")
+            except Exception as e:
+                st.error("Sorry, I encountered an error while processing your question. Please try again.")
+                st.error(f"Error details: {str(e)}")
+    else:
+        st.warning("Please enter a question or select an example question.")
+
+# Add some styling
+st.markdown("""
+<style>
+    .stButton button {
+        width: 100%;
+    }
+    .stButton>button:first-child {
+        margin-top: 20px;
+    }
+    .stMarkdown {
+        text-align: justify;
+    }
+</style>
+""", unsafe_allow_html=True)