update

README.md

@@ -57,6 +57,159 @@ The summary tab provides a summary of two of the prompts: the Trump and Harris p
 
 # Documentation
 
+## Datasets
+
+Built-in Dataset Structure
+
+The application includes several pre-built datasets for analysis:
+
+Format: Simple text files with structured format:
+\prompt= [prompt text]
+\response1= [first model response]
+\model1= [first model name]
+\response2= [second model response]
+\model2= [second model name]
+
+Included Datasets:
+
+Political Figures Responses: Comparisons of how different LLMs discuss political figures
+
+- person-harris.txt: Responses about Kamala Harris
+- person-trump.txt: Responses about Donald Trump
+
+Political Topics Responses: Comparisons on general political topics
+
+- topic-foreign_policy.txt: Responses about foreign policy views
+- topic-the_economy.txt: Responses about economic views
+
+Dataset Collection Process:
+
+- Prompts were designed to elicit substantive responses on political topics
+- Identical prompts were submitted to different commercial LLMs
+- Responses were collected verbatim without modification
+- Model identifiers were preserved for attribution
+- Responses were formatted into the standardized text format
+
+Dataset Size and Characteristics:
+
+- Each dataset contains one prompt and two model responses
+- Response length ranges from approximately 300-600 words
+- Models represented include ExaOne3.5, Granite3.2, and others
+- Topics were selected to span typical political discussion areas
+
+## Frameworks
+
+- Gradio is the main framework used to build the app.  It provides a simple interface for creating web applications with Python.
+- Matplotlib is used for some basic plotting in the visuals tab.
+- NLTK is used mainly for the VADER sentiment analysis classifier.
+ - This is for both the basic classifier and bias detection.
+- Hugging Face Transformers is used for the RoBERTa transformer model.
+- Scikit-learn is used for the Bag of Words and N-grams analysis.
+- Pandas is used for data manipulation and analysis.
+- NumPy is used for numerical computations.
+- JSON and os are used for file handling in relation to the datasets.
+- re, Regular Expressions, is used for text processing and cleaning.
+
+## App Flow
+
+We start with the dataset input.  This can be a user entered dataset or a built-in dataset.  We then go to the analysis tab which has four options.  After that is a RoBERTa classifier, which is a transformer model compared to a non-transformer classifier used in the analysis tab.  We have a summary after that, followed by some basic visual plots.
+
+## Bag of Words
+
+Basic preprocessing is done to the text data, including:
+- Lowercasing
+- Removing punctuation
+- Removing stop words
+- Tokenization
+- Lemmatization
+- Removing special characters
+
+Here is an example of the results from the Harris text file:
+
+Top Words Used by ExaOne3.5
+
+harris (8), policy (8), justice (5), attorney (4), issue (4), measure (4), political (4), aimed (3), approach (3), general (3)
+
+Top Words Used by Granite3.2
+
+harris (7), support (6), view (6), issue (5), right (5), policy (4), party (3), political (3), president (3), progressive (3)
+
+Similarity Metrics
+
+- Cosine Similarity: 0.67 (higher means more similar word frequency patterns)
+- Jaccard Similarity: 0.22 (higher means more word overlap)
+- Semantic Similarity: 0.53 (higher means more similar meaning)
+- Common Words: 71 words appear in both responses
+
+The main concepts here of comparison are the top words used by each model, the similarity metrics, and the common words.  The top words are the most frequently used words in each response.  The similarity metrics are calculated using cosine similarity, Jaccard similarity, and semantic similarity.  The common words are the words that appear in both responses.
+
+## N-grams
+
+
+
+## The Classifiers
+
+There is a RoBERTa transformer based classifier and one that uses NLTK VADER sentiment analysis. The RoBERTa classifier is a transformer model that is trained on a large corpus of text data and is designed to understand the context and meaning of words in a sentence.  The NLTK VADER sentiment analysis classifier is a rule-based model that uses a lexicon of words and their associated sentiment scores to determine the sentiment of a sentence.  Both classifiers are used to analyze the sentiment of the responses from the LLMs.  The VADER one is simpler and faster, while the RoBERTa one is more complex and takes longer to run.  The RoBERTa classifier is also more accurate than the VADER classifier, but it requires more computational resources to run.
+
+### RoBERTa
+
+Architecture: RoBERTa (Robustly Optimized BERT Pretraining Approach) is a transformer-based language model that improves upon BERT through modifications to the pretraining process.
+
+Training Procedure:
+
+- Trained on a massive dataset of 160GB of text
+- Uses dynamic masking pattern for masked language modeling
+- Trained with larger batches and learning rates than BERT
+- Eliminates BERT's next-sentence prediction objective
+
+Implementation Details:
+
+- Uses the transformers library from Hugging Face
+- Specifically uses RobertaForSequenceClassification for sentiment analysis
+- Model loaded: roberta-large-mnli for natural language inference tasks
+
+Compute Requirements:
+
+- Inference requires moderate GPU resources or CPU with sufficient memory
+- Model size: ~355M parameters
+- Typical memory usage: ~1.3GB when loaded
+
+Training Data:
+
+- BookCorpus (800M words)
+- English Wikipedia (2,500M words)
+- CC-News (63M articles, 76GB)
+- OpenWebText (38GB)
+- Stories (31GB)
+
+Known Limitations:
+
+- May struggle with highly domain-specific language
+- Limited context window (512 tokens)
+- Performance can degrade on very short texts
+- Has potential biases from training data
+
+### NLTK VADER
+
+Components Used:
+
+- NLTK's SentimentIntensityAnalyzer (VADER lexicon-based model)
+- WordNet Lemmatizer
+- Tokenizers (word, sentence)
+- Stopword filters
+
+Training Data:
+
+- VADER sentiment analyzer was trained on social media content, movie reviews, and product reviews
+- NLTK word tokenizers trained on standard English corpora
+
+Limitations:
+
+- Rule-based classifiers have lower accuracy than deep learning models
+- Limited ability to understand context and nuance
+- VADER sentiment analyzer works best on short social media-like texts
+
+## Bias Detection
 
 
 # Contributions

app.py

@@ -67,6 +67,9 @@ def create_app():
         analysis_results_state = gr.State({})
         roberta_results_state = gr.State({})
         
+        # NEW: Add a state for storing user dataset analysis results
+        user_analysis_log = gr.State({})
+        
         # Dataset Input Tab
         with gr.Tab("Dataset Input"):
             # Filter out files that start with 'summary' for the Dataset Input tab
@@ -131,11 +134,12 @@ def create_app():
             status_message = gr.Markdown(visible=False)
             
             # Define a helper function to extract parameter values and run the analysis
-            def run_analysis(dataset, selected_analysis, ngram_n, topic_count):
+            def run_analysis(dataset, selected_analysis, ngram_n, topic_count, existing_log):
                 try:
                     if not dataset or "entries" not in dataset or not dataset["entries"]:
                         return (
                             {},  # analysis_results_state
+                            existing_log,  # no changes to user_analysis_log
                             False,  # analysis_output visibility
                             False,  # visualization_area_visible
                             gr.update(visible=False),  # analysis_title
@@ -164,10 +168,44 @@ def create_app():
                     # Process the analysis request - passing selected_analysis as a string
                     analysis_results, _ = process_analysis_request(dataset, selected_analysis, parameters)
                     
+                    # NEW: Store the results in the user_analysis_log
+                    updated_log = existing_log.copy() if existing_log else {}
+                    
+                    # Get the prompt text for identifying this analysis
+                    prompt_text = None
+                    if analysis_results and "analyses" in analysis_results:
+                        prompt_text = list(analysis_results["analyses"].keys())[0] if analysis_results["analyses"] else None
+                    
+                    if prompt_text:
+                        # Initialize this prompt in the log if it doesn't exist
+                        if prompt_text not in updated_log:
+                            updated_log[prompt_text] = {}
+                        
+                        # Store the results for this analysis type
+                        if selected_analysis in ["Bag of Words", "N-gram Analysis", "Bias Detection", "Classifier"]:
+                            # Only store if the analysis was actually performed and has results
+                            analyses = analysis_results["analyses"][prompt_text]
+                            
+                            # Map the selected analysis to its key in the analyses dict
+                            analysis_key_map = {
+                                "Bag of Words": "bag_of_words",
+                                "N-gram Analysis": "ngram_analysis",
+                                "Bias Detection": "bias_detection",
+                                "Classifier": "classifier"
+                            }
+                            
+                            if analysis_key_map[selected_analysis] in analyses:
+                                # Store the specific analysis result
+                                updated_log[prompt_text][selected_analysis] = {
+                                    "timestamp": gr.utils.datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
+                                    "result": analyses[analysis_key_map[selected_analysis]]
+                                }
+                    
                     # If there's an error or no results
                     if not analysis_results or "analyses" not in analysis_results or not analysis_results["analyses"]:
                         return (
                             analysis_results,
+                            updated_log,  # Return the updated log
                             False,
                             False,
                             gr.update(visible=False),
@@ -212,6 +250,7 @@ def create_app():
                     if "message" in analyses:
                         return (
                             analysis_results,
+                            updated_log,  # Return the updated log
                             False,
                             False,
                             gr.update(visible=False),
@@ -535,6 +574,7 @@ def create_app():
                     if not visualization_area_visible:
                         return (
                             analysis_results,
+                            updated_log,  # Return the updated log
                             False,
                             False,
                             gr.update(visible=False),
@@ -545,6 +585,7 @@ def create_app():
                             gr.update(visible=False),
                             gr.update(visible=False),
                             gr.update(visible=False),
+                            gr.update(visible=False),
                             True,  # status_message_visible
                             gr.update(visible=True, value="❌ **No visualization data found.** Make sure to select a valid analysis option.")
                         )
@@ -552,6 +593,7 @@ def create_app():
                     # Return all updated component values
                     return (
                         analysis_results,  # analysis_results_state
+                        updated_log,  # Return the updated log
                         False,  # analysis_output visibility
                         True,   # visualization_area_visible
                         gr.update(visible=True),  # analysis_title
@@ -574,6 +616,7 @@ def create_app():
                     
                     return (
                         {"error": error_msg},  # analysis_results_state
+                        existing_log,  # Return unchanged log
                         True,  # analysis_output visibility (show raw JSON for debugging)
                         False,  # visualization_area_visible
                         gr.update(visible=False),
@@ -601,12 +644,13 @@ def create_app():
                 roberta_viz_content = gr.HTML("", visible=False)
             
             # Function to run RoBERTa sentiment analysis (FIXED)
-            def run_roberta_analysis(dataset):
+            def run_roberta_analysis(dataset, existing_log):
                 try:
                     print("Starting run_roberta_analysis function")
                     if not dataset or "entries" not in dataset or not dataset["entries"]:
                         return (
                             {},  # roberta_results_state
+                            existing_log,  # no change to user_analysis_log
                             gr.update(visible=True, value="❌ **Error:** No dataset loaded. Please create or load a dataset first."),  # roberta_status
                             gr.update(visible=False),  # roberta_output
                             gr.update(visible=False),  # roberta_viz_title
@@ -620,10 +664,32 @@ def create_app():
 
                     print(f"RoBERTa results obtained. Size: {len(str(roberta_results))} characters")
                     
+                    # NEW: Update the user analysis log with RoBERTa results
+                    updated_log = existing_log.copy() if existing_log else {}
+                    
+                    # Get the prompt text
+                    prompt_text = None
+                    if "analyses" in roberta_results:
+                        prompt_text = list(roberta_results["analyses"].keys())[0] if roberta_results["analyses"] else None
+                    
+                    if prompt_text:
+                        # Initialize this prompt in the log if it doesn't exist
+                        if prompt_text not in updated_log:
+                            updated_log[prompt_text] = {}
+                        
+                        # Store the RoBERTa results
+                        if "analyses" in roberta_results and prompt_text in roberta_results["analyses"]:
+                            if "roberta_sentiment" in roberta_results["analyses"][prompt_text]:
+                                updated_log[prompt_text]["RoBERTa Sentiment"] = {
+                                    "timestamp": gr.utils.datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
+                                    "result": roberta_results["analyses"][prompt_text]["roberta_sentiment"]
+                                }
+                    
                     # Check if we have results
                     if "error" in roberta_results:
                         return (
                             roberta_results,  # Store in state anyway for debugging
+                            updated_log,  # Return updated log
                             gr.update(visible=True, value=f"❌ **Error:** {roberta_results['error']}"),  # roberta_status
                             gr.update(visible=False),  # Hide raw output
                             gr.update(visible=False),  # roberta_viz_title
@@ -674,6 +740,7 @@ def create_app():
                     # Return updated values
                     return (
                         roberta_results,  # roberta_results_state
+                        updated_log,  # Return updated log
                         gr.update(visible=False),  # roberta_status (hide status message)
                         gr.update(visible=False),  # roberta_output (hide raw output)
                         gr.update(visible=True),   # roberta_viz_title (show title)
@@ -687,6 +754,7 @@ def create_app():
                     
                     return (
                         {"error": error_msg},  # roberta_results_state
+                        existing_log,  # Return unchanged log
                         gr.update(visible=True, value=f"❌ **Error during RoBERTa analysis:**\n\n```\n{str(e)}\n```"),  # roberta_status
                         gr.update(visible=False),  # Hide raw output
                         gr.update(visible=False),  # roberta_viz_title
@@ -696,9 +764,10 @@ def create_app():
             # Connect the run button to the analysis function (FIXED)
             run_roberta_btn.click(
                 fn=run_roberta_analysis,
-                inputs=[dataset_state],
+                inputs=[dataset_state, user_analysis_log],
                 outputs=[
                     roberta_results_state,
+                    user_analysis_log,
                     roberta_status,
                     roberta_output,
                     roberta_viz_title,
@@ -715,11 +784,12 @@ def create_app():
                     # Get summary files from dataset directory
                     summary_files = [f for f in os.listdir("dataset") if f.startswith("summary-") and f.endswith(".txt")]
                     
+                    # Add "YOUR DATASET RESULTS" to dropdown choices if we have user analysis
                     summary_dropdown = gr.Dropdown(
-                        choices=summary_files,
+                        choices=["YOUR DATASET RESULTS"] + summary_files,
                         label="Select Summary",
                         info="Choose a summary to display",
-                        value=summary_files[0] if summary_files else None
+                        value="YOUR DATASET RESULTS"
                     )
                     
                     load_summary_btn = gr.Button("Load Summary", variant="primary")
@@ -734,11 +804,173 @@ def create_app():
                     
                     summary_status = gr.Markdown("*No summary loaded*")
             
-            # Function to load summary content from file
-            def load_summary_file(file_name):
+            # Function to load summary content from file or user analysis
+            def load_summary_content(file_name, user_log):
                 if not file_name:
                     return "", "*No summary selected*"
+                
+                # Handle the special "YOUR DATASET RESULTS" option
+                if file_name == "YOUR DATASET RESULTS":
+                    if not user_log or not any(user_log.values()):
+                        return "", "❌ **No analysis results available.** Run some analyses in the Analysis tab first."
+                    
+                    # Format the user analysis log as text
+                    content = "# YOUR DATASET ANALYSIS RESULTS\n\n"
+                    
+                    for prompt, analyses in user_log.items():
+                        content += f"## Analysis of Prompt: \"{prompt[:100]}{'...' if len(prompt) > 100 else ''}\"\n\n"
+                        
+                        if not analyses:
+                            content += "_No analyses run for this prompt._\n\n"
+                            continue
+                        
+                        # Order the analyses in a specific sequence
+                        analysis_order = ["Bag of Words", "N-gram Analysis", "Classifier", "Bias Detection", "RoBERTa Sentiment"]
+                        
+                        for analysis_type in analysis_order:
+                            if analysis_type in analyses:
+                                analysis_data = analyses[analysis_type]
+                                timestamp = analysis_data.get("timestamp", "")
+                                result = analysis_data.get("result", {})
+                                
+                                content += f"### {analysis_type} ({timestamp})\n\n"
+                                
+                                # Format based on analysis type
+                                if analysis_type == "Bag of Words":
+                                    models = result.get("models", [])
+                                    if len(models) >= 2:
+                                        content += f"Comparing responses from {models[0]} and {models[1]}\n\n"
+                                        
+                                        # Add important words for each model
+                                        important_words = result.get("important_words", {})
+                                        for model_name in models:
+                                            if model_name in important_words:
+                                                content += f"Top Words Used by {model_name}\n"
+                                                word_list = [f"{item['word']} ({item['count']})" for item in important_words[model_name][:10]]
+                                                content += ", ".join(word_list) + "\n\n"
+                                        
+                                        # Add similarity metrics
+                                        comparisons = result.get("comparisons", {})
+                                        comparison_key = f"{models[0]} vs {models[1]}"
+                                        if comparison_key in comparisons:
+                                            metrics = comparisons[comparison_key]
+                                            content += "Similarity Metrics\n"
+                                            content += f"Cosine Similarity: {metrics.get('cosine_similarity', 0):.2f} (higher means more similar word frequency patterns)\n"
+                                            content += f"Jaccard Similarity: {metrics.get('jaccard_similarity', 0):.2f} (higher means more word overlap)\n"
+                                            content += f"Semantic Similarity: {metrics.get('semantic_similarity', 0):.2f} (higher means more similar meaning)\n"
+                                            content += f"Common Words: {metrics.get('common_word_count', 0)} words appear in both responses\n\n"
+                                
+                                elif analysis_type == "N-gram Analysis":
+                                    models = result.get("models", [])
+                                    ngram_size = result.get("ngram_size", 2)
+                                    size_name = "Unigrams" if ngram_size == 1 else f"{ngram_size}-grams"
+                                    
+                                    if len(models) >= 2:
+                                        content += f"{size_name} Analysis: Comparing responses from {models[0]} and {models[1]}\n\n"
+                                        
+                                        # Add important n-grams for each model
+                                        important_ngrams = result.get("important_ngrams", {})
+                                        for model_name in models:
+                                            if model_name in important_ngrams:
+                                                content += f"Top {size_name} Used by {model_name}\n"
+                                                ngram_list = [f"{item['ngram']} ({item['count']})" for item in important_ngrams[model_name][:10]]
+                                                content += ", ".join(ngram_list) + "\n\n"
+                                        
+                                        # Add similarity metrics
+                                        if "comparisons" in result:
+                                            comparison_key = f"{models[0]} vs {models[1]}"
+                                            if comparison_key in result["comparisons"]:
+                                                metrics = result["comparisons"][comparison_key]
+                                                content += "Similarity Metrics\n"
+                                                content += f"Common {size_name}: {metrics.get('common_ngram_count', 0)} {size_name.lower()} appear in both responses\n\n"
+                                
+                                elif analysis_type == "Classifier":
+                                    models = result.get("models", [])
+                                    if len(models) >= 2:
+                                        content += f"Classifier Analysis for {models[0]} and {models[1]}\n\n"
+                                        
+                                        # Add classification results
+                                        classifications = result.get("classifications", {})
+                                        if classifications:
+                                            content += "Classification Results\n"
+                                            for model_name in models:
+                                                if model_name in classifications:
+                                                    model_results = classifications[model_name]
+                                                    content += f"{model_name}:\n"
+                                                    content += f"- Formality: {model_results.get('formality', 'N/A')}\n"
+                                                    content += f"- Sentiment: {model_results.get('sentiment', 'N/A')}\n"
+                                                    content += f"- Complexity: {model_results.get('complexity', 'N/A')}\n\n"
+                                            
+                                            # Add differences
+                                            differences = result.get("differences", {})
+                                            if differences:
+                                                content += "Classification Comparison\n"
+                                                for category, diff in differences.items():
+                                                    content += f"- {category}: {diff}\n"
+                                                content += "\n"
+                                
+                                elif analysis_type == "Bias Detection":
+                                    models = result.get("models", [])
+                                    if len(models) >= 2:
+                                        content += f"Bias Analysis: Comparing responses from {models[0]} and {models[1]}\n\n"
+                                        
+                                        # Add comparative results
+                                        if "comparative" in result:
+                                            comparative = result["comparative"]
+                                            content += "Bias Detection Summary\n"
+                                            
+                                            if "partisan" in comparative:
+                                                part = comparative["partisan"]
+                                                is_significant = part.get("significant", False)
+                                                content += f"Partisan Leaning: {models[0]} appears {part.get(models[0], 'N/A')}, "
+                                                content += f"while {models[1]} appears {part.get(models[1], 'N/A')}. "
+                                                content += f"({'Significant' if is_significant else 'Minor'} difference)\n\n"
+                                            
+                                            if "overall" in comparative:
+                                                overall = comparative["overall"]
+                                                significant = overall.get("significant_bias_difference", False)
+                                                content += f"Overall Assessment: "
+                                                content += f"Analysis shows a {overall.get('difference', 0):.2f}/1.0 difference in bias patterns. "
+                                                content += f"({'Significant' if significant else 'Minor'} overall bias difference)\n\n"
+                                            
+                                            # Add partisan terms
+                                            content += "Partisan Term Analysis\n"
+                                            for model_name in models:
+                                                if model_name in result and "partisan" in result[model_name]:
+                                                    partisan = result[model_name]["partisan"]
+                                                    content += f"{model_name}:\n"
+                                                    
+                                                    lib_terms = partisan.get("liberal_terms", [])
+                                                    con_terms = partisan.get("conservative_terms", [])
+                                                    
+                                                    content += f"- Liberal terms: {', '.join(lib_terms) if lib_terms else 'None detected'}\n"
+                                                    content += f"- Conservative terms: {', '.join(con_terms) if con_terms else 'None detected'}\n\n"
+                                
+                                elif analysis_type == "RoBERTa Sentiment":
+                                    models = result.get("models", [])
+                                    if len(models) >= 2:
+                                        content += "Sentiment Analysis Results\n"
+                                        
+                                        # Add comparison info
+                                        if "comparison" in result:
+                                            comparison = result["comparison"]
+                                            if "difference_direction" in comparison:
+                                                content += f"{comparison['difference_direction']}\n\n"
+                                        
+                                        # Add individual model results
+                                        sentiment_analysis = result.get("sentiment_analysis", {})
+                                        for model_name in models:
+                                            if model_name in sentiment_analysis:
+                                                model_result = sentiment_analysis[model_name]
+                                                score = model_result.get("sentiment_score", 0)
+                                                label = model_result.get("label", "neutral")
+                                                
+                                                content += f"{model_name}\n"
+                                                content += f"Sentiment: {label} (Score: {score:.2f})\n\n"
+                    
+                    return content, f"✅ **Loaded user analysis results**"
                     
+                # Regular file loading for built-in summaries
                 file_path = os.path.join("dataset", file_name)
                 if os.path.exists(file_path):
                     try:
@@ -749,18 +981,24 @@ def create_app():
                         return "", f"❌ **Error loading summary**: {str(e)}"
                 else:
                     return "", f"❌ **File not found**: {file_path}"
+                
+            def update_summary_dropdown(user_log):
+                """Update summary dropdown options based on user log state"""
+                choices = ["YOUR DATASET RESULTS"]
+                choices.extend([f for f in os.listdir("dataset") if f.startswith("summary-") and f.endswith(".txt")])
+                return gr.Dropdown.update(choices=choices, value="YOUR DATASET RESULTS")
             
             # Connect the load button to the function
             load_summary_btn.click(
-                fn=load_summary_file,
-                inputs=[summary_dropdown],
+                fn=load_summary_content,
+                inputs=[summary_dropdown, user_analysis_log],
                 outputs=[summary_content, summary_status]
             )
             
             # Also load summary when dropdown changes
             summary_dropdown.change(
-                fn=load_summary_file,
-                inputs=[summary_dropdown],
+                fn=load_summary_content,
+                inputs=[summary_dropdown, user_analysis_log],
                 outputs=[summary_content, summary_status]
             )
                 # Add a Visuals tab for plotting graphs
@@ -946,9 +1184,10 @@ def create_app():
         # Run analysis with proper parameters
         run_analysis_btn.click(
             fn=run_analysis,
-            inputs=[dataset_state, analysis_options, ngram_n, topic_count],
+            inputs=[dataset_state, analysis_options, ngram_n, topic_count, user_analysis_log],
             outputs=[
                 analysis_results_state,
+                user_analysis_log,
                 analysis_output,
                 visualization_area_visible,
                 analysis_title,
@@ -965,6 +1204,15 @@ def create_app():
             ]
         )
     
+    app.load(
+        fn=lambda log: (
+            update_summary_dropdown(log),
+            load_summary_content("YOUR DATASET RESULTS", log)
+        ),
+        inputs=[user_analysis_log],
+        outputs=[summary_dropdown, summary_content, summary_status]
+    )
+
     return app
 
 if __name__ == "__main__":