week 7

Reference files/week 7/W7_Lab_KNN_clustering.ipynb

@@ -0,0 +1,481 @@
+{
+  "cells": [
+    {
+      "cell_type": "markdown",
+      "id": "b1c6a137",
+      "metadata": {
+        "id": "b1c6a137"
+      },
+      "source": [
+        "# Clustering Lab: State Crime Pattern Analysis\n",
+        "\n",
+        "## Lab Overview\n",
+        "\n",
+        "Welcome to your hands-on clustering lab! You'll be working as a policy analyst for the Department of Justice, analyzing crime patterns across US states. Your mission: discover hidden safety profiles that could inform federal resource allocation and crime prevention strategies.\n",
+        "\n",
+        "**Your Deliverable**: A policy brief with visualizations and recommendations based on your clustering analysis.\n",
+        "\n",
+        "---\n",
+        "\n",
+        "## Exercise 1: Data Detective Work\n",
+        "**Time: 15 minutes | Product: Data Summary Report**\n",
+        "\n",
+        "### Your Task\n",
+        "Before any analysis, you need to understand what you're working with. Create a brief data summary that a non-technical policy maker could understand.\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "```python\n",
+        "import numpy as np\n",
+        "import pandas as pd\n",
+        "import matplotlib.pyplot as plt\n",
+        "from statsmodels.datasets import get_rdataset\n",
+        "from sklearn.preprocessing import StandardScaler\n",
+        "from sklearn.cluster import KMeans, AgglomerativeClustering\n",
+        "\n",
+        "# Load the data\n",
+        "USArrests = get_rdataset('USArrests').data\n",
+        "print(\"Dataset shape:\", USArrests.shape)\n",
+        "print(\"\\nVariables:\", USArrests.columns.tolist())\n",
+        "print(\"\\nFirst 5 states:\")\n",
+        "print(USArrests.head())\n",
+        "```"
+      ],
+      "metadata": {
+        "colab": {
+          "base_uri": "https://localhost:8080/",
+          "height": 106
+        },
+        "id": "mqRVE1hlXK9x",
+        "outputId": "5a1bbd64-15cd-4e1c-9344-64a901d8a396"
+      },
+      "id": "mqRVE1hlXK9x",
+      "execution_count": null,
+      "outputs": [
+        {
+          "output_type": "error",
+          "ename": "SyntaxError",
+          "evalue": "invalid syntax (<ipython-input-1-2035427107>, line 1)",
+          "traceback": [
+            "\u001b[0;36m  File \u001b[0;32m\"<ipython-input-1-2035427107>\"\u001b[0;36m, line \u001b[0;32m1\u001b[0m\n\u001b[0;31m    ```python\u001b[0m\n\u001b[0m    ^\u001b[0m\n\u001b[0;31mSyntaxError\u001b[0m\u001b[0;31m:\u001b[0m invalid syntax\n"
+          ]
+        }
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "## Your Investigation\n",
+        "Complete this data summary table:\n",
+        "\n",
+        "| Variable | What it measures | Average Value | Highest State | Lowest State |\n",
+        "|----------|------------------|---------------|---------------|--------------|\n",
+        "| Murder | Rate per 100,000 people | ??? | ??? | ??? |\n",
+        "| Assault | Rate per 100,000 people | ??? | ??? | ??? |\n",
+        "| UrbanPop | Percentage living in cities | ??? | ??? | ??? |\n",
+        "| Rape | Rate per 100,000 people | ??? | ??? | ??? |\n",
+        "\n",
+        "**Deliverable**: Write 2-3 sentences describing the biggest surprises in this data. Which states are not what you expected?\n",
+        "\n",
+        "---\n",
+        "\n",
+        "## Exercise 2: The Scaling Challenge\n",
+        "**Time: 10 minutes | Product: Before/After Comparison**\n",
+        "\n",
+        "### Your Task\n",
+        "Demonstrate why scaling is critical for clustering crime data.\n",
+        "\n"
+      ],
+      "metadata": {
+        "id": "7qkDKTe4XLtG"
+      },
+      "id": "7qkDKTe4XLtG"
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "```python\n",
+        "# Check the scale differences\n",
+        "print(\"Original data ranges:\")\n",
+        "print(USArrests.describe())\n",
+        "\n",
+        "print(\"\\nVariances (how spread out the data is):\")\n",
+        "print(USArrests.var())\n",
+        "\n",
+        "# Scale the data\n",
+        "scaler = StandardScaler()\n",
+        "USArrests_scaled = scaler.fit_transform(USArrests)\n",
+        "scaled_df = pd.DataFrame(USArrests_scaled,\n",
+        "                        columns=USArrests.columns,\n",
+        "                        index=USArrests.index)\n",
+        "\n",
+        "print(\"\\nAfter scaling - all variables now have similar ranges:\")\n",
+        "print(scaled_df.describe())\n",
+        "```"
+      ],
+      "metadata": {
+        "id": "zQ3VowYNXLeQ"
+      },
+      "id": "zQ3VowYNXLeQ",
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Your Analysis\n",
+        "1. **Before scaling**: Which variable would dominate the clustering? Why?\n",
+        "2. **After scaling**: Explain in simple terms what StandardScaler did to the data.\n",
+        "\n",
+        "**Deliverable**: One paragraph explaining why a policy analyst should care about data scaling.\n",
+        "\n",
+        "---\n",
+        "\n",
+        "## Exercise 3: Finding the Right Number of Groups\n",
+        "**Time: 20 minutes | Product: Recommendation with Visual Evidence**\n",
+        "\n",
+        "### Your Task\n",
+        "Use the elbow method to determine how many distinct crime profiles exist among US states.\n"
+      ],
+      "metadata": {
+        "id": "FnOT700SXLPh"
+      },
+      "id": "FnOT700SXLPh"
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "```python\n",
+        "# Test different numbers of clusters\n",
+        "inertias = []\n",
+        "K_values = range(1, 11)\n",
+        "\n",
+        "for k in K_values:\n",
+        "    kmeans = KMeans(n_clusters=k, random_state=42, n_init=20)\n",
+        "    kmeans.fit(USArrests_scaled)\n",
+        "    inertias.append(kmeans.inertia_)\n",
+        "\n",
+        "# Create the elbow plot\n",
+        "plt.figure(figsize=(10, 6))\n",
+        "plt.plot(K_values, inertias, 'bo-', linewidth=2, markersize=8)\n",
+        "plt.xlabel('Number of Clusters (K)')\n",
+        "plt.ylabel('Within-Cluster Sum of Squares')\n",
+        "plt.title('Finding the Optimal Number of State Crime Profiles')\n",
+        "plt.grid(True, alpha=0.3)\n",
+        "plt.show()\n",
+        "\n",
+        "# Print the inertia values\n",
+        "for k, inertia in zip(K_values, inertias):\n",
+        "    print(f\"K={k}: Inertia = {inertia:.1f}\")\n",
+        "```"
+      ],
+      "metadata": {
+        "id": "zOQrS9lmXpTF"
+      },
+      "id": "zOQrS9lmXpTF",
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "id": "2e388ef2",
+      "metadata": {
+        "id": "2e388ef2"
+      },
+      "source": [
+        "### Your Decision\n",
+        "Based on your elbow plot:\n",
+        "1. **What value of K do you recommend?** (Look for the \"elbow\" where the line starts to flatten)\n",
+        "2. **What does this mean in policy terms?** (How many distinct types of state crime profiles exist?)\n",
+        "\n",
+        "**Deliverable**: A one-paragraph recommendation with your chosen K value and reasoning.\n",
+        "\n",
+        "---\n",
+        "\n",
+        "## Exercise 4: K-Means State Profiling\n",
+        "**Time: 25 minutes | Product: State Crime Profile Report**\n",
+        "\n",
+        "### Your Task\n",
+        "Create distinct crime profiles and identify which states belong to each category.\n",
+        "\n",
+        "\n",
+        "\n",
+        "\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "```python\n",
+        "# Use your chosen K value from Exercise 3\n",
+        "optimal_k = 4  # Replace with your chosen value\n",
+        "\n",
+        "# Perform K-means clustering\n",
+        "kmeans = KMeans(n_clusters=optimal_k, random_state=42, n_init=20)\n",
+        "cluster_labels = kmeans.fit_predict(USArrests_scaled)\n",
+        "\n",
+        "# Add cluster labels to original data\n",
+        "USArrests_clustered = USArrests.copy()\n",
+        "USArrests_clustered['Cluster'] = cluster_labels\n",
+        "\n",
+        "# Analyze each cluster\n",
+        "print(\"State Crime Profiles Analysis\")\n",
+        "print(\"=\" * 50)\n",
+        "\n",
+        "for cluster_num in range(optimal_k):\n",
+        "    cluster_states = USArrests_clustered[USArrests_clustered['Cluster'] == cluster_num]\n",
+        "    print(f\"\\nCLUSTER {cluster_num}: {len(cluster_states)} states\")\n",
+        "    print(\"States:\", \", \".join(cluster_states.index.tolist()))\n",
+        "    print(\"Average characteristics:\")\n",
+        "    avg_profile = cluster_states[['Murder', 'Assault', 'UrbanPop', 'Rape']].mean()\n",
+        "    for var, value in avg_profile.items():\n",
+        "        print(f\"  {var}: {value:.1f}\")\n",
+        "```"
+      ],
+      "metadata": {
+        "id": "_5b0nE6KXv1P"
+      },
+      "id": "_5b0nE6KXv1P",
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Your Analysis\n",
+        "For each cluster, create a profile:\n",
+        "\n",
+        "**Cluster 0: \"[Your Creative Name]\"**\n",
+        "- **States**: [List them]\n",
+        "- **Characteristics**: [Describe the pattern]\n",
+        "- **Policy Insight**: [What should federal agencies know about these states?]\n",
+        "\n",
+        "**Deliverable**: A table summarizing each cluster with creative names and policy recommendations.\n",
+        "\n",
+        "---\n",
+        "\n",
+        "## Exercise 5: Hierarchical Clustering Exploration\n",
+        "**Time: 25 minutes | Product: Family Tree Interpretation**\n",
+        "\n",
+        "### Your Task\n",
+        "Create a dendrogram to understand how states naturally group together.\n"
+      ],
+      "metadata": {
+        "id": "J1WVGb_nX4ye"
+      },
+      "id": "J1WVGb_nX4ye"
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "```python\n",
+        "from scipy.cluster.hierarchy import dendrogram, linkage\n",
+        "\n",
+        "# Create hierarchical clustering\n",
+        "linkage_matrix = linkage(USArrests_scaled, method='complete')\n",
+        "\n",
+        "# Plot the dendrogram\n",
+        "plt.figure(figsize=(15, 8))\n",
+        "dendrogram(linkage_matrix,\n",
+        "           labels=USArrests.index.tolist(),\n",
+        "           leaf_rotation=90,\n",
+        "           leaf_font_size=10)\n",
+        "plt.title('State Crime Pattern Family Tree')\n",
+        "plt.xlabel('States')\n",
+        "plt.ylabel('Distance Between Groups')\n",
+        "plt.tight_layout()\n",
+        "plt.show()\n",
+        "```"
+      ],
+      "metadata": {
+        "id": "Y9a_cbZKX7QX"
+      },
+      "id": "Y9a_cbZKX7QX",
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Your Interpretation\n",
+        "1. **Closest Pairs**: Which two states are most similar in crime patterns?\n",
+        "2. **Biggest Divide**: Where is the largest split in the tree? What does this represent?\n",
+        "3. **Surprising Neighbors**: Which states cluster together that surprised you geographically?\n",
+        "\n",
+        "### Code to Compare Methods"
+      ],
+      "metadata": {
+        "id": "0PaImqZtX6f3"
+      },
+      "id": "0PaImqZtX6f3"
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "```python\n",
+        "# Compare your K-means results with hierarchical clustering\n",
+        "from scipy.cluster.hierarchy import fcluster\n",
+        "\n",
+        "# Cut the tree to get the same number of clusters as K-means\n",
+        "hierarchical_labels = fcluster(linkage_matrix, optimal_k, criterion='maxclust') - 1\n",
+        "\n",
+        "# Create comparison\n",
+        "comparison_df = pd.DataFrame({\n",
+        "    'State': USArrests.index,\n",
+        "    'K_Means_Cluster': cluster_labels,\n",
+        "    'Hierarchical_Cluster': hierarchical_labels\n",
+        "})\n",
+        "\n",
+        "print(\"Comparison of K-Means vs Hierarchical Clustering:\")\n",
+        "print(comparison_df.sort_values('State'))\n",
+        "\n",
+        "# Count agreements\n",
+        "agreements = sum(comparison_df['K_Means_Cluster'] == comparison_df['Hierarchical_Cluster'])\n",
+        "print(f\"\\nMethods agreed on {agreements} out of {len(comparison_df)} states ({agreements/len(comparison_df)*100:.1f}%)\")\n",
+        "```"
+      ],
+      "metadata": {
+        "id": "tJQ-C5GFYBRT"
+      },
+      "id": "tJQ-C5GFYBRT",
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "**Deliverable**: A paragraph explaining the key differences between what K-means and hierarchical clustering revealed.\n",
+        "\n",
+        "---\n",
+        "\n",
+        "## Exercise 6: Policy Brief Creation\n",
+        "**Time: 20 minutes | Product: Executive Summary**\n",
+        "\n",
+        "### Your Task\n",
+        "Synthesize your findings into a policy brief for Department of Justice leadership.\n",
+        "\n",
+        "### Code Framework for Final Visualization"
+      ],
+      "metadata": {
+        "id": "dx1fNhu4YD7-"
+      },
+      "id": "dx1fNhu4YD7-"
+    },
+    {
+      "cell_type": "code",
+      "source": [
+        "```python\n",
+        "# Create a comprehensive visualization\n",
+        "fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2, 2, figsize=(15, 12))\n",
+        "\n",
+        "# Plot 1: Murder vs Assault by cluster\n",
+        "colors = ['red', 'blue', 'green', 'orange', 'purple']\n",
+        "for i in range(optimal_k):\n",
+        "    cluster_data = USArrests_clustered[USArrests_clustered['Cluster'] == i]\n",
+        "    ax1.scatter(cluster_data['Murder'], cluster_data['Assault'],\n",
+        "               c=colors[i], label=f'Cluster {i}', s=60, alpha=0.7)\n",
+        "ax1.set_xlabel('Murder Rate')\n",
+        "ax1.set_ylabel('Assault Rate')\n",
+        "ax1.set_title('Murder vs Assault by Crime Profile')\n",
+        "ax1.legend()\n",
+        "ax1.grid(True, alpha=0.3)\n",
+        "\n",
+        "# Plot 2: Urban Population vs Rape by cluster\n",
+        "for i in range(optimal_k):\n",
+        "    cluster_data = USArrests_clustered[USArrests_clustered['Cluster'] == i]\n",
+        "    ax2.scatter(cluster_data['UrbanPop'], cluster_data['Rape'],\n",
+        "               c=colors[i], label=f'Cluster {i}', s=60, alpha=0.7)\n",
+        "ax2.set_xlabel('Urban Population %')\n",
+        "ax2.set_ylabel('Rape Rate')\n",
+        "ax2.set_title('Urban Population vs Rape Rate by Crime Profile')\n",
+        "ax2.legend()\n",
+        "ax2.grid(True, alpha=0.3)\n",
+        "\n",
+        "# Plot 3: Cluster size comparison\n",
+        "cluster_sizes = USArrests_clustered['Cluster'].value_counts().sort_index()\n",
+        "ax3.bar(range(len(cluster_sizes)), cluster_sizes.values, color=colors[:len(cluster_sizes)])\n",
+        "ax3.set_xlabel('Cluster Number')\n",
+        "ax3.set_ylabel('Number of States')\n",
+        "ax3.set_title('Number of States in Each Crime Profile')\n",
+        "ax3.set_xticks(range(len(cluster_sizes)))\n",
+        "\n",
+        "# Plot 4: Average crime rates by cluster\n",
+        "cluster_means = USArrests_clustered.groupby('Cluster')[['Murder', 'Assault', 'Rape']].mean()\n",
+        "cluster_means.plot(kind='bar', ax=ax4)\n",
+        "ax4.set_xlabel('Cluster Number')\n",
+        "ax4.set_ylabel('Average Rate')\n",
+        "ax4.set_title('Average Crime Rates by Profile')\n",
+        "ax4.legend()\n",
+        "ax4.tick_params(axis='x', rotation=0)\n",
+        "\n",
+        "plt.tight_layout()\n",
+        "plt.show()\n",
+        "```"
+      ],
+      "metadata": {
+        "id": "N8bkxURpYHJF"
+      },
+      "id": "N8bkxURpYHJF",
+      "execution_count": null,
+      "outputs": []
+    },
+    {
+      "cell_type": "markdown",
+      "source": [
+        "### Your Policy Brief Template\n",
+        "\n",
+        "**EXECUTIVE SUMMARY: US State Crime Profile Analysis**\n",
+        "\n",
+        "**Key Findings:**\n",
+        "- We identified [X] distinct crime profiles among US states\n",
+        "- [State examples] represent the highest-risk profile\n",
+        "- [State examples] represent the lowest-risk profile\n",
+        "- Urban population [does/does not] strongly correlate with violent crime\n",
+        "\n",
+        "**Policy Recommendations:**\n",
+        "1. **High-Priority States**: [List and explain why]\n",
+        "2. **Resource Allocation**: [Suggest how to distribute federal crime prevention funds]\n",
+        "3. **Best Practice Sharing**: [Which states should learn from which others?]\n",
+        "\n",
+        "**Methodology Note**: Analysis used unsupervised clustering on 4 crime variables across 50 states, with data standardization to ensure fair comparison.\n",
+        "\n",
+        "**Deliverable**: A complete 1-page policy brief with your clustering insights and specific recommendations.\n"
+      ],
+      "metadata": {
+        "id": "rAy_Ye0WYLK0"
+      },
+      "id": "rAy_Ye0WYLK0"
+    }
+  ],
+  "metadata": {
+    "jupytext": {
+      "cell_metadata_filter": "-all",
+      "formats": "Rmd,ipynb",
+      "main_language": "python"
+    },
+    "kernelspec": {
+      "display_name": "Python 3 (ipykernel)",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.10.4"
+    },
+    "colab": {
+      "provenance": []
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 5
+}

Reference files/week 7/w7_curriculum

@@ -0,0 +1,178 @@
+Unsupervised Learning: K-means and Hierarchical Clustering
+1. Course Overview
+The State Safety Profile Challenge
+In this week, we'll explore unsupervised machine learning through a compelling real-world challenge: Understanding crime patterns across US states without any predetermined categories.
+Unsupervised Learning: A type of machine learning where we find hidden patterns in data without being told what to look for. Think of it like being a detective who examines evidence without knowing what crime was committed - you're looking for patterns and connections that emerge naturally from the data.
+Example: Instead of being told "find violent states vs. peaceful states," unsupervised learning lets the data reveal its own natural groupings, like "states with high murder but low assault" or "urban states with moderate crime."
+Imagine you're a policy researcher working with the FBI's crime statistics. You have data on violent crime rates across all 50 US states - murder rates, assault rates, urban population percentages, and rape statistics. But here's the key challenge: you don't know how states naturally group together in terms of crime profiles.
+Your Mission: Discover hidden patterns in state crime profiles without any predefined classifications!
+The Challenge: Without any predetermined safety categories, you need to:
+●	Uncover natural groupings of states based on their crime characteristics
+●	Identify which crime factors tend to cluster together
+●	Understand regional patterns that might not follow obvious geographic boundaries
+●	Find states with surprisingly similar or different crime profiles
+Cluster: A group of similar things. In our case, states that have similar crime patterns naturally group together in a cluster.
+Example: You might discover that Alaska, Nevada, and Florida cluster together because they all have high crime rates despite being in different regions of the country.
+Why This Matters: Traditional approaches might group states by region (South, Northeast, etc.) or population size. But what if crime patterns reveal different natural groupings? What if some Southern states cluster more closely with Western states based on crime profiles? What if urban percentage affects crime differently than expected?
+Urban Percentage: The proportion of a state's population that lives in cities rather than rural areas.
+Example: New York has a high urban percentage (87%) while Wyoming has a low urban percentage (29%).
+What You'll Discover Through This Challenge
+●	Hidden State Safety Types: Use clustering to identify groups of states with similar crime profiles
+●	Crime Pattern Relationships: Find unexpected connections between different types of violent crime
+●	Urban vs. Rural Effects: Discover how urbanization relates to different crime patterns
+●	Policy Insights: Understand which states face similar challenges and might benefit from shared approaches
+Clustering: The process of grouping similar data points together. It's like organizing your music library - songs naturally group by genre, but clustering might reveal unexpected groups like "workout songs" or "rainy day music" that cross traditional genre boundaries.
+Core Techniques We'll Master
+K-Means Clustering: A method that divides data into exactly K groups (where you choose the number K). It's like being asked to organize 50 students into exactly 4 study groups based on their academic interests.
+Hierarchical Clustering: A method that creates a tree-like structure showing how data points relate to each other at different levels. It's like a family tree, but for data - showing which states are "cousins" and which are "distant relatives" in terms of crime patterns.
+Both K-Means and Hierarchical Clustering are examples of unsupervised learning.
+ 
+2. K-Means Clustering
+
+What it does: Divides data into exactly K groups by finding central points (centroids).
+Central Points (Centroids): The "center" or average point of each group. Think of it like the center of a basketball team huddle - it's the point that best represents where all the players are standing.
+Example: If you have a cluster of high-crime states, the centroid might represent "average murder rate of 8.5, average assault rate of 250, average urban population of 70%."
+USArrests Example: Analyzing crime data across 50 states, you might discover 4 distinct state safety profiles:
+●	High Crime States (above average in murder, assault, and rape rates)
+●	Urban Safe States (high urban population but lower violent crime rates)
+●	Rural Traditional States (low urban population, moderate crime rates)
+●	Mixed Profile States (high in some crime types but not others)
+How to Read K-Means Results:
+●	Scatter Plot: Points (states) colored by cluster membership
+○	Well-separated colors indicate distinct state profiles
+○	Mixed colors suggest overlapping crime patterns
+●	Cluster Centers: Average crime characteristics of each state group
+●	Elbow Plot: Helps choose optimal number of state groupings
+Cluster Membership: Which group each data point belongs to. Like being assigned to a team - each state gets assigned to exactly one crime profile group.
+Example: Texas might be assigned to "High Crime States" while Vermont is assigned to "Rural Traditional States."
+Scatter Plot: A graph where each point represents one observation (in our case, one state). Points that are close together have similar characteristics.
+Elbow Plot: A graph that helps you choose the right number of clusters. It's called "elbow" because you look for a bend in the line that looks like an elbow joint.
+Key Parameters:
+python
+# Essential parameters from the lab
+KMeans(
+    n_clusters=4,        # Number of state safety profiles to discover
+    random_state=42,     # For reproducible results
+    n_init=20           # Run algorithm 20 times, keep best result
+)
+Parameters: Settings that control how the algorithm works. Like settings on your phone - you can adjust them to get different results.
+n_clusters: How many groups you want to create. You have to decide this ahead of time.
+random_state: A number that ensures you get the same results every time you run the analysis. Like setting a specific starting point so everyone gets the same answer.
+n_init: How many times to run the algorithm. The computer tries multiple starting points and picks the best result. More tries = better results.
+
+3. Hierarchical Clustering
+What it does: Creates a tree structure (dendrogram) showing how data points group together at different levels.
+Dendrogram: A tree-like diagram that shows how groups form at different levels. Think of it like a family tree, but for data. At the bottom are individuals (states), and as you go up, you see how they group into families, then extended families, then larger clans.
+Example: At the bottom level, you might see Vermont and New Hampshire grouped together. Moving up, they might join with Maine to form a "New England Low Crime" group. Moving up further, this group might combine with other regional groups.
+USArrests Example: Analyzing state crime patterns might reveal:
+●	Level 1: High Crime vs. Low Crime states
+●	Level 2: Within high crime: Urban-driven vs. Rural-driven crime patterns
+●	Level 3: Within urban-driven: Assault-heavy vs. Murder-heavy profiles
+How to Read Dendrograms:
+●	Height: Distance between groups when they merge
+○	Higher merges = very different crime profiles
+○	Lower merges = similar crime patterns
+●	Branches: Each split shows a potential state grouping
+●	Cutting the Tree: Draw a horizontal line to create clusters
+Height: In a dendrogram, height represents how different two groups are. Think of it like difficulty level - it takes more "effort" (higher height) to combine very different groups.
+Example: Combining two very similar states (like Vermont and New Hampshire) happens at low height. Combining very different groups (like "High Crime States" and "Low Crime States") happens at high height.
+Cutting the Tree: Drawing a horizontal line across the dendrogram to create a specific number of groups. Like slicing a layer cake - where you cut determines how many pieces you get.
+Three Linkage Methods:
+●	Complete Linkage: Measures distance between most different states (good for distinct profiles)
+●	Average Linkage: Uses average distance between all states (balanced approach)
+●	Single Linkage: Uses closest states (tends to create chains, often less useful)
+Linkage Methods: Different ways to measure how close or far apart groups are. It's like different ways to measure the distance between two cities - you could use the distance between the farthest suburbs (complete), the average distance between all neighborhoods (average), or the distance between the closest points (single).
+Example: When deciding if "High Crime Group" and "Medium Crime Group" should merge, complete linkage looks at the most different states between the groups, while average linkage looks at the typical difference.
+Choosing Between K-Means and Hierarchical:
+●	Use K-Means when: You want to segment states into specific number of safety categories for policy targeting
+●	Use Hierarchical when: You want to explore the natural structure of crime patterns without assumptions
+Segmentation: Dividing your data into groups for specific purposes. Like organizing students into study groups - you might want exactly 4 groups so each has a teaching assistant.
+Exploratory Analysis: Looking at data to discover patterns without knowing what you'll find. Like being an explorer in uncharted territory - you're not looking for a specific destination, just seeing what interesting things you can discover.
+ 
+4. Data Exploration
+Step 1: Understanding Your Data
+Essential Checks (from the USArrests example):
+python
+# Check the basic structure
+print(data.shape)          # How many observations and variables?
+print(data.columns)        # What variables do you have?
+print(data.head())         # What do the first few rows look like?
+
+# Examine the distribution
+print(data.mean())         # Average values
+print(data.var())          # Variability
+print(data.describe())     # Full statistical summary
+Observations: Individual data points we're studying. In our case, each of the 50 US states is one observation.
+Variables: The characteristics we're measuring for each observation. In USArrests, we have 4 variables: Murder rate, Assault rate, Urban Population percentage, and Rape rate.
+Example: For California (one observation), we might have Murder=9.0, Assault=276, UrbanPop=91, Rape=40.6 (four variables).
+Distribution: How values are spread out. Like looking at test scores in a class - are most scores clustered around the average, or spread out widely?
+Variability (Variance): How much the values differ from each other. High variance means values are spread out; low variance means they're clustered together.
+Why This Matters: The USArrests data showed vastly different scales:
+●	Murder: Average 7.8, Variance 19
+●	Assault: Average 170.8, Variance 6,945
+●	This scale difference would dominate any analysis without preprocessing
+Scales: The range and units of measurement for different variables. Like comparing dollars ($50,000 salary) to percentages (75% approval rating) - they're measured very differently.
+Example: Assault rates are in the hundreds (like 276 per 100,000) while murder rates are single digits (like 7.8 per 100,000). Without adjustment, assault would seem much more important just because the numbers are bigger.
+Step 2: Data Preprocessing
+Standardization (Critical for clustering):
+python
+from sklearn.preprocessing import StandardScaler
+
+# Always scale when variables have different units
+scaler = StandardScaler()
+data_scaled = scaler.fit_transform(data)
+Standardization: Converting all variables to the same scale so they can be fairly compared. Like converting all measurements to the same units - instead of comparing feet to meters, you convert everything to inches.
+StandardScaler: A tool that transforms data so each variable has an average of 0 and standard deviation of 1. Think of it like grading on a curve - it makes all variables equally important.
+Example: After standardization, a murder rate of 7.8 might become 0.2, and an assault rate of 276 might become 1.5. Now they're on comparable scales.
+When to Scale:
+●	✅ Always scale when variables have different units (dollars vs. percentages)
+●	✅ Scale when variances differ by orders of magnitude
+●	❓ Consider not scaling when all variables are in the same meaningful units
+Orders of Magnitude: When one number is 10 times, 100 times, or 1000 times bigger than another. In USArrests, assault variance (6,945) is about 365 times bigger than murder variance (19) - that's two orders of magnitude difference.
+Step 3: Exploratory Analysis
+For K-Means Clustering:
+python
+# Try different numbers of clusters to find optimal K
+inertias = []
+K_range = range(1, 11)
+for k in K_range:
+    kmeans = KMeans(n_clusters=k, random_state=42, n_init=20)
+    kmeans.fit(data_scaled)
+    inertias.append(kmeans.inertia_)
+
+# Plot elbow curve
+plt.plot(K_range, inertias, 'bo-')
+plt.xlabel('Number of Clusters (K)')
+plt.ylabel('Within-Cluster Sum of Squares')
+plt.title('Elbow Method for Optimal K')
+Inertias: A measure of how tightly grouped each cluster is. Lower inertia means points in each cluster are closer together (better clustering). It's like measuring how close teammates stand to each other - closer teammates indicate better team cohesion.
+Within-Cluster Sum of Squares: The total distance from each point to its cluster center. Think of it as measuring how far each student sits from their group's center - smaller distances mean tighter, more cohesive groups.
+Elbow Method: A technique for choosing the best number of clusters. You plot the results and look for the "elbow" - the point where adding more clusters doesn't help much anymore.
+For Hierarchical Clustering:
+python
+# Create dendrogram to explore natural groupings
+from sklearn.cluster import AgglomerativeClustering
+from ISLP.cluster import compute_linkage
+from scipy.cluster.hierarchy import dendrogram
+
+hc = AgglomerativeClustering(distance_threshold=0, n_clusters=None, linkage='complete')
+hc.fit(data_scaled)
+linkage_matrix = compute_linkage(hc)
+
+plt.figure(figsize=(12, 8))
+dendrogram(linkage_matrix, color_threshold=-np.inf, above_threshold_color='black')
+plt.title('Hierarchical Clustering Dendrogram')
+AgglomerativeClustering: A type of hierarchical clustering that starts with individual points and gradually combines them into larger groups. Like building a pyramid from the bottom up.
+distance_threshold=0: A setting that tells the algorithm to build the complete tree structure without stopping early.
+Linkage Matrix: A mathematical representation of how the tree structure was built. Think of it as the blueprint showing how the dendrogram was constructed.
+Step 4: Validation Questions
+Before proceeding with analysis, ask:
+1.	Do the variables make sense together? (e.g., don't cluster height with income)
+2.	Are there obvious outliers that need attention?
+3.	Do you have enough data points? (Rule of thumb: at least 10x more observations than variables)
+4.	Are there missing values that need handling?
+Outliers: Data points that are very different from all the others. Like a 7-foot-tall person in a group of average-height people - they're so different they might skew your analysis.
+Example: If most states have murder rates between 1-15, but one state has a rate of 50, that's probably an outlier that needs special attention.
+Missing Values: Data points where we don't have complete information. Like a student who didn't take one of the tests - you need to decide how to handle that gap in the data.
+Rule of Thumb: A general guideline that works in most situations. For clustering, having at least 10 times more observations than variables helps ensure reliable results.
+

app/main.py

@@ -29,9 +29,24 @@ st.set_page_config(
     page_title="Data Science Course App",
     page_icon="📚",
     layout="wide",
-    initial_sidebar_state="expanded"
+    initial_sidebar_state="expanded",
+    menu_items={
+        'Get Help': None,
+        'Report a bug': None,
+        'About': None
+    }
 )
 
+# Disable URL paths and hide Streamlit elements
+st.markdown("""
+    <style>
+        #MainMenu {visibility: hidden;}
+        footer {visibility: hidden;}
+        .stDeployButton {display: none;}
+        .viewerBadge_container__1QSob {display: none;}
+    </style>
+""", unsafe_allow_html=True)
+
 # Custom CSS
 def load_css():
     try:
@@ -56,11 +71,6 @@ def load_css():
             margin-bottom: 1rem;
         }
         
-        /* Sidebar styling */
-        .sidebar .sidebar-content {
-            background-color: #f8f9fa;
-        }
-        
         /* Button styling */
         .stButton>button {
             width: 100%;
@@ -125,16 +135,17 @@ def sidebar_navigation():
                 st.rerun()
         
         st.markdown("---")
-        st.subheader("Course Progress")
-        progress = st.progress(st.session_state.current_week / 10)
-        st.write(f"Week {st.session_state.current_week} of 10")
+        st.subheader("Course Content")
         
-        st.markdown("---")
-        st.subheader("Quick Links")
-        for week in range(1, 11):
-            if st.button(f"Week {week}", key=f"week_{week}"):
-                st.session_state.current_week = week
-                st.rerun()
+        # Create a container for week buttons
+        week_container = st.container()
+        
+        # Add week buttons with custom styling
+        with week_container:
+            for week in range(1, 11):
+                if st.button(f"Week {week}", key=f"week_{week}"):
+                    st.session_state.current_week = week
+                    st.rerun()
 
 def show_week_content():
     # Debug print to show current week

app/pages/week_7.py

@@ -6,16 +6,21 @@ import seaborn as sns
 import plotly.express as px
 import plotly.graph_objects as go
 from plotly.subplots import make_subplots
+from sklearn.cluster import KMeans
+from sklearn.neighbors import KNeighborsClassifier
+from sklearn.preprocessing import StandardScaler
+from sklearn.metrics import silhouette_score
+from statsmodels.datasets import get_rdataset
+from scipy.cluster.hierarchy import dendrogram, linkage, fcluster
 
 # Set up the style for all plots
 plt.style.use('default')
 sns.set_theme(style="whitegrid", palette="husl")
 
-def load_titanic_data():
-    """Load and return the Titanic dataset"""
-    url = "https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv"
-    df = pd.read_csv(url)
-    return df
+def load_arrests_data():
+    """Load and return the US Arrests dataset"""
+    USArrests = get_rdataset('USArrests').data
+    return USArrests
 
 def create_categorical_plot(df, column, target='Survived'):
     """Create an interactive plot for categorical variables"""
@@ -54,284 +59,758 @@ def create_numeric_plot(df, column, target='Survived'):
     return fig
 
 def show():
-    st.title("Week 7: Data Cleaning and EDA with Categorical Variables")
+    st.title("Week 7: Clustering Lab - State Crime Pattern Analysis")
     
-    # Introduction Section
-    st.header("Course Overview")
-    st.write("""
-    This week, we'll explore data cleaning and exploratory data analysis (EDA) with a focus on categorical variables.
-    We'll use the Titanic dataset to demonstrate:
-    - Data cleaning techniques
-    - Handling missing values
-    - Analyzing categorical variables
-    - Creating meaningful visualizations
-    - Feature engineering
+    # Code Example: Loading and Basic Data Exploration
+    with st.expander("Code Example: Loading and Exploring Data"):
+        st.code("""
+# Load the data
+from statsmodels.datasets import get_rdataset
+USArrests = get_rdataset('USArrests').data
+
+# Basic data exploration
+print("Dataset shape:", USArrests.shape)
+print("\\nVariables:", USArrests.columns.tolist())
+print("\\nFirst 5 states:")
+print(USArrests.head())
+
+# Basic statistics
+print("\\nData Summary:")
+print(USArrests.describe())
+        """, language="python")
+    
+    # Introduction Section with Learning Objectives
+    st.header("Learning Objectives")
+    st.markdown("""
+    In this week, you'll master:
+    1. **Unsupervised Learning**: Discover hidden patterns in crime data without predefined categories
+    2. **K-Means Clustering**: Learn to divide states into distinct safety profiles
+    3. **Hierarchical Clustering**: Create a "family tree" of state crime patterns
+    4. **Data Preprocessing**: Understand why scaling is crucial for fair comparisons
     """)
     
-    # Learning Path
-    st.subheader("Learning Path")
+    # Interactive Overview
+    st.header("Lab Overview")
     st.write("""
-    1. Understanding the Dataset: Titanic passenger data
-    2. Data Cleaning: Handling missing values and outliers
-    3. Categorical Variables: Analysis and visualization
-    4. Feature Engineering: Creating new features
-    5. Data Visualization: Interactive plots and insights
-    6. Practical Applications: Real-world data analysis
+    Welcome to your hands-on clustering lab! You'll be working as a policy analyst for the Department of Justice, 
+    analyzing crime patterns across US states. Your mission: discover hidden safety profiles that could inform 
+    federal resource allocation and crime prevention strategies.
     """)
-
+    
     # Load Data
-    st.header("The Dataset")
-    st.write("""
-    We'll be working with the Titanic dataset, which contains information about passengers aboard the Titanic.
-    The dataset includes both categorical and numerical variables, making it perfect for learning data cleaning and EDA.
+    st.header("Exercise 1: Data Detective Work")
+    st.write("Let's start by understanding our dataset - the US Arrests data.")
+    
+    df = load_arrests_data()
+    
+    # Code Example: Data Visualization
+    with st.expander("Code Example: Creating Visualizations"):
+        st.code("""
+# Create correlation heatmap
+import plotly.express as px
+fig = px.imshow(df.corr(), 
+                labels=dict(color="Correlation"),
+                color_continuous_scale="RdBu")
+fig.show()
+
+# Create box plots
+fig = px.box(df, title="Data Distribution")
+fig.show()
+        """, language="python")
+    
+    # Interactive Data Exploration
+    col1, col2 = st.columns(2)
+    
+    with col1:
+        st.subheader("Dataset Overview")
+        st.write(f"Number of states: {len(df)}")
+        st.write(f"Number of variables: {len(df.columns)}")
+        st.write("\nVariables:", df.columns.tolist())
+        
+        # Interactive data summary
+        st.subheader("Data Summary")
+        summary = df.describe()
+        st.dataframe(summary)
+    
+    with col2:
+        st.subheader("First 5 States")
+        st.dataframe(df.head())
+        
+        # Interactive correlation heatmap
+        st.subheader("Correlation Heatmap")
+        fig = px.imshow(df.corr(), 
+                       labels=dict(color="Correlation"),
+                       color_continuous_scale="RdBu")
+        st.plotly_chart(fig)
+    
+    # Exercise 2: Scaling Challenge
+    st.header("Exercise 2: The Scaling Challenge")
+    
+    # Code Example: Data Scaling
+    with st.expander("Code Example: Scaling Data"):
+        st.code("""
+# Import StandardScaler
+from sklearn.preprocessing import StandardScaler
+
+# Create and fit the scaler
+scaler = StandardScaler()
+df_scaled = scaler.fit_transform(df)
+
+# Convert back to DataFrame
+df_scaled = pd.DataFrame(df_scaled, 
+                        columns=df.columns, 
+                        index=df.index)
+
+# Compare original vs scaled data
+print("Original data ranges:")
+print(df.describe())
+print("\\nScaled data ranges:")
+print(df_scaled.describe())
+        """, language="python")
+    
+    # Explanation of scaling
+    st.markdown("""
+    ### Why Do We Need Scaling?
+    
+    In our crime data, we have variables measured in very different scales:
+    - Murder rates: typically 0-20 per 100,000
+    - Assault rates: typically 50-350 per 100,000
+    - Urban population: 0-100 percentage
+    - Rape rates: typically 0-50 per 100,000
+    
+    Without scaling, variables with larger numbers (like Assault) would dominate our analysis, 
+    making smaller-scale variables (like Murder) less influential. This would be like comparing 
+    dollars to cents - the cents would seem insignificant even if they were important!
+    """)
+    
+    # Show original data ranges
+    st.subheader("Original Data Ranges")
+    col1, col2 = st.columns(2)
+    
+    with col1:
+        # Create a bar chart of variances
+        fig_var = px.bar(
+            x=df.columns,
+            y=df.var(),
+            title="Variance of Each Variable (Before Scaling)",
+            labels={'x': 'Crime Variables', 'y': 'Variance'},
+            color=df.var(),
+            color_continuous_scale='Viridis'
+        )
+        st.plotly_chart(fig_var)
+        
+        st.write("""
+        Notice how Assault has a much larger variance (6,945) compared to Murder (19).
+        This means Assault would dominate our clustering if we didn't scale the data!
+        """)
+    
+    with col2:
+        # Create box plots of original data
+        fig_box = px.box(df, title="Original Data Distribution")
+        fig_box.update_layout(
+            xaxis_title="Crime Variables",
+            yaxis_title="Rate per 100,000"
+        )
+        st.plotly_chart(fig_box)
+    
+    # Explain standardization
+    st.markdown("""
+    ### What is Standardization?
+    
+    Standardization (also called Z-score normalization) transforms our data so that:
+    1. Each variable has a mean of 0
+    2. Each variable has a standard deviation of 1
+    
+    The formula is: z = (x - μ) / σ
+    - x is the original value
+    - μ is the mean of the variable
+    - σ is the standard deviation of the variable
     """)
     
-    df = load_titanic_data()
-    
-    # Display basic information
-    st.subheader("Dataset Overview")
-    st.write(f"Number of rows: {len(df)}")
-    st.write(f"Number of columns: {len(df.columns)}")
-    
-    # Display missing values
-    st.subheader("Missing Values Analysis")
-    missing_values = df.isnull().sum()
-    fig_missing = px.bar(
-        x=missing_values.index,
-        y=missing_values.values,
-        title='Missing Values by Column',
-        labels={'x': 'Columns', 'y': 'Number of Missing Values'}
+    # Scale the data
+    scaler = StandardScaler()
+    df_scaled = scaler.fit_transform(df)
+    df_scaled = pd.DataFrame(df_scaled, columns=df.columns, index=df.index)
+    
+    # Show scaled data
+    st.subheader("After Scaling")
+    
+    # Create box plots of scaled data
+    fig_scaled = px.box(df_scaled, title="Scaled Data Distribution")
+    fig_scaled.update_layout(
+        xaxis_title="Crime Variables",
+        yaxis_title="Standardized Values"
     )
-    fig_missing.update_layout(
-        title_x=0.5,
-        title_font_size=20,
+    st.plotly_chart(fig_scaled)
+    
+    st.write("""
+    After scaling, all variables are on the same scale:
+    - Mean = 0
+    - Standard Deviation = 1
+    - Values typically range from -3 to +3
+    """)
+    
+    # Show before/after comparison for a few states
+    st.write("### Before vs After Scaling (Sample States)")
+    comparison_df = pd.DataFrame({
+        'State': df.index[:5],
+        'Original Murder': df['Murder'][:5],
+        'Scaled Murder': df_scaled['Murder'][:5],
+        'Original Assault': df['Assault'][:5],
+        'Scaled Assault': df_scaled['Assault'][:5]
+    })
+    st.dataframe(comparison_df)
+    
+    st.write("""
+    Notice how the relative differences between states are preserved,
+    but now all variables contribute equally to our analysis!
+    """)
+    
+    # Why scaling matters for clustering
+    st.markdown("""
+    ### Why Scaling Matters for Clustering
+    
+    In clustering, we measure distances between data points. Without scaling:
+    - States might be grouped together just because they have similar assault rates
+    - Important differences in murder rates might be ignored
+    
+    With scaling:
+    - All variables contribute equally to the distance calculations
+    - We can find true patterns in the data, not just patterns in the largest numbers
+    """)
+    
+    # Exercise 3: Finding Optimal Clusters
+    st.header("Exercise 3: Finding the Right Number of Groups")
+    
+    # Code Example: Elbow Method
+    with st.expander("Code Example: Finding Optimal K"):
+        st.code("""
+# Calculate inertias for different K values
+inertias = []
+K_values = range(1, 11)
+
+for k in K_values:
+    kmeans = KMeans(n_clusters=k, random_state=42, n_init=20)
+    kmeans.fit(df_scaled)
+    inertias.append(kmeans.inertia_)
+
+# Create elbow plot
+import plotly.graph_objects as go
+fig = go.Figure()
+fig.add_trace(go.Scatter(
+    x=list(K_values),
+    y=inertias,
+    mode='lines+markers',
+    name='Inertia'
+))
+fig.update_layout(
+    title='Finding the Optimal Number of Clusters',
+    xaxis_title='Number of Clusters (K)',
+    yaxis_title='Within-Cluster Sum of Squares'
+)
+fig.show()
+        """, language="python")
+    
+    st.markdown("""
+    ### The Elbow Method Explained
+    
+    The elbow method helps us find the optimal number of clusters (K) by looking at how the "within-cluster sum of squares" 
+    (WCSS) changes as we increase the number of clusters. Think of it like this:
+    
+    - **What is WCSS?** It's a measure of how spread out the points are within each cluster
+    - **Lower WCSS** means points are closer to their cluster center (better clustering)
+    - **Higher WCSS** means points are more spread out from their cluster center
+    
+    As we increase K:
+    1. WCSS always decreases (more clusters = tighter groups)
+    2. The rate of decrease slows down
+    3. We look for the "elbow" - where adding more clusters doesn't help much anymore
+    """)
+    
+    # Calculate inertias for different K values
+    inertias = []
+    K_values = range(1, 11)
+    
+    for k in K_values:
+        kmeans = KMeans(n_clusters=k, random_state=42, n_init=20)
+        kmeans.fit(df_scaled)
+        inertias.append(kmeans.inertia_)
+    
+    # Create interactive elbow plot
+    fig_elbow = go.Figure()
+    fig_elbow.add_trace(go.Scatter(
+        x=list(K_values),
+        y=inertias,
+        mode='lines+markers',
+        name='Inertia'
+    ))
+    fig_elbow.update_layout(
+        title='Finding the Optimal Number of State Crime Profiles',
+        xaxis_title='Number of Clusters (K)',
+        yaxis_title='Within-Cluster Sum of Squares',
         plot_bgcolor='rgb(30, 30, 30)',
         paper_bgcolor='rgb(30, 30, 30)',
         font=dict(color='white')
     )
-    st.plotly_chart(fig_missing)
+    st.plotly_chart(fig_elbow)
+    
+    # Interpretation guide
+    st.markdown("""
+    ### How to Interpret the Elbow Plot
+    
+    Look at the plot above and ask yourself:
+    1. **Where is the "elbow"?** 
+       - The point where the line starts to level off
+       - Adding more clusters doesn't give much improvement
+       - In our case, it's around K=4
+    
+    2. **What do the numbers mean?**
+       - K=1: All states in one group (not useful)
+       - K=2: Basic high/low crime split
+       - K=3: More nuanced grouping
+       - K=4: Our "elbow" - good balance of detail and simplicity
+       - K>4: Diminishing returns - more complexity without much benefit
+    
+    3. **Why not just use more clusters?**
+       - More clusters = more complex to interpret
+       - Small clusters might not be meaningful
+       - Goal is to find the simplest model that captures the main patterns
+    """)
+    
+    # Show the actual values
+    st.write("### WCSS Values for Each K")
+    wcss_df = pd.DataFrame({
+        'Number of Clusters (K)': K_values,
+        'Within-Cluster Sum of Squares': inertias,
+        'Improvement from Previous K': [0] + [inertias[i-1] - inertias[i] for i in range(1, len(inertias))]
+    })
+    st.dataframe(wcss_df)
+    
+    st.markdown("""
+    ### Making the Decision
+    
+    Based on our elbow plot and the numbers above:
+    1. The biggest improvements happen from K=1 to K=4
+    2. After K=4, the improvements get much smaller
+    3. K=4 gives us a good balance of:
+       - Capturing meaningful patterns
+       - Keeping the model simple enough to interpret
+       - Having enough states in each cluster to be meaningful
+    
+    This is why we'll use K=4 for our clustering analysis!
+    """)
+    
+    # Exercise 4: K-Means Clustering
+    st.header("Exercise 4: K-Means State Profiling")
+    
+    # Code Example: K-Means Clustering
+    with st.expander("Code Example: K-Means Implementation"):
+        st.code("""
+# Perform K-means clustering
+from sklearn.cluster import KMeans
+
+# Create and fit the model
+kmeans = KMeans(
+    n_clusters=4,        # Number of clusters
+    random_state=42,     # For reproducibility
+    n_init=20           # Number of times to run with different centroids
+)
+cluster_labels = kmeans.fit_predict(df_scaled)
+
+# Add cluster labels to original data
+df_clustered = df.copy()
+df_clustered['Cluster'] = cluster_labels
+
+# Visualize the clusters
+import plotly.express as px
+fig = px.scatter(df_clustered, 
+                x='Murder', 
+                y='Assault',
+                color='Cluster',
+                hover_data=['UrbanPop', 'Rape'],
+                title='State Crime Profiles')
+fig.show()
+
+# Show cluster centers
+centers_df = pd.DataFrame(
+    kmeans.cluster_centers_,
+    columns=df.columns
+)
+print("Cluster Centers:")
+print(centers_df)
+        """, language="python")
     
-    # Data Cleaning Section
-    st.header("Data Cleaning")
+    st.markdown("""
+    ### What is K-Means Clustering?
     
-    # Handle missing values
-    st.subheader("Handling Missing Values")
+    K-means is an unsupervised learning algorithm that groups similar data points together. Think of it like organizing 
+    students into study groups based on their interests:
+    
+    1. **Initialization**: 
+       - We randomly place K "centers" (centroids) in our data space
+       - Each center represents the "average" of its cluster
+       - In our case, each center represents a typical crime profile
+    
+    2. **Assignment**:
+       - Each state is assigned to its nearest center
+       - "Nearest" is measured by Euclidean distance
+       - States with similar crime patterns end up in the same cluster
+    
+    3. **Update**:
+       - Centers move to the average position of their assigned states
+       - This process repeats until centers stop moving
+       - The algorithm converges when states are optimally grouped
+    """)
+    
+    # Visualize the process
+    st.subheader("K-Means in Action")
     st.write("""
-    Let's clean the data by:
-    1. Filling missing Age values with median
-    2. Filling missing Embarked values with mode
-    3. Creating a new feature for Cabin availability
+    Let's see how K-means works with our state crime data. We'll use K=4 clusters to find distinct crime profiles.
     """)
     
-    # Create a copy for cleaning
-    df_cleaned = df.copy()
+    # Let user choose number of clusters
+    k = st.slider("Choose number of clusters (K)", 2, 6, 4)
     
-    # Fill missing values
-    df_cleaned['Age'].fillna(df_cleaned['Age'].median(), inplace=True)
-    df_cleaned['Embarked'].fillna(df_cleaned['Embarked'].mode()[0], inplace=True)
-    df_cleaned['HasCabin'] = df_cleaned['Cabin'].notna().astype(int)
+    # Perform K-means clustering
+    kmeans = KMeans(n_clusters=k, random_state=42, n_init=20)
+    cluster_labels = kmeans.fit_predict(df_scaled)
     
-    # Categorical Variables Analysis
-    st.header("Categorical Variables Analysis")
+    # Add cluster labels to original data
+    df_clustered = df.copy()
+    df_clustered['Cluster'] = cluster_labels
     
-    # Select categorical column to analyze
-    categorical_cols = ['Pclass', 'Sex', 'Embarked', 'HasCabin']
-    selected_col = st.selectbox(
-        "Select Categorical Variable to Analyze",
-        categorical_cols
-    )
+    # Create interactive scatter plot
+    fig = px.scatter(df_clustered, 
+                    x='Murder', 
+                    y='Assault',
+                    color='Cluster',
+                    hover_data=['UrbanPop', 'Rape'],
+                    title='State Crime Profiles')
+    st.plotly_chart(fig)
     
-    # Create and display categorical plot
-    fig_cat = create_categorical_plot(df_cleaned, selected_col)
-    st.plotly_chart(fig_cat)
+    # Explain hyperparameters
+    st.markdown("""
+    ### K-Means Hyperparameters Explained
     
-    # Numeric Variables Analysis
-    st.header("Numeric Variables Analysis")
+    1. **n_clusters (K)**
+       - The number of groups we want to create
+       - We chose K=4 based on the elbow method
+       - Each cluster represents a distinct crime profile
     
-    # Select numeric column to analyze
-    numeric_cols = ['Age', 'Fare', 'SibSp', 'Parch']
-    selected_num_col = st.selectbox(
-        "Select Numeric Variable to Analyze",
-        numeric_cols
-    )
+    2. **random_state**
+       - Controls the random initialization of centroids
+       - Setting it to 42 ensures reproducible results
+       - Different values might give slightly different clusters
+    
+    3. **n_init**
+       - Number of times to run the algorithm with different initial centroids
+       - We use 20 to find the best possible clustering
+       - Higher values give more reliable results but take longer
+    
+    4. **max_iter**
+       - Maximum number of iterations for each run
+       - Default is 300, which is usually enough
+       - Algorithm stops earlier if it converges
+    
+    5. **algorithm**
+       - 'auto': Automatically chooses the best algorithm
+       - 'full': Traditional K-means
+       - 'elkan': More efficient for well-separated clusters
+    """)
     
-    # Create and display numeric plot
-    fig_num = create_numeric_plot(df_cleaned, selected_num_col)
-    st.plotly_chart(fig_num)
+    # Show cluster centers
+    st.subheader("Cluster Centers (Typical Crime Profiles)")
+    centers_df = pd.DataFrame(
+        kmeans.cluster_centers_,
+        columns=df.columns
+    )
+    st.dataframe(centers_df)
     
-    # Reference Code Section
-    st.header("Reference Code")
     st.write("""
-    Below is the reference code for the data cleaning and analysis we just performed.
-    Study this code to understand how we implemented the analysis.
+    Each row represents the "average" crime profile for that cluster. For example:
+    - High values in Murder and Assault indicate a high-crime cluster
+    - High UrbanPop with low crime rates might indicate urban safety
+    - Low values across all metrics might indicate rural safety
     """)
     
-    with st.expander("View Reference Code"):
+    # Display cluster analysis
+    st.subheader("State Crime Profiles Analysis")
+    
+    for cluster_num in range(k):
+        cluster_states = df_clustered[df_clustered['Cluster'] == cluster_num]
+        st.write(f"\n**CLUSTER {cluster_num}: {len(cluster_states)} states**")
+        st.write("States:", ", ".join(cluster_states.index.tolist()))
+        st.write("Average characteristics:")
+        avg_profile = cluster_states[['Murder', 'Assault', 'UrbanPop', 'Rape']].mean()
+        st.write(avg_profile)
+    
+    # Explain the results
+    st.markdown("""
+    ### Interpreting the Results
+    
+    Each cluster represents a distinct crime profile:
+    1. **Cluster Characteristics**
+       - Look at the average values for each crime type
+       - Compare urban population percentages
+       - Identify the defining features of each cluster
+    
+    2. **State Groupings**
+       - States in the same cluster have similar crime patterns
+       - Geographic proximity doesn't always mean similar profiles
+       - Some states might surprise you with their cluster membership
+    
+    3. **Policy Implications**
+       - Clusters help identify states with similar challenges
+       - Can guide resource allocation and policy development
+       - Enables targeted interventions based on crime profiles
+    """)
+    
+    # Exercise 5: Hierarchical Clustering
+    st.header("Exercise 5: Hierarchical Clustering Exploration")
+    
+    # Code Example: Hierarchical Clustering
+    with st.expander("Code Example: Hierarchical Clustering"):
         st.code("""
-# Data Cleaning
-df_cleaned = df.copy()
-df_cleaned['Age'].fillna(df_cleaned['Age'].median(), inplace=True)
-df_cleaned['Embarked'].fillna(df_cleaned['Embarked'].mode()[0], inplace=True)
-df_cleaned['HasCabin'] = df_cleaned['Cabin'].notna().astype(int)
+# Create hierarchical clustering
+from scipy.cluster.hierarchy import linkage, dendrogram
 
-# Categorical Analysis
-def create_categorical_plot(df, column, target='Survived'):
-    fig = px.bar(
-        df.groupby(column)[target].mean().reset_index(),
-        x=column,
-        y=target,
-        title=f'Survival Rate by {column}',
-        labels={target: 'Survival Rate', column: column},
-        color=target,
-        color_continuous_scale='RdBu'
-    )
-    return fig
+# Create linkage matrix
+linkage_matrix = linkage(df_scaled, method='complete')
 
-# Numeric Analysis
-def create_numeric_plot(df, column, target='Survived'):
-    fig = px.box(
-        df,
-        x=target,
-        y=column,
-        title=f'{column} Distribution by Survival',
-        labels={target: 'Survived', column: column},
-        color=target,
-        color_discrete_sequence=px.colors.qualitative.Set1
-    )
-    return fig
+# Plot dendrogram
+import plotly.graph_objects as go
+dendro = dendrogram(linkage_matrix, labels=df.index.tolist(), no_plot=True)
 
-# Feature Engineering
-df_cleaned['FamilySize'] = df_cleaned['SibSp'] + df_cleaned['Parch'] + 1
-df_cleaned['AgeGroup'] = pd.cut(
-    df_cleaned['Age'],
-    bins=[0, 12, 18, 35, 60, 100],
-    labels=['Child', 'Teenager', 'Young Adult', 'Adult', 'Senior']
+fig = go.Figure()
+fig.add_trace(go.Scatter(
+    x=dendro['icoord'],
+    y=dendro['dcoord'],
+    mode='lines',
+    line=dict(color='white')
+))
+fig.update_layout(
+    title='State Crime Pattern Family Tree',
+    xaxis_title='States',
+    yaxis_title='Distance Between Groups'
 )
-df_cleaned['FarePerPerson'] = df_cleaned['Fare'] / df_cleaned['FamilySize']
+fig.show()
+
+# Cut the tree to get clusters
+from scipy.cluster.hierarchy import fcluster
+hierarchical_labels = fcluster(linkage_matrix, k, criterion='maxclust') - 1
         """, language="python")
     
-    # Knowledge Check Quiz
-    st.header("Knowledge Check")
-    st.write("Test your understanding of the concepts covered in this section.")
-    
-    # Initialize session state for quiz if not exists
-    if 'quiz_submitted' not in st.session_state:
-        st.session_state.quiz_submitted = False
-    
-    # Quiz questions
-    questions = {
-        "q1": {
-            "question": "What is the best way to handle missing values in the 'Age' column?",
-            "options": [
-                "Fill with 0",
-                "Fill with the median age",
-                "Remove all rows with missing age",
-                "Fill with the mean age"
-            ],
-            "correct": 1
-        },
-        "q2": {
-            "question": "Why do we create the 'HasCabin' feature?",
-            "options": [
-                "To reduce the number of missing values",
-                "To create a binary indicator for cabin availability",
-                "To make the data more complex",
-                "To remove the Cabin column"
-            ],
-            "correct": 1
-        },
-        "q3": {
-            "question": "What does the FamilySize feature represent?",
-            "options": [
-                "Number of siblings only",
-                "Number of parents only",
-                "Total family members (including the passenger)",
-                "Number of children only"
-            ],
-            "correct": 2
-        }
-    }
-    
-    # Display quiz if not submitted
-    if not st.session_state.quiz_submitted:
-        answers = {}
-        for q_id, q_data in questions.items():
-            st.write(f"**{q_data['question']}**")
-            answers[q_id] = st.radio(
-                "Select your answer:",
-                q_data["options"],
-                key=q_id
-            )
-        
-        if st.button("Submit Quiz"):
-            # Calculate score
-            score = sum(1 for q_id, q_data in questions.items() 
-                       if answers[q_id] == q_data["options"][q_data["correct"]])
-            
-            # Show results
-            st.write(f"Your score: {score}/{len(questions)}")
-            
-            # Show correct answers
-            st.write("Correct answers:")
-            for q_id, q_data in questions.items():
-                st.write(f"- {q_data['question']}")
-                st.write(f"  Correct answer: {q_data['options'][q_data['correct']]}")
-            
-            st.session_state.quiz_submitted = True
-    
-    # Reset quiz button
-    if st.session_state.quiz_submitted:
-        if st.button("Take Quiz Again"):
-            st.session_state.quiz_submitted = False
-            st.rerun()
-    
-    # Feature Engineering
-    st.header("Feature Engineering")
-    st.write("""
-    Let's create some new features:
-    1. Family Size = SibSp + Parch + 1
-    2. Age Groups
-    3. Fare per Person
+    st.markdown("""
+    ### What is Hierarchical Clustering?
+    
+    Hierarchical clustering creates a tree-like structure (dendrogram) that shows how data points are related at different levels. 
+    Think of it like building a family tree for states based on their crime patterns:
+    
+    1. **Bottom-Up Approach (Agglomerative)**:
+       - Start with each state as its own cluster
+       - Find the two closest states and merge them
+       - Continue merging until all states are in one cluster
+       - Creates a complete hierarchy of relationships
+    
+    2. **Distance Measurement**:
+       - Complete Linkage: Uses the maximum distance between states
+       - Average Linkage: Uses the average distance between states
+       - Single Linkage: Uses the minimum distance between states
+       - We use complete linkage for more distinct clusters
     """)
     
-    # Create new features
-    df_cleaned['FamilySize'] = df_cleaned['SibSp'] + df_cleaned['Parch'] + 1
-    df_cleaned['AgeGroup'] = pd.cut(
-        df_cleaned['Age'],
-        bins=[0, 12, 18, 35, 60, 100],
-        labels=['Child', 'Teenager', 'Young Adult', 'Adult', 'Senior']
+    # Create hierarchical clustering
+    linkage_matrix = linkage(df_scaled, method='complete')
+    
+    # Create interactive dendrogram
+    fig_dendro = go.Figure()
+    dendro = dendrogram(linkage_matrix, labels=df.index.tolist(), no_plot=True)
+    
+    fig_dendro.add_trace(go.Scatter(
+        x=dendro['icoord'],
+        y=dendro['dcoord'],
+        mode='lines',
+        line=dict(color='white')
+    ))
+    
+    fig_dendro.update_layout(
+        title='State Crime Pattern Family Tree',
+        xaxis_title='States',
+        yaxis_title='Distance Between Groups',
+        plot_bgcolor='rgb(30, 30, 30)',
+        paper_bgcolor='rgb(30, 30, 30)',
+        font=dict(color='white')
     )
-    df_cleaned['FarePerPerson'] = df_cleaned['Fare'] / df_cleaned['FamilySize']
+    st.plotly_chart(fig_dendro)
+    
+    # Explain how to read the dendrogram
+    st.markdown("""
+    ### How to Read the Dendrogram
+    
+    1. **Height of Connections**:
+       - Higher connections = more different groups
+       - Lower connections = more similar groups
+       - The height shows how different two groups are
+    
+    2. **Cutting the Tree**:
+       - Draw a horizontal line to create clusters
+       - Where you cut determines the number of clusters
+       - We'll cut at a height that gives us 4 clusters (like K-means)
+    """)
+    
+    # Cut the tree to get clusters
+    hierarchical_labels = fcluster(linkage_matrix, k, criterion='maxclust') - 1
+    
+    # Compare K-means and Hierarchical Clustering
+    st.header("Comparing K-Means and Hierarchical Clustering")
+    
+    # Create side-by-side comparison
+    col1, col2 = st.columns(2)
+    
+    with col1:
+        st.subheader("K-Means Clustering")
+        fig_kmeans = px.scatter(df_clustered, 
+                              x='Murder', 
+                              y='Assault',
+                              color='Cluster',
+                              title='K-Means Clustering (K=4)',
+                              hover_data=['UrbanPop', 'Rape'])
+        st.plotly_chart(fig_kmeans)
+        
+        st.markdown("""
+        **K-Means Characteristics**:
+        - Requires specifying number of clusters upfront
+        - Creates clusters of similar size
+        - Works well with spherical clusters
+        - Faster for large datasets
+        - Can be sensitive to outliers
+        """)
+    
+    with col2:
+        st.subheader("Hierarchical Clustering")
+        df_hierarchical = df.copy()
+        df_hierarchical['Cluster'] = hierarchical_labels
+        fig_hierarchical = px.scatter(df_hierarchical, 
+                                    x='Murder', 
+                                    y='Assault',
+                                    color='Cluster',
+                                    title='Hierarchical Clustering (4 clusters)',
+                                    hover_data=['UrbanPop', 'Rape'])
+        st.plotly_chart(fig_hierarchical)
+        
+        st.markdown("""
+        **Hierarchical Clustering Characteristics**:
+        - Creates a complete hierarchy of clusters
+        - Can handle non-spherical clusters
+        - More flexible in cluster shapes
+        - Slower for large datasets
+        - Less sensitive to outliers
+        """)
     
-    # Display new features
-    st.subheader("New Features Analysis")
+    # Show agreement between methods
+    st.subheader("Comparing the Results")
     
-    # Family Size Analysis
-    fig_family = create_categorical_plot(df_cleaned, 'FamilySize')
-    st.plotly_chart(fig_family)
+    # Create comparison dataframe
+    comparison_df = pd.DataFrame({
+        'State': df.index,
+        'K-Means Cluster': cluster_labels,
+        'Hierarchical Cluster': hierarchical_labels
+    })
     
-    # Age Group Analysis
-    fig_age = create_categorical_plot(df_cleaned, 'AgeGroup')
-    st.plotly_chart(fig_age)
+    # Count agreements
+    agreements = sum(comparison_df['K-Means Cluster'] == comparison_df['Hierarchical Cluster'])
+    agreement_percentage = (agreements / len(comparison_df)) * 100
+    
+    st.write(f"Methods agreed on {agreements} out of {len(comparison_df)} states ({agreement_percentage:.1f}%)")
+    
+    # Show states where methods disagree
+    disagreements = comparison_df[comparison_df['K-Means Cluster'] != comparison_df['Hierarchical Cluster']]
+    if not disagreements.empty:
+        st.write("States where the methods disagreed:")
+        st.dataframe(disagreements)
+    
+    st.markdown("""
+    ### When to Use Each Method
+    
+    1. **Use K-Means when**:
+       - You know the number of clusters
+       - Your data has spherical clusters
+       - You need fast computation
+       - You want clusters of similar size
+    
+    2. **Use Hierarchical Clustering when**:
+       - You don't know the number of clusters
+       - You want to explore the hierarchy
+       - Your clusters might be non-spherical
+       - You need to handle outliers carefully
+    
+    In our case, both methods found similar patterns, suggesting our clusters are robust!
+    """)
+    
+    # Exercise 6: Policy Brief
+    st.header("Exercise 6: Policy Brief Creation")
+    
+    # Code Example: Creating Final Visualizations
+    with st.expander("Code Example: Creating Policy Brief Visualizations"):
+        st.code("""
+# Create a comprehensive visualization
+import plotly.graph_objects as go
+from plotly.subplots import make_subplots
+
+# Create subplots
+fig = make_subplots(rows=2, cols=2)
+
+# Plot 1: Murder vs Assault by cluster
+for i in range(k):
+    cluster_data = df_clustered[df_clustered['Cluster'] == i]
+    fig.add_trace(
+        go.Scatter(
+            x=cluster_data['Murder'],
+            y=cluster_data['Assault'],
+            mode='markers',
+            name=f'Cluster {i}'
+        ),
+        row=1, col=1
+    )
+
+# Plot 2: Urban Population vs Rape by cluster
+for i in range(k):
+    cluster_data = df_clustered[df_clustered['Cluster'] == i]
+    fig.add_trace(
+        go.Scatter(
+            x=cluster_data['UrbanPop'],
+            y=cluster_data['Rape'],
+            mode='markers',
+            name=f'Cluster {i}'
+        ),
+        row=1, col=2
+    )
+
+# Update layout
+fig.update_layout(
+    title_text="State Crime Profile Analysis",
+    showlegend=True
+)
+fig.show()
+        """, language="python")
     
-    # Conclusion
-    st.header("Conclusion")
     st.write("""
-    Through this analysis, we've learned:
-    - How to handle missing values in real-world datasets
-    - Techniques for analyzing categorical variables
-    - Methods for creating meaningful visualizations
-    - Feature engineering approaches
-    - Best practices for data cleaning and EDA
+    Based on our analysis, here's a summary of findings and recommendations:
+    
+    **Key Findings:**
+    - We identified distinct crime profiles among US states
+    - Each cluster represents a unique pattern of crime rates and urban population
+    - Some states show surprising similarities despite geographic distance
+    
+    **Policy Recommendations:**
+    1. High-Priority States: Focus on states in high-crime clusters
+    2. Resource Allocation: Distribute federal crime prevention funds based on cluster profiles
+    3. Best Practice Sharing: Encourage states within the same cluster to share successful strategies
     """)
     
     # Additional Resources
     st.header("Additional Resources")
     st.write("""
-    - [Pandas Documentation](https://pandas.pydata.org/docs/)
-    - [Seaborn Documentation](https://seaborn.pydata.org/)
-    - [Plotly Documentation](https://plotly.com/python/)
-    - [Data Cleaning Best Practices](https://towardsdatascience.com/data-cleaning-steps-and-process-8ae2d0f5147)
-    - [Colab Notebook](https://colab.research.google.com/drive/1ScwSa8WBcOMCloXsTV5TPFoVrcPHXlW2#scrollTo=VDMRGRbSR0gc)
-    - [Overleaf Project](https://www.overleaf.com/project/68228f4ccb9d18d92c26ba13)
+    - [Scikit-learn Clustering Documentation](https://scikit-learn.org/stable/modules/clustering.html)
+    - [KNN Documentation](https://scikit-learn.org/stable/modules/neighbors.html)
     """)