Spaces:
Running
Running
File size: 23,822 Bytes
21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa 21fd477 d3ff7fa |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 |
# Prompts moved from test_pdf_parser.py to make the agent self-contained
REVIEWER_SYSTEM_PROMPT = """You are a senior AI research expert and technology assessment consultant, specializing in evaluating the potential for scientific research work to be automated by current or near-future AI systems.
Your assessment should be:
1. Systematic and evidence-based using the 12-dimensional framework
2. Objective in analyzing current AI capability boundaries
3. Realistic in predicting technology development trends
4. Comprehensive in considering automation barriers and societal impacts
Maintain critical thinking and provide detailed justifications for each score. Your evaluation will influence research directions and resource allocation decisions."""
EVALUATION_PROMPT_TEMPLATE = """
# Systematic AI Automation Assessment Framework
Please conduct a comprehensive evaluation of the provided academic work using the following 12-dimensional framework. Your output should be organized into four sections: executive_summary, dimensions, scores, recommendations, and limitations_uncertainties.
IMPORTANT: Follow the exact JSON schema structure provided. The 'dimensions' section should contain detailed analysis objects with 'score' and 'analysis' fields. The 'scores' section should contain only the numerical scores as a flat object. Do not include dimension scores as top-level fields.
## Executive Summary
Please provide a concise 150-word summary of key findings and overall assessment.
## 12-Dimensional Evaluation
### 1. **Task Formalization** (Score: 0-4)
**What to Evaluate**: Whether the task has clear rules/mathematical objectives
**Score Anchors**:
- 0: Ill-defined
- 1: Partly formal
- 2: Mostly formal
- 3: Fully formal with minor caveats
- 4: Mathematically exact
**Analysis Required**: Examine the clarity of problem definition, mathematical formulation, and objective functions.
### 2. **Data & Resource Availability** (Score: 0-4)
**What to Evaluate**: Public data, simulators, tool chains availability
**Score Anchors**:
- 0: None
- 1: Sparse/private
- 2: Moderate
- 3: Rich
- 4: Abundant & public
**Analysis Required**: Assess the availability and quality of datasets, existing tools, and computational resources.
### 3. **Input-Output Complexity** (Score: 0-4)
**What to Evaluate**: Modal diversity, structure and length complexity
**Score Anchors**:
- 0: Chaotic
- 1: High complexity
- 2: Moderate complexity
- 3: Low complexity
- 4: Highly regular
**Analysis Required**: Evaluate the complexity of input processing and output generation requirements.
### 4. **Real-World Interaction** (Score: 0-4)
**What to Evaluate**: Need for physical/social/online feedback
**Score Anchors**:
- 0: Constant interaction needed
- 1: Frequent interaction
- 2: Occasional interaction
- 3: Rare interaction
- 4: None (offline)
**Analysis Required**: Determine the extent of real-world interaction and feedback requirements.
### 5. **Existing AI Coverage** (Score: 0-4)
**What to Evaluate**: Proportion of work already completed by existing AI models
**Score Anchors**:
- 0: < 5%
- 1: ≈ 25%
- 2: ≈ 50%
- 3: ≈ 75%
- 4: > 95%
**Analysis Required**: Identify specific existing AI tools/models and quantify coverage percentage.
### 6. **Automation Barriers** (Qualitative Analysis - No Score)
**What to Evaluate**: Major obstacles like creativity, common sense, legal issues
**Analysis Required**: List and explain key barriers preventing full automation:
- Creativity requirements
- Common sense reasoning
- Domain expertise
- Legal/ethical constraints
- Tacit knowledge
- Other specific barriers
### 7. **Human Originality/Irreplaceability** (Score: 0-4)
**What to Evaluate**: Dependence on human creativity and originality
**Score Anchors**:
- 0: Routine work
- 1: Incremental innovation
- 2: Moderately novel
- 3: Clearly novel
- 4: Paradigm-shifting
**Analysis Required**: Assess the level of human creativity, insight, and original thinking required.
### 8. **Safety & Ethical Criticality** (Score: 0-4, Reverse Scoring)
**What to Evaluate**: Consequences of failure/misuse
**Score Anchors**:
- 0: Catastrophic consequences
- 1: Serious consequences
- 2: Manageable consequences
- 3: Minor consequences
- 4: Negligible consequences
**Analysis Required**: Evaluate risks and potential negative impacts of automation.
### 9. **Societal/Economic Impact** (Qualitative Analysis - No Score)
**What to Evaluate**: Net impact after full automation
**Analysis Required**: Describe comprehensive societal and economic implications:
- Job displacement effects
- Research quality changes
- Innovation ecosystem impacts
- Economic benefits/costs
- Social implications
### 10. **Technical Maturity Needed** (Score: 0-4)
**What to Evaluate**: Required R&D depth for automation
**Score Anchors**:
- 0: Multiple breakthroughs needed
- 1: One major breakthrough needed
- 2: Cutting-edge R&D required
- 3: Incremental work needed
- 4: Already solved
**Analysis Required**: Identify specific technical advances needed and their feasibility.
### 11. **3-Year Feasibility** (Probability: 0-100%)
**What to Evaluate**: Probability of AI reaching expert level within 3 years
**Analysis Required**: Provide realistic probability estimate with detailed justification considering:
- Current AI development pace
- Required technical breakthroughs
- Resource availability
- Market incentives
### 12. **Overall Automatability** (Score: 0-4)
**What to Evaluate**: Comprehensive automation feasibility
**Score Anchors**:
- 0: Not automatable
- 1: Hard to automate
- 2: Moderately automatable
- 3: Highly automatable
- 4: Already automatable
**Analysis Required**: Synthesize all dimensions into overall assessment.
## Recommendations
### For Researchers
Please provide specific recommendations for researchers in this field.
### For Institutions
Please provide recommendations for research institutions and funding bodies.
### For AI Development
Please provide recommendations for AI researchers and developers.
## Assessment Limitations and Uncertainties
Please list any limitations or uncertainties in your assessment.
---
**Instructions**:
- Provide specific evidence and examples for each score
- Be conservative in scoring when uncertain
- Consider both current capabilities and realistic near-term developments
- Justify all numerical scores with detailed reasoning
- For qualitative dimensions, provide comprehensive analysis
- Please use `return_assessment` tool to return the complete AI automation assessment as a single JSON object.
- Do not mention the tool in your response in order to avoid model hallucination.
Now please begin the systematic evaluation of the provided academic work.
"""
# Tools schema for function calling (Anthropic tools)
# The model must call `return_assessment` to output a strict JSON object
TOOLS = [
{
"name": "return_assessment",
"description": "Return the complete AI automation assessment as a single JSON object.",
"input_schema": {
"type": "object",
"properties": {
"executive_summary": {
"type": "string",
"description": "A concise 150-word summary of key findings and overall assessment."
},
"dimensions": {
"type": "object",
"description": "Detailed analysis of each dimension with scores and justifications.",
"properties": {
"task_formalization": {
"type": "object",
"properties": {
"score": {
"type": "number",
"description": "The score for the task formalization dimension, on a scale of 0-4."
},
"analysis": {
"type": "string",
"description": "A detailed analysis of the task formalization dimension, including the score and the justification for the score."
}
},
"required": [
"score",
"analysis"
]
},
"data_resource_availability": {
"type": "object",
"properties": {
"score": {
"type": "number",
"description": "The score for the data resource availability dimension, on a scale of 0-4."
},
"analysis": {
"type": "string",
"description": "A detailed analysis of the data resource availability dimension, including the score and the justification for the score."
}
},
"required": [
"score",
"analysis"
]
},
"input_output_complexity": {
"type": "object",
"properties": {
"score": {
"type": "number",
"description": "The score for the input output complexity dimension, on a scale of 0-4."
},
"analysis": {
"type": "string",
"description": "A detailed analysis of the input output complexity dimension, including the score and the justification for the score."
}
},
"required": [
"score",
"analysis"
]
},
"real_world_interaction": {
"type": "object",
"properties": {
"score": {
"type": "number",
"description": "The score for the real world interaction dimension, on a scale of 0-4."
},
"analysis": {
"type": "string",
"description": "A detailed analysis of the real world interaction dimension, including the score and the justification for the score."
}
},
"required": [
"score",
"analysis"
]
},
"existing_ai_coverage": {
"type": "object",
"properties": {
"score": {
"type": "number",
"description": "The score for the existing AI coverage dimension, on a scale of 0-4."
},
"analysis": {
"type": "string",
"description": "A detailed analysis of the existing AI coverage dimension, including the score and the justification for the score."
},
"tools_models": {
"type": "array",
"items": {
"type": "string"
}
},
"coverage_pct_estimate": {
"type": "number"
}
},
"required": [
"score",
"analysis"
]
},
"automation_barriers": {
"type": "object",
"properties": {
"analysis": {
"type": "string",
"description": "A detailed analysis of the automation barriers dimension, including the score and the justification for the score."
}
},
"required": [
"analysis"
]
},
"human_originality": {
"type": "object",
"properties": {
"score": {
"type": "number",
"description": "The score for the human originality dimension, on a scale of 0-4."
},
"analysis": {
"type": "string",
"description": "A detailed analysis of the human originality dimension, including the score and the justification for the score."
}
},
"required": [
"score",
"analysis"
]
},
"safety_ethics": {
"type": "object",
"properties": {
"score": {
"type": "number",
"description": "The score for the safety and ethics dimension, on a scale of 0-4."
},
"analysis": {
"type": "string",
"description": "A detailed analysis of the safety and ethics dimension, including the score and the justification for the score."
}
},
"required": [
"score",
"analysis"
]
},
"societal_economic_impact": {
"type": "object",
"properties": {
"analysis": {
"type": "string"
}
},
"required": [
"analysis"
]
},
"technical_maturity_needed": {
"type": "object",
"properties": {
"score": {
"type": "number"
},
"analysis": {
"type": "string"
}
},
"required": [
"score",
"analysis"
]
},
"three_year_feasibility": {
"type": "object",
"properties": {
"probability_pct": {
"type": "number",
"description": "The probability of AI reaching expert level within 3 years, on a scale of 0-100%."
},
"analysis": {
"type": "string",
"description": "A detailed analysis of the three year feasibility dimension, including the probability and the justification for the probability."
}
},
"required": [
"probability_pct",
"analysis"
]
},
"overall_automatability": {
"type": "object",
"properties": {
"score": {
"type": "number",
"description": "The score for the overall automatability dimension, on a scale of 0-4."
},
"analysis": {
"type": "string",
"description": "A detailed analysis of the overall automatability dimension, including the score and the justification for the score."
}
},
"required": [
"score",
"analysis"
]
}
},
"required": [
"task_formalization",
"data_resource_availability",
"input_output_complexity",
"real_world_interaction",
"existing_ai_coverage",
"automation_barriers",
"human_originality",
"safety_ethics",
"societal_economic_impact",
"technical_maturity_needed",
"three_year_feasibility",
"overall_automatability"
]
},
"scores": {
"type": "object",
"properties": {
"task_formalization": {
"type": "number",
"description": "The score for the task formalization dimension, on a scale of 0-4."
},
"data_resource_availability": {
"type": "number",
"description": "The score for the data resource availability dimension, on a scale of 0-4."
},
"input_output_complexity": {
"type": "number",
"description": "The score for the input output complexity dimension, on a scale of 0-4."
},
"real_world_interaction": {
"type": "number",
"description": "The score for the real world interaction dimension, on a scale of 0-4."
},
"existing_ai_coverage": {
"type": "number",
"description": "The score for the existing AI coverage dimension, on a scale of 0-4."
},
"human_originality": {
"type": "number",
"description": "The score for the human originality dimension, on a scale of 0-4."
},
"safety_ethics": {
"type": "number",
"description": "The score for the safety and ethics dimension, on a scale of 0-4."
},
"technical_maturity_needed": {
"type": "number",
"description": "The score for the technical maturity needed dimension, on a scale of 0-4."
},
"three_year_feasibility_pct": {
"type": "number",
"description": "The probability of AI reaching expert level within 3 years, on a scale of 0-100%."
},
"overall_automatability": {
"type": "number",
"description": "The score for the overall automatability dimension, on a scale of 0-4."
}
},
"required": [
"task_formalization",
"data_resource_availability",
"input_output_complexity",
"real_world_interaction",
"existing_ai_coverage",
"human_originality",
"safety_ethics",
"technical_maturity_needed",
"three_year_feasibility_pct",
"overall_automatability"
]
},
"recommendations": {
"type": "object",
"properties": {
"for_researchers": {
"type": "array",
"items": {
"type": "string",
"description": "A specific recommendation for researchers in this field."
}
},
"for_institutions": {
"type": "array",
"items": {
"type": "string",
"description": "A recommendation for research institutions and funding bodies."
}
},
"for_ai_development": {
"type": "array",
"items": {
"type": "string",
"description": "A recommendation for AI researchers and developers."
}
}
},
"required": [
"for_researchers",
"for_institutions",
"for_ai_development"
]
},
"limitations_uncertainties": {
"type": "array",
"items": {
"type": "string",
"description": "A limitation or uncertainty in the assessment."
}
}
},
"required": [
"executive_summary",
"dimensions",
"scores",
"recommendations",
"limitations_uncertainties"
],
"additionalProperties": False,
"description": "Complete evaluation output with executive summary, detailed dimensions analysis, numerical scores, recommendations, and limitations."
}
}
]
TOOL_CHOICE = {
"type": "tool",
"name": "return_assessment"
} |