Merge branch 'main' of hf.co:spaces/double-ai/FormulaOne-Leaderboard

README.md

@@ -13,35 +13,28 @@ sdk_version: 5.42.0
 python_version: 3.12.0
 ---
 
-# Start the configuration
 
-Most of the variables to change for a default leaderboard are in `src/env.py` and `src/about.py`.
+### Competition terms
 
-Results files should have the following format and be stored as json files:
+- By submitting, you agree to the **FormulaOne Submission Agreement
+(v1.2)** and our **Privacy Notice**.
+- Your uploaded file remains yours; we only use it to evaluate, score,
+and contact you about your result.
+**Licensing for the public benchmark assets (informational)**
+- **Evaluator code:** Apache License 2.0
+- **Problem statements & public tests:** Creative Commons **CC BY 4.0**
+See the project's **README licence section** and full texts: `LICENSE-
+APACHE2`, `LICENSE-CC-BY` in our GitHub repo.
+**Platform**
+- Your use of Hugging Face is also governed by Hugging Face's Terms and
+Privacy Policy.
+- 
 
-```json
-{
-    "config": {
-        "model_dtype": "torch.float16", # or torch.bfloat16 or 8bit or 4bit
-        "model_name": "path of the model on the hub: org/model",
-        "model_sha": "revision on the hub",
-    },
-    "results": {
-        "task_name": {
-            "metric_name": score,
-        },
-        "task_name2": {
-            "metric_name": score,
-        }
-    }
-}
-```
+# FormulaOne: Measuring the Depth of Algorithmic Reasoning Beyond Competitive Programming
 
-Request files are created automatically by this tool.
+This is the official Hugging Face space for the paper:
 
-# Code logic for more complex edits
+**FormulaOne: Measuring the Depth of Algorithmic Reasoning Beyond Competitive Programming** <br>
+*Gal Beniamini, Yuval Dor, Alon Vinnikov, Shir Granot Peled, Or Weinstein, Or Sharir, Noam Wies, Tomer Nussbaum, Nadav Schweiger, Ido Ben Shaul, Tomer Zekharya, Yoav Levine, Shai Shalev-Shwartz, Amnon Shashua* <br>
+**AAI, July 2025**
 
-You'll find 
-- The main table columns names and properties in `src/display/utils.py`
-- The logic to read all results and request files, then convert them in dataframe lines, in `src/populate.py`
-- The logic to allow or filter submissions in `src/submission/submit.py`.

SECURITY.md

@@ -0,0 +1,4 @@
+To report a vulnerability or security concern, email
+[email protected].
+Please include steps to reproduce and any relevant logs. We will
+acknowledge receipt within 3 business days.

TRADEMARKS.md

@@ -0,0 +1,5 @@
+"FormulaOne" is used solely to identify this research benchmark.
+You may use the name to refer to the benchmark, but you may not use it:
+- to imply sponsorship or endorsement of your project or results, or
+- as part of your own product or service name without written
+permission.

app.py

@@ -512,6 +512,8 @@ with blocks:
             with gr.Row():
                 gr.Markdown("# ✉️✨ Submit your solutions", elem_classes="markdown-text")
 
+            gr.Markdown(SUBMISSION_TERMS_TEXT, elem_classes="markdown-text")
+
             login_box = gr.Group(visible=True, elem_id="f1-login-box")
             with login_box:
                 gr.Markdown("Please sign in with Hugging Face to submit")
@@ -521,16 +523,47 @@ with blocks:
             with submit_panel:
                 with gr.Row():
                     with gr.Column():
-                        gr.Markdown(SUBMISSION_TERMS_TEXT, elem_classes="markdown-text")
-
                         system_name_textbox = gr.Textbox(label=AutoEvalColumn.system.name)
                         org_textbox = gr.Textbox(label=AutoEvalColumn.organization.name)
                         submission_file = gr.File(label="JSONL solutions file", file_types=[".jsonl"])
 
+                        # Required checkboxes
+                        agreement_checkbox = gr.Checkbox(
+                            label="I agree to the FormulaOne Submission Agreement (v1.2).",
+                            value=False,
+                            elem_classes="markdown-text",
+                        )
+
+                        privacy_checkbox = gr.Checkbox(
+                            label="I have read the Privacy Notice.", value=False, elem_classes="markdown-text"
+                        )
+
+                        security_checkbox = gr.Checkbox(
+                            label="I confirm this submission does not attempt to access private tests or exfiltrate data.",
+                            value=False,
+                            elem_classes="markdown-text",
+                        )
+
+                        gr.Markdown(
+                            "[Privacy Notice](docs/privacy-policy.md); [Submission Agreement](terms/submission-agreement.md)",
+                            elem_classes="markdown-text",
+                        )
+
                         logger.info("Submit button")
-                        submit_button = gr.Button("Submit", variant="primary")
+                        submit_button = gr.Button("Submit", variant="primary", interactive=False)
                         submission_result = gr.Markdown()
 
+                # Update submit button interactivity based on checkboxes
+                def update_submit_button(agreement, privacy, security):
+                    return gr.update(interactive=agreement and privacy and security)
+
+                for checkbox in [agreement_checkbox, privacy_checkbox, security_checkbox]:
+                    checkbox.change(
+                        update_submit_button,
+                        inputs=[agreement_checkbox, privacy_checkbox, security_checkbox],
+                        outputs=submit_button,
+                    )
+
                 submit_button.click(
                     add_solution_cbk,
                     [

docs/privacy-policy.md

@@ -0,0 +1,56 @@
+FormulaOne AI Coding Challenge - Privacy Notice v1.2
+Last updated: 6 Aug 2025
+1 Controller
+FormulaOne Team, <legal entity and address>
+Contact: [email protected]
+2 Data We Collect
+- GitHub or Hugging Face username and public profile ID
+- Email address (if supplied via OAuth)
+- Uploaded Submission files, compile/runtime logs, numeric scores
+- IP address and browser telemetry for security and rate limiting
+3 Purposes and GDPR Legal Bases
+Purpose | Basis
+---------------------------------------------- | -------------------
+---
+Evaluate and rank submissions | Contract Art
+6(1)(b)
+Site security and abuse prevention | Legitimate
+interests 6(1)(f)
+Academic, non-commercial research on models | Legitimate
+interests 6(1)(f)
+Contacting participants about results/collab | Legitimate
+interests 6(1)(f)
+
+4 Public Disclosure
+Your display name, score and rank appear on a public leaderboard.
+You may use a pseudonym.
+5 Data Sharing
+Processors: Hugging Face Inc., GitHub Inc., <cloud host> - under
+written contracts.
+Independent controllers: GitHub and Hugging Face regarding your
+platform accounts.
+6 International Transfers
+Where data moves outside the EEA/UK, we rely on EU Standard
+Contractual Clauses or adequacy decisions.
+7 Retention
+- Leaderboard entries: indefinitely.
+- Raw submission files and logs: up to 5 years, then deleted or
+anonymised.
+- Security logs: 12 months.
+8 Your Rights
+Access, correction, erasure, restriction, portability, and
+objection. Contact us at the address above.
+You may lodge a complaint with your supervisory authority.
+9 Security
+Submissions run in network-restricted containers. Access is limited
+to authorised staff using multi-factor authentication.
+10 Cookies
+If you use only GitHub and Hugging Face, their cookies apply under
+their policies. If you later self-host a site for the leaderboard,
+publish your own cookie notice.
+11 Children
+The Competition is not directed to children under 16. Do not submit
+personal data if you are under 16.
+12 Changes
+We will post updates here and notify registered participants by
+email when material.

src/about.py

@@ -108,7 +108,7 @@ WHAT_IS_F1_HTML_AFTER_TIER1FIG_TAIL = """
 
 
 SUBMISSION_TERMS_TEXT = """
-**Competition terms**
+### Competition terms
 - By submitting, you agree to the **FormulaOne Submission Agreement
 (v1.2)** and our **Privacy Notice**.
 - Your uploaded file remains yours; we only use it to evaluate, score,

src/display/css_html_js.py

@@ -143,6 +143,43 @@ custom_css = """
   max-width: 800px !important; width: 100% !important; margin-left: auto !important; margin-right: auto !important;
 }
 #hf-login-btn:hover, #hf-login-btn button:hover, button[data-testid="login-button"]:hover, [data-testid="login-button"] button:hover, div[data-testid="login-button"] > button:hover { background: #f9fafb !important; }
+
+/* Checkbox styling for submission requirements */
+.markdown-text input[type="checkbox"] {
+  margin-right: 8px !important;
+  width: 16px !important;
+  height: 16px !important;
+  accent-color: #2563eb !important;
+  cursor: pointer !important;
+}
+
+.markdown-text label[data-testid="checkbox"] {
+  color: var(--f1-text) !important;
+  font-size: 14px !important;
+  line-height: 1.5 !important;
+  cursor: pointer !important;
+  display: flex !important;
+  align-items: flex-start !important;
+  margin-bottom: 8px !important;
+}
+
+/* Submit button styling when disabled */
+button[data-testid="button"]:disabled {
+  opacity: 0.5 !important;
+  cursor: not-allowed !important;
+  background: #f3f4f6 !important;
+  color: #9ca3af !important;
+  border-color: #d1d5db !important;
+}
+
+/* Required Agreements section 
+.markdown-text h3 {
+  color: var(--f1-text) !important;
+  font-weight: 600 !important;
+  margin: 20px 0 12px 0 !important;
+  font-size: 18px !important;
+}
+*/
 """
 
 get_window_url_params = """

terms/submission-agreement.md

@@ -0,0 +1,66 @@
+FormulaOne AI Coding Challenge - Submission Agreement v1.2
+Last updated: 6 Aug 2025
+1 Definitions
+"Submission" means the code file you upload for a listed problem.
+"Organiser" means the FormulaOne Team, c/o <legal entity and address>.
+
+"Competition Site" means the leaderboard hosted at <HF URL>.
+2 Licence to Organiser
+You retain all IP in your Submission. You grant the Organiser and its
+academic partners a worldwide, royalty-free, non-exclusive licence to:
+a) copy, run, test and modify the Submission solely to evaluate it
+against public and private test cases;
+b) store the Submission on secure servers;
+c) manually review the Submission for non-commercial, scientific
+research;
+d) reproduce small code excerpts (<= 25 lines) in research papers or
+blogs;
+e) contact you via the email linked to your platform account to ask
+clarification questions, explore collaboration, or publicise notable
+results.
+3 Security, Integrity and Acceptable Use
+- You confirm the Submission contains no deliberately malicious code,
+back-doors, attempts to exfiltrate data, or calls to external network
+resources beyond what the submission guidelines permit.
+- You will not attempt to discover, reconstruct, scrape, or share
+private test cases, nor to probe the sandbox environment.
+- The Organiser may refuse, delete, re-run or disqualify any
+Submission at its sole discretion for suspected abuse, tampering or
+rule violations.
+4 Competition Rules
+- One account per team.
+- You must own or have rights to your Submission.
+- We may re-execute Submissions to confirm scores; ties may be broken
+by earliest valid submission or additional hidden tests.
+- We may correct or withdraw scores if errors or rule breaches are
+discovered later.
+5 Leaderboard and Publicity
+Your chosen display name, organisation (if provided), total score and
+ranking may be shown publicly and archived indefinitely. You may use a
+pseudonym.
+6 Ownership of Benchmark Assets
+All evaluation code, problem statements and test data remain (c) 2025
+FormulaOne Team and are licensed under Apache 2.0 (code) and CC BY 4.0
+(content) as described in the public repository.
+7 Export Control and Sanctions Compliance
+You represent that your participation is not prohibited by applicable
+export-control or sanctions laws and that you are not located in, under
+the control of, or a national or resident of any country or person
+embargoed by relevant authorities.
+
+8 No Warranty
+The benchmark and infrastructure are provided "as is." To the maximum
+extent permitted by law, the Organiser disclaims all warranties,
+express or implied.
+9 Limitation of Liability
+The Organiser's total liability arising out of or in connection with
+the Competition will not exceed USD 100.
+10 Changes to Rules
+We may update these terms and technical rules from time to time.
+Material changes will be posted on the Competition Site and apply to
+submissions made after the effective date.
+11 Governing Law and Venue
+This Agreement is governed by the laws of England and Wales. The courts
+of London, UK, have exclusive jurisdiction.
+12 Contact
+Questions about these terms: [email protected]