more docs

docs/source/competition_space.mdx

@@ -1 +1,38 @@
-# Competition Space
+# Competition Space
+
+A competition space is a Hugging Face Space where the actual competition takes place. It is a space where you can submit your model and get a score. It is also a space where competitors can see the leaderboard, discuss, and make submissions.
+
+Check out an example competition space below:
+
+![competition space](https://github.com/abhishekkrthakur/public_images/blob/main/competition_space.png?raw=true)
+
+The competition space consists of the following:
+
+- Competition description (content is fetched from private competition repo)
+- Dataset description (content is fetched from private competition repo)
+- Leaderboard (content is fetched from private competition repo)
+    - Public (available to everyone, all the time)
+    - Private (available to everyone, but only after the competition ends)
+- Submissions
+    - Submission guidelines (content is fetched from private competition repo)
+    - My submissions (users can see their own submissions)
+    - New submission (users can make new submissions)
+- Discussions (accessible via community tab)
+
+### Secrets
+
+The competition space requires two secrets:
+
+- `HF_TOKEN`: this is the Hugging Face write token of the user who created the competition space. This token must be kept alive for the duration of the competition. In case the token expires, the competition space will stop working. If you change/refresh/delete this token, you will need to update this secret.
+- `COMPETITION_ID`: this is the path of private competition repo. e.g. `abhishek/private-competition-data`. If you change the name of the private competition repo, you will need to update this secret.
+
+Note: The above two secrets are crucial for the competition space to work!
+
+### Public & private competition spaces
+
+A competition space can be public or private. A public competition space is available to everyone, all the time. 
+A private competition space is only available to the members of the organization the competition space is created in.
+
+You can at any point make the competition public.
+
+Generally, we recommend testing every aspect of the competition space in a private competition space before making it public.

docs/source/custom_metric.mdx

@@ -0,0 +1,138 @@
+# Custom metric
+
+In case you don't settle for the default scikit-learn metrics, you can define your own metric.
+
+Here, we expect the organizer to know python. 
+
+### How to define a custom metric
+
+To define a custom metric, change `EVAL_METRIC` in `conf.json` to `custom`. You must also make sure that `EVAL_HIGHER_IS_BETTER` is set to `1` or `0` depending on whether a higher value of the metric is better or not.
+
+The second step is to create a file `metric.py` in the private competition repo.
+The file should contain a `compute` function that takes competition params as input.
+
+Here is the part where we check if metric is custom and calculate the metric value:
+
+```python
+def compute_metrics(params):
+    if params.metric == "custom":
+        metric_file = hf_hub_download(
+            repo_id=params.competition_id,
+            filename="metric.py",
+            token=params.token,
+            repo_type="dataset",
+        )
+        sys.path.append(os.path.dirname(metric_file))
+        metric = importlib.import_module("metric")
+        evaluation = metric.compute(params)
+    .
+    .
+    .
+````
+
+You can find the above part in competitions github repo `compute_metrics.py`
+
+`params` is defined as:
+
+```python
+class EvalParams(BaseModel):
+    competition_id: str
+    competition_type: str
+    metric: str
+    token: str
+    team_id: str
+    submission_id: str
+    submission_id_col: str
+    submission_cols: List[str]
+    submission_rows: int
+    output_path: str
+    submission_repo: str
+    time_limit: int
+```
+
+You are free to do whatever you want to in the `compute` function. 
+In the end it must return a dictionary with the following keys:
+
+```python
+{
+    "public_score": metric_value,
+    "private_score": metric_value,
+}
+```
+
+public and private scores can be floats or can also be dictionaries in case you want to use multiple metrics.
+Example for multiple metrics:
+
+```python
+{
+    "public_score": {
+        "metric1": metric_value,
+        "metric2": metric_value,
+    },
+    "private_score": {
+        "metric1": metric_value,
+        "metric2": metric_value,
+    },
+}
+```
+
+Note: When using multiple metrics, a base metric (the first one) will be used to rank the participants in the competition.
+
+### Example of a custom metric
+
+```python
+import pandas as pd
+from huggingface_hub import hf_hub_download
+
+
+def compute(params):
+    solution_file = hf_hub_download(
+        repo_id=params.competition_id,
+        filename="solution.csv",
+        token=params.token,
+        repo_type="dataset",
+    )
+
+    solution_df = pd.read_csv(solution_file)
+
+    submission_filename = f"submissions/{params.team_id}-{params.submission_id}.csv"
+    submission_file = hf_hub_download(
+        repo_id=params.competition_id,
+        filename=submission_filename,
+        token=params.token,
+        repo_type="dataset",
+    )
+    submission_df = pd.read_csv(submission_file)
+
+    public_ids = solution_df[solution_df.split == "public"][params.submission_id_col].values
+    private_ids = solution_df[solution_df.split == "private"][params.submission_id_col].values
+
+    public_solution_df = solution_df[solution_df[params.submission_id_col].isin(public_ids)]
+    public_submission_df = submission_df[submission_df[params.submission_id_col].isin(public_ids)]
+
+    private_solution_df = solution_df[solution_df[params.submission_id_col].isin(private_ids)]
+    private_submission_df = submission_df[submission_df[params.submission_id_col].isin(private_ids)]
+
+    public_solution_df = public_solution_df.sort_values(params.submission_id_col).reset_index(drop=True)
+    public_submission_df = public_submission_df.sort_values(params.submission_id_col).reset_index(drop=True)
+
+    private_solution_df = private_solution_df.sort_values(params.submission_id_col).reset_index(drop=True)
+    private_submission_df = private_submission_df.sort_values(params.submission_id_col).reset_index(drop=True)
+
+    # CALCULATE METRICS HERE.......
+    # _metric = SOME METRIC FUNCTION
+    target_cols = [col for col in solution_df.columns if col not in [params.submission_id_col, "split"]]
+    public_score = _metric(public_solution_df[target_cols], public_submission_df[target_cols])
+    private_score = _metric(private_solution_df[target_cols], private_submission_df[target_cols])
+
+    evaluation = {
+        "public_score": public_score,
+        "private_score": private_score,
+    }
+    return evaluation
+```
+
+Take a careful look at the above code. 
+You can see that we are downloading the solution file and the submission file from the dataset repo.
+We are then calculating the metric on the public and private splits of the solution and submission files. 
+Finally, we are returning the metric values in a dictionary.

docs/source/leaderboard.mdx

@@ -0,0 +1 @@
+# Understanding the leaderboard

docs/source/pricing.mdx

@@ -2,7 +2,7 @@
 
 Creating a competition is free. However, you will need to pay for the compute resources used to run the competition. The cost of the compute resources depends the type of competition you create.
 
-- generic: generic competitions are free to run. you can, however, upgrade the compute to cpu-basic to speed up the metric calculation and reduce the waiting time for the participants.
+- generic: generic competitions are free to run. you can, however, upgrade the compute to cpu-upgrade to speed up the metric calculation and reduce the waiting time for the participants.
 
 - script: script competitions require a compute resource to run the participant's code. you can choose between a variety of cpu and gpu instances (T4, A10g and even A100). the cost of the compute resource is charged per hour.
 

docs/source/submit.mdx

@@ -0,0 +1 @@
+# Making a submission

docs/source/teams.mdx

@@ -0,0 +1,3 @@
+# Teaming up
+
+Coming soon!