Update README.md (#1)
Browse files- Update README.md (98e2e344d7fd5c819137899d4e840c8d29a9f670)
Co-authored-by: A J <[email protected]>
README.md
CHANGED
|
@@ -26,7 +26,7 @@ widget:
|
|
| 26 |
|
| 27 |
## Model Summary
|
| 28 |
|
| 29 |
-
The JavaCoder models are
|
| 30 |
|
| 31 |
- **Repository:**
|
| 32 |
- **Project Website:**
|
|
@@ -88,7 +88,7 @@ The model has been trained on source code from 80+ programming languages. The pr
|
|
| 88 |
## Hardware
|
| 89 |
|
| 90 |
- **GPUs:** 6 NVIDIA A100 80GB
|
| 91 |
-
- **Training time:** days
|
| 92 |
|
| 93 |
## Software
|
| 94 |
|
|
|
|
| 26 |
|
| 27 |
## Model Summary
|
| 28 |
|
| 29 |
+
The JavaCoder models are 1B parameter models trained on 80+ programming languages from [The Stack (v1.2)](https://huggingface.co/datasets/bigcode/the-stack), with opt-out requests excluded. The model uses [Multi Query Attention](https://arxiv.org/abs/1911.02150), [a context window of 8192 tokens](https://arxiv.org/abs/2205.14135), and was trained using the [Fill-in-the-Middle objective](https://arxiv.org/abs/2207.14255) on 1 trillion tokens.
|
| 30 |
|
| 31 |
- **Repository:**
|
| 32 |
- **Project Website:**
|
|
|
|
| 88 |
## Hardware
|
| 89 |
|
| 90 |
- **GPUs:** 6 NVIDIA A100 80GB
|
| 91 |
+
- **Training time:** 4 days
|
| 92 |
|
| 93 |
## Software
|
| 94 |
|