Padakovec/futolm-cs-base-gguf

futolm Czech and Slovak models

Separate 29.37M-parameter Czech and Slovak language models for next-word prediction in FUTO Keyboard. All files are F16 FUTO KeyboardLM GGUF exports with the shared SentencePiece tokenizer embedded.

For typing, use the synth models: cs_synth.gguf (Czech) or sk_synth.gguf (Slovak). They continue the chat models on synthetic text messages and predict everyday messages best. The chat and base models are kept for comparison and further training.

Synth models

Model	Download	FUTO language metadata	Initialization	Training tokens in this run	Final step
Czech synth	cs_synth.gguf	cs	Czech chat	30,932,992	236
Slovak synth	sk_synth.gguf	sk	Slovak chat	32,505,856	248

About one million words of everyday text messages per language were generated with OpenAI gpt-6-luna (Codex CLI), with varied recipients, topics, registers and lengths: 108,431 unique Czech and 111,329 unique Slovak messages. Each generation call is one split unit, so held-out messages never come from a call that produced training text. The training mix is 25% synthetic messages, with the chat mix (subtitles, forum pages, FineWeb2) for the rest, about four passes over the synthetic training split. Peak LR 5e-5, 50 warmup steps, cosine decay to 10%.

Language	Data	Chat top-1 / top-3	Synth top-1 / top-3
Czech	held-out synthetic messages	13.8 / 22.8	20.6 / 32.6
Czech	chat probes	41.8 / 64.1	50.9 / 71.8
Czech	subtitles	23.0 / 33.6	22.0 / 32.4
Czech	FineWeb2	21.4 / 33.6	21.4 / 33.4
Slovak	held-out synthetic messages	14.8 / 24.2	24.8 / 36.0
Slovak	chat probes	42.3 / 64.5	53.6 / 70.5
Slovak	subtitles	24.0 / 35.2	24.2 / 34.6
Slovak	FineWeb2	20.2 / 31.8	20.6 / 32.0

Subtitle and FineWeb2 differences are within noise (about ±4 points with 500 samples). The probes and the synthetic messages were both written by OpenAI models, and most probe phrases (for example "Dobrou noc") also occur in the synthetic text, so the probe gain partly reflects shared phrasing; the held-out synthetic messages, subtitles, and FineWeb2 are the less biased comparisons.

Chat models

Model	Download	FUTO language metadata	Initialization	Training tokens in this run	Final step
Czech chat	cs_chat.gguf	cs	Czech base	1,000,341,504	7,632
Slovak chat	sk_chat.gguf	sk	Slovak base	1,000,341,504	7,632

The mix is OpenSubtitles v2024 dialogue, discussion-forum pages selected from FineWeb2, and FineWeb2 replay: Czech 50/15/35, Slovak 48/12/40. Peak LR 1.5e-4, 300 warmup steps, cosine decay to 10%.

Whole-word next-word prediction, model only (FUTO's dictionary not involved). Probes are 220 handwritten chat prefixes per language; subtitles and FineWeb2 are 500 held-out word positions (about ±4 points).

Language	Data	Base top-1 / top-3	Chat top-1 / top-3
Czech	chat probes	22.7 / 43.6	41.8 / 64.1
Czech	subtitles	16.6 / 24.0	23.0 / 33.6
Czech	FineWeb2	22.0 / 34.6	21.4 / 33.6
Slovak	chat probes	24.6 / 38.6	42.3 / 64.5
Slovak	subtitles	16.2 / 28.0	24.0 / 35.2
Slovak	FineWeb2	21.0 / 35.4	20.2 / 31.8

Examples (top three): Ahoj, jak se → máš, máte, vede; Dobrou → noc, chuť, zprávou; Ahoj, ako sa → máš, máte, darí.

The probes were written by an AI model and have not been reviewed by a native speaker. OpenSubtitles has no blanket licence; these weights are shared for personal and research use.

Base models

Model	Download	FUTO language metadata	Initialization	Training tokens in this run	Final step	Latest logged validation loss
Czech base	cs_base.gguf	cs	From scratch	3,000,238,080	22,890	2.7750 at step 22,499
Slovak base	sk_base.gguf	sk	Czech base weights	3,001,024,512 additional Slovak tokens	22,896	2.6691 at step 22,499

Download the file for your language and import it through FUTO Keyboard's model import flow. The models are separately selectable; each declares its own language.

Model and training

Property	Value
Architecture	Llama, 8 layers, hidden size 512, FFN size 1,024, 8 attention/KV heads
Parameters per model	29,368,832
Vocabulary	8,192-piece SentencePiece Unigram, shared Czech/Slovak tokenizer
Context capability	2,048 positions; trained on 512-token sequences
Czech data	Cleaned Czech FineWeb2 (ces_Latn)
Slovak data	Cleaned Slovak FineWeb2 (slk_Latn)
FineWeb2 revision	af9c13333eb981300149d5ca60a8e9d659b276b9
Export	GGUF v3, F16 matrices and F32 normalization weights
Size of each GGUF	59,313,504 bytes (59.3 MB)
FUTO features	base_v1 inverted_space

Slovak continued pretraining starts from the clean Czech base with the same tokenizer and architecture. It uses a fresh optimizer, schedule, and Slovak data position. The Slovak token count above excludes the Czech pretraining tokens inherited through initialization. Its peak learning rate was 1e-4, with 300 warmup steps and cosine decay to 1e-5.

Verification

All six files passed structural checks and loaded with FUTO Keyboard's native code pinned to commit 0246cbe2fb2f130a5dc475c508215760044dea20 (version 0.1.30). Each model's native inference was compared with its PyTorch export across 11 cases: eight next-word prompts and three embedding-mix cases. All top-1 predictions matched and cosine similarity was at least 0.999. Top-five token sets matched in every case except one embedding-mix case of cs_synth (4 of 5, a near-tie at fifth place); embedding mixing is not used by these models' declared features.

See the verification reports (cs_base, sk_base, cs_chat, sk_chat, cs_synth, sk_synth) and SHA256SUMS.

Limitations

Keyboard-task autocorrect fine-tuning has not been performed, and autocorrect features are not declared. Embedding-mix verification checks numerical compatibility, not correction quality. Phone import, latency, and typing quality have not yet been tested.