update

README.md

@@ -1,4 +1,65 @@
+# Fine-tuning Gemma 3n E4B on MacBook M1 (Apple Silicon) with Unsloth
+
+This project supports local fine-tuning of the Gemma 3n E4B model using Unsloth, PEFT/LoRA, and export to GGUF Q4_K_XL for efficient inference. The workflow is optimized for Apple Silicon (M1/M2/M3) and avoids CUDA/bitsandbytes dependencies.
+
+## Prerequisites
+
+- Python 3.10+
+- macOS with Apple Silicon (M1/M2/M3)
+- PyTorch with MPS backend (install via `pip install torch`)
+- All dependencies in `requirements.txt` (install with `pip install -r requirements.txt`)
+
+## Training Script Usage
+
+Run the training script with your dataset (JSON/JSONL or Hugging Face format):
+
+```bash
+python training/train_gemma_unsloth.py \
+  --job-id myjob \
+  --output-dir training_runs/myjob \
+  --dataset sample_data/train.jsonl \
+  --prompt-field prompt --response-field response \
+  --epochs 1 --batch-size 1 --gradient-accumulation 8 \
+  --use-fp16 \
+  --grpo --cpt \
+  --export-gguf --gguf-out training_runs/myjob/adapter-gguf-q4_k_xl
+```
+
+**Flags:**
+
+- `--grpo`: Enable GRPO (if supported by Unsloth)
+- `--cpt`: Enable CPT (if supported by Unsloth)
+- `--export-gguf`: Export to GGUF Q4_K_XL after training
+- `--gguf-out`: Path to save GGUF file
+
+**Notes:**
+
+- On Mac, bitsandbytes/xformers are disabled automatically.
+- Training is slower than on CUDA GPUs; use small batch sizes and gradient accumulation.
+- If Unsloth's GGUF export is unavailable, follow the printed instructions to use llama.cpp's `convert-hf-to-gguf.py`.
+
+## Troubleshooting
+
+- If you see errors about missing CUDA or bitsandbytes, ensure you are running on Apple Silicon and have the latest Unsloth/Transformers.
+- For memory errors, reduce `--batch-size` or `--cutoff-len`.
+- For best results, use datasets formatted to match the official Gemma 3n chat template.
+
+## Example: Manual GGUF Export with llama.cpp
+
+If the script prints a message about manual conversion, run:
+
+```bash
+python convert-hf-to-gguf.py --outtype q4_k_xl --outfile training_runs/myjob/adapter-gguf-q4_k_xl training_runs/myjob/adapter
+```
+
+## References
+
+- [Unsloth Documentation](https://unsloth.ai/)
+- [Gemma 3n E4B Model Card](https://huggingface.co/unsloth/gemma-3n-E4B-it)
+- [llama.cpp GGUF Export Guide](https://github.com/ggerganov/llama.cpp)
+
 ---
+
 title: Multimodal AI Backend Service
 emoji: 🚀
 colorFrom: yellow
@@ -6,6 +67,7 @@ colorTo: purple
 sdk: docker
 app_port: 8000
 pinned: false
+
 ---
 
 # firstAI - Multimodal AI Backend 🚀

README_DEPLOY_HF.md

@@ -0,0 +1,68 @@
+# Hugging Face Inference Endpoint: Gemma-3n-E4B-it LoRA Adapter
+
+This repository provides a LoRA adapter fine-tuned on top of a Hugging Face Transformers model (e.g., Gemma-3n-E4B-it) using PEFT. It is ready to be deployed as a Hugging Face Inference Endpoint.
+
+## How to Deploy as an Endpoint
+
+1. **Upload the `adapter` directory (produced by training) to your Hugging Face Hub repository.**
+
+   - The directory should contain `adapter_config.json`, `adapter_model.bin`, and tokenizer files.
+
+2. **Add a `handler.py` file to define the endpoint logic.**
+
+3. **Push to the Hugging Face Hub.**
+
+4. **Deploy as an Inference Endpoint via the Hugging Face UI.**
+
+---
+
+## Example `handler.py`
+
+This file loads the base model and LoRA adapter, and exposes a `__call__` method for inference.
+
+```python
+from typing import Dict, Any
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import PeftModel, PeftConfig
+import torch
+
+class EndpointHandler:
+    def __init__(self, path="."):
+        # Load base model and tokenizer
+        base_model_id = "<BASE_MODEL_ID>"  # e.g., "google/gemma-2b"
+        self.tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)
+        base_model = AutoModelForCausalLM.from_pretrained(base_model_id, trust_remote_code=True)
+        # Load LoRA adapter
+        self.model = PeftModel.from_pretrained(base_model, f"{path}/adapter")
+        self.model.eval()
+        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+        self.model.to(self.device)
+
+    def __call__(self, data: Dict[str, Any]) -> Dict[str, Any]:
+        prompt = data["inputs"] if isinstance(data, dict) else data
+        inputs = self.tokenizer(prompt, return_tensors="pt").to(self.device)
+        with torch.no_grad():
+            output = self.model.generate(**inputs, max_new_tokens=256)
+        decoded = self.tokenizer.decode(output[0], skip_special_tokens=True)
+        return {"generated_text": decoded}
+```
+
+- Replace `<BASE_MODEL_ID>` with the correct base model (e.g., `google/gemma-2b`).
+- The endpoint will accept a JSON payload with an `inputs` field containing the prompt.
+
+---
+
+## Notes
+
+- Make sure your `requirements.txt` includes `transformers`, `peft`, and `torch`.
+- For large models, use an Inference Endpoint with GPU.
+- You can customize the handler for chat formatting, streaming, etc.
+
+---
+
+## Quickstart
+
+1. Train your adapter with `train_gemma_unsloth.py`.
+2. Upload the `adapter` directory and `handler.py` to your Hugging Face repo.
+3. Deploy as an Inference Endpoint.
+4. Send requests to your endpoint!

handler.py

@@ -0,0 +1,23 @@
+from typing import Dict, Any
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import PeftModel
+import torch
+
+class EndpointHandler:
+    def __init__(self, path="."):
+        # Set your base model here (must match the one used for LoRA training)
+        base_model_id = "google/gemma-2b"  # CHANGE if you used a different base
+        self.tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)
+        base_model = AutoModelForCausalLM.from_pretrained(base_model_id, trust_remote_code=True)
+        self.model = PeftModel.from_pretrained(base_model, f"{path}/adapter")
+        self.model.eval()
+        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+        self.model.to(self.device)
+
+    def __call__(self, data: Dict[str, Any]) -> Dict[str, Any]:
+        prompt = data["inputs"] if isinstance(data, dict) else data
+        inputs = self.tokenizer(prompt, return_tensors="pt").to(self.device)
+        with torch.no_grad():
+            output = self.model.generate(**inputs, max_new_tokens=256)
+        decoded = self.tokenizer.decode(output[0], skip_special_tokens=True)
+        return {"generated_text": decoded}

requirements.txt

@@ -1,12 +1,5 @@
-fastapi
-uvicorn[standard]
-pydantic
-llama-cpp-python
-# Training dependencies for CCUF/Unsloth
-unsloth>=2024.7.0
-datasets>=2.20.0
-trl>=0.9.6
-peft>=0.11.1
-transformers>=4.36.0
-torch>=2.0.0
-accelerate>=0.24.0
+
+transformers
+peft
+torch
+datasets

sample_data/mini_test.jsonl

@@ -0,0 +1,2 @@
+{"prompt": "What is 2+2?", "response": "2+2 is 4."}
+{"prompt": "What color is the sky?", "response": "The sky is blue."}

training/train_gemma_unsloth.py

@@ -28,42 +28,20 @@ def _import_training_libs() -> Dict[str, Any]:
       If mode=="hf": AutoTokenizer, AutoModelForCausalLM, get_peft_model, LoraConfig, torch
     """
     # Avoid heavy optional deps on macOS (no xformers/bitsandbytes)
-    os.environ.setdefault("UNSLOTH_DISABLE_XFORMERS", "1")
-    os.environ.setdefault("UNSLOTH_DISABLE_BITSANDBYTES", "1")
     from datasets import load_dataset
-    from trl import SFTTrainer, SFTConfig
-    try:
-        from unsloth import FastLanguageModel
-        from transformers import AutoTokenizer
-        return {
-            "mode": "unsloth",
-            "load_dataset": load_dataset,
-            "SFTTrainer": SFTTrainer,
-            "SFTConfig": SFTConfig,
-            "FastLanguageModel": FastLanguageModel,
-            "AutoTokenizer": AutoTokenizer,
-        }
-    except ImportError as e:
-        logger.warning(
-            "Primary Unsloth import failed, falling back to HF+PEFT: %s",
-            e,
-            exc_info=True,
-        )
-        # Fallback: pure HF + PEFT (CPU / MPS friendly)
-        from transformers import AutoTokenizer, AutoModelForCausalLM
-        from peft import get_peft_model, LoraConfig
-        import torch
-        return {
-            "mode": "hf",
-            "load_dataset": load_dataset,
-            "SFTTrainer": SFTTrainer,
-            "SFTConfig": SFTConfig,
-            "AutoTokenizer": AutoTokenizer,
-            "AutoModelForCausalLM": AutoModelForCausalLM,
-            "get_peft_model": get_peft_model,
-            "LoraConfig": LoraConfig,
-            "torch": torch,
-        }
+    from transformers import AutoTokenizer, AutoModelForCausalLM, Trainer, TrainingArguments
+    from peft import get_peft_model, LoraConfig
+    import torch
+    return {
+        "load_dataset": load_dataset,
+        "AutoTokenizer": AutoTokenizer,
+        "AutoModelForCausalLM": AutoModelForCausalLM,
+        "get_peft_model": get_peft_model,
+        "LoraConfig": LoraConfig,
+        "Trainer": Trainer,
+        "TrainingArguments": TrainingArguments,
+        "torch": torch,
+    }
 
 
 def parse_args():
@@ -87,6 +65,10 @@ def parse_args():
     p.add_argument("--use-fp16", dest="use_fp16", action="store_true")
     p.add_argument("--seed", type=int, default=42)
     p.add_argument("--dry-run", dest="dry_run", action="store_true", help="Write DONE and exit without training (for CI)")
+    p.add_argument("--grpo", dest="use_grpo", action="store_true", help="Enable GRPO (if supported by Unsloth)")
+    p.add_argument("--cpt", dest="use_cpt", action="store_true", help="Enable CPT (if supported by Unsloth)")
+    p.add_argument("--export-gguf", dest="export_gguf", action="store_true", help="Export model to GGUF Q4_K_XL after training")
+    p.add_argument("--gguf-out", dest="gguf_out", default=None, help="Path to save GGUF file (if exporting)")
     return p.parse_args()
 
 
@@ -127,74 +109,46 @@ def main():
     # Training imports (supports Unsloth fast path and HF fallback)
     libs: Dict[str, Any] = _import_training_libs()
     load_dataset = libs["load_dataset"]
-    SFTTrainer = libs["SFTTrainer"]
-    SFTConfig = libs["SFTConfig"]
+    AutoTokenizer = libs["AutoTokenizer"]
+    AutoModelForCausalLM = libs["AutoModelForCausalLM"]
+    get_peft_model = libs["get_peft_model"]
+    LoraConfig = libs["LoraConfig"]
+    Trainer = libs["Trainer"]
+    TrainingArguments = libs["TrainingArguments"]
+    torch = libs["torch"]
 
-    # Environment for stability on T4 etc per Unsloth guidance
     os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
     os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")
 
     print(f"[train] Loading base model: {args.model_id}")
-    if libs["mode"] == "unsloth":
-        FastLanguageModel = libs["FastLanguageModel"]
-        AutoTokenizer = libs["AutoTokenizer"]
-        model, tokenizer = FastLanguageModel.from_pretrained(
-            model_name=args.model_id,
-            max_seq_length=args.cutoff_len,
-            # Avoid bitsandbytes/xformers
-            load_in_4bit=False,
-            dtype=None,
-            use_gradient_checkpointing="unsloth",
-        )
-        # Prepare LoRA via Unsloth helper
-        print("[train] Attaching LoRA adapter (Unsloth)")
-        model = FastLanguageModel.get_peft_model(
-            model,
-            r=args.lora_r,
-            lora_alpha=args.lora_alpha,
-            lora_dropout=0,
-            bias="none",
-            target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
-            use_rslora=True,
-            loftq_config=None,
-        )
-    else:
-        # HF + PEFT fallback (CPU / MPS)
-        AutoTokenizer = libs["AutoTokenizer"]
-        AutoModelForCausalLM = libs["AutoModelForCausalLM"]
-        get_peft_model = libs["get_peft_model"]
-        LoraConfig = libs["LoraConfig"]
-        torch = libs["torch"]
-
-        tokenizer = AutoTokenizer.from_pretrained(args.model_id, use_fast=True, trust_remote_code=True)
-        # Prefer MPS on Apple Silicon if available
-        use_mps = hasattr(torch.backends, "mps") and torch.backends.mps.is_available()
-        if not use_mps:
-            if args.use_fp16:
-                dtype = torch.float16
-            elif args.use_bf16:
-                dtype = torch.bfloat16
-            else:
-                dtype = torch.float32
+    tokenizer = AutoTokenizer.from_pretrained(args.model_id, use_fast=True, trust_remote_code=True)
+    use_mps = hasattr(torch.backends, "mps") and torch.backends.mps.is_available()
+    if not use_mps:
+        if args.use_fp16:
+            dtype = torch.float16
+        elif args.use_bf16:
+            dtype = torch.bfloat16
         else:
             dtype = torch.float32
-        model = AutoModelForCausalLM.from_pretrained(
-            args.model_id,
-            torch_dtype=dtype,
-            trust_remote_code=True,
-        )
-        if use_mps:
-            model.to("mps")
-        print("[train] Attaching LoRA adapter (HF/PEFT)")
-        lora_config = LoraConfig(
-            r=args.lora_r,
-            lora_alpha=args.lora_alpha,
-            target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
-            lora_dropout=0.0,
-            bias="none",
-            task_type="CAUSAL_LM",
-        )
-        model = get_peft_model(model, lora_config)
+    else:
+        dtype = torch.float32
+    model = AutoModelForCausalLM.from_pretrained(
+        args.model_id,
+        torch_dtype=dtype,
+        trust_remote_code=True,
+    )
+    if use_mps:
+        model.to("mps")
+    print("[train] Attaching LoRA adapter (PEFT)")
+    lora_config = LoraConfig(
+        r=args.lora_r,
+        lora_alpha=args.lora_alpha,
+        target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
+        lora_dropout=0.0,
+        bias="none",
+        task_type="CAUSAL_LM",
+    )
+    model = get_peft_model(model, lora_config)
 
     # Load dataset
     print(f"[train] Loading dataset: {args.dataset}")
@@ -229,29 +183,37 @@ def main():
 
     ds = ds.map(map_fn, remove_columns=[c for c in ds.column_names if c != "text"])
 
+    # Tokenize dataset
+    def tokenize_fn(ex):
+        return tokenizer(
+            ex["text"],
+            truncation=True,
+            max_length=args.cutoff_len,
+            padding="max_length",
+        )
+    tokenized_ds = ds.map(tokenize_fn, batched=True)
+
     # Trainer
-    trainer = SFTTrainer(
+    training_args = TrainingArguments(
+        output_dir=str(out_dir / "hf"),
+        per_device_train_batch_size=args.batch_size,
+        gradient_accumulation_steps=args.gradient_accumulation,
+        learning_rate=args.lr,
+        num_train_epochs=args.epochs,
+        max_steps=args.max_steps if args.max_steps else -1,
+        logging_steps=10,
+        save_steps=200,
+        save_total_limit=2,
+        bf16=args.use_bf16,
+        fp16=args.use_fp16,
+        seed=args.seed,
+        report_to=[],
+    )
+    trainer = Trainer(
         model=model,
+        args=training_args,
+        train_dataset=tokenized_ds,
         tokenizer=tokenizer,
-        train_dataset=ds,
-        max_seq_length=args.cutoff_len,
-        dataset_text_field="text",
-        packing=True,
-        args=SFTConfig(
-            output_dir=str(out_dir / "hf"),
-            per_device_train_batch_size=args.batch_size,
-            gradient_accumulation_steps=args.gradient_accumulation,
-            learning_rate=args.lr,
-            num_train_epochs=args.epochs,
-            max_steps=args.max_steps if args.max_steps else -1,
-            logging_steps=10,
-            save_steps=200,
-            save_total_limit=2,
-            bf16=args.use_bf16,
-            fp16=args.use_fp16,
-            seed=args.seed,
-            report_to=[],
-        ),
     )
 
     print("[train] Starting training...")
@@ -259,20 +221,18 @@ def main():
     print("[train] Saving adapter...")
     adapter_path = out_dir / "adapter"
     adapter_path.mkdir(parents=True, exist_ok=True)
-    # Save adapter-only weights if PEFT; Unsloth path is also PEFT-compatible
     try:
-        # Primary model saving logic
         model.save_pretrained(str(adapter_path))
     except Exception as e:
-        logger.error("Error during primary model saving: %s", e, exc_info=True)  # type: ignore
-        try:
-             # Fallback model saving logic
-             model.base_model.save_pretrained(str(adapter_path))  # type: ignore[attr-defined]
-        except Exception as fallback_e:
-             logger.error("Fallback model saving failed: %s", fallback_e, exc_info=True)  # type: ignore
-             pass  # Optionally re-raise or handle accordingly
+        logger.error("Error during model saving: %s", e, exc_info=True)
     tokenizer.save_pretrained(str(adapter_path))
 
+    # Optionally export to GGUF Q4_K_XL
+    if args.export_gguf:
+        print("[train] Export to GGUF is not supported in Hugging Face-only mode. Use llama.cpp's convert-hf-to-gguf.py after training.")
+        gguf_path = args.gguf_out or str(out_dir / "adapter-gguf-q4_k_xl")
+        print(f"python convert-hf-to-gguf.py --outtype q4_k_xl --outfile {gguf_path} {adapter_path}")
+
     # Write done file
     (out_dir / "DONE").write_text("ok")
     elapsed = time.time() - start

training_runs/devlocal/meta.json

@@ -1,6 +1,6 @@
 {
   "job_id": "devlocal",
-  "model_id": "unsloth/gemma-3n-E4B-it",
-  "dataset": "sample_data/train.jsonl",
-  "created_at": 1754620844
-}
+  "model_id": "unsloth/gemma-2b",
+  "dataset": "sample_data/mini_test.jsonl",
+  "created_at": 1754645651
+}

training_runs/realtrain/DONE

@@ -0,0 +1 @@
+dry_run

training_runs/realtrain/meta.json

@@ -0,0 +1,6 @@
+{
+  "job_id": "realtrain",
+  "model_id": "unsloth/gemma-3n-E4B-it",
+  "dataset": "sample_data/mini_test.jsonl",
+  "created_at": 1754644903
+}

training_runs/testload/DONE

@@ -0,0 +1 @@
+dry_run

training_runs/testload/meta.json

@@ -0,0 +1,6 @@
+{
+  "job_id": "testload",
+  "model_id": "unsloth/gemma-3n-E4B-it",
+  "dataset": "sample_data/mini_test.jsonl",
+  "created_at": 1754643124
+}