arjun988/promptModel

PromptForge

Local-first prompt quality scoring and optimization.

Published on PyPI as tuneprompt.

PromptForge scores LLM prompts across seven quality dimensions, then rewrites weak prompts into clear, intent-preserving instructions — runnable on your machine via Python API, CLI, or Gradio.

"Make an app about social media like facebook"
                    │
                    ▼
         ┌─────────────────────┐
         │  Quality Scorer     │  ModernBERT · ~150M
         │  41.5 → issues…     │
         └──────────┬──────────┘
                    ▼
         ┌─────────────────────┐
         │  Prompt Optimizer   │  Qwen2.5-1.5B + LoRA
         └──────────┬──────────┘
                    ▼
   Build a social media app similar to Facebook…
   profiles · feed · likes · constraints · output format
                    │
                    ▼
              41.5 → 94.0

PyPI · Docs · Product plan · Contributing · License

---

Why PromptForge

Most prompt tools either judge quality or rewrite text. PromptForge does both in one local pipeline:

| Capability | What you get | |------------|--------------| | Multi-dimension scoring | Clarity, specificity, context, goals, constraints, completeness, actionability | | Intent-preserving rewrite | Optimizes the same topic — not a generic template | | Validation & fallback | Rejects repetitive / off-topic generations | | Runs locally | ~1.65B total params; trains on 8 GB GPUs | | Dev-ready surface | pip package, CLI, Gradio demo, Colab notebooks |

No API key required for inference once models are on disk.

---

Example

Input

Make an app about social media like facebook and stuff

Output (optimizer)

Build a social media app similar to Facebook for product managers.
This is for a portfolio demo.
Core features:
User profiles and friend connections
News feed with posts, likes, and comments
Basic notifications

Requirements:
Use Python and Flask.
Keep the first version simple and usable
Include error handling and clear project structure

Include short examples.

Quality: 41.5 → 94.0 (Δ +52.5) · topic preserved · no fallback

---

Models

| Component | Base | Size | Training | |-----------|------|------|----------| | Scorer | ModernBERT-base | ~150M | Full fine-tune | | Optimizer | Qwen2.5-1.5B-Instruct | 1.5B | LoRA (base frozen) |

Weights are not stored in git. Train locally or download from Hugging Face:

pip install tuneprompt
python -m promptforge download \
  --quality-repo ArjunShukla/PromptForge-Quality \
  --optimizer-repo ArjunShukla/PromptForge-Optimizer

• Quality: https://huggingface.co/ArjunShukla/PromptForge-Quality
• Optimizer: https://huggingface.co/ArjunShukla/PromptForge-Optimizer

Results

Quality scorer (held-out):

| Split | MAE | Pearson | |-------|----:|--------:| | Validation | 2.73 | 0.993 | | Test (overall) | 0.96 | 0.999 |

---

Quickstart

Install

pip install tuneprompt

Package: tuneprompt on PyPI Import module: promptforge · CLI: tuneprompt or promptforge

Download models & run

python -m promptforge download \
  --quality-repo ArjunShukla/PromptForge-Quality \
  --optimizer-repo ArjunShukla/PromptForge-Optimizer
python -m promptforge init
python -m promptforge doctor
python -m promptforge run "Build me a website for a startup"
python -m promptforge analyze "Make an app." --json

Same via CLI entrypoints:

tuneprompt run "Build me a website for a startup"
or
promptforge run "Build me a website for a startup"

On some Windows setups, Application Control blocks .venv\Scripts\*.exe. Prefer python -m promptforge ….

Python API

from promptforge import PromptForge
After download + init, or pass Hub / local paths:
pf = PromptForge(
    quality_model_path="ArjunShukla/PromptForge-Quality",
    optimizer_model_path="ArjunShukla/PromptForge-Optimizer",
)
print(pf.analyze("Make an app."))
result = pf.run("Make an app about social media like facebook and stuff")
print(result["optimized_prompt"])
print(result["delta"]["quality_score"])

Install from source (optional)

git clone https://github.com/arjun988/promptModel.git
cd promptModel
python -m venv .venv
Windows:  .venv\Scripts\activate
Unix:     source .venv/bin/activate

pip install -U pip
pip install -e ".[demo,dev]"

GPU tip: default pip install torch is often CPU-only. For NVIDIA (incl. RTX 50-series):

pip uninstall -y torch torchvision torchaudio
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

Full local guide: docs/LOCAL.md

---

Train your own

Anyone can improve the models with their own data:

# Phase 1 — quality scorer
python scripts/train_quality.py --require-gpu --regenerate
Phase 2 — optimizer (recommended on 8GB GPUs)
python scripts/train_optimizer.py --require-gpu --fast --regenerate

| Flag / config | Purpose | |---------------|---------| | --fast | Loads configs/optimizer_fast_8gb.yaml | | --regenerate | Rebuild curated optimizer dataset | | load_in_4bit: true | Use if 1.5B LoRA OOMs |

---

CLI

| Command | Description | |---------|-------------| | init | Create ~/.promptforge and register model paths | | doctor | GPU / config / model health check | | download | Pull models from Hugging Face | | analyze | Score a prompt | | optimize | Rewrite a prompt | | run | Score → optimize → compare | | eval | Pipeline evaluation reports | | space | Launch Gradio demo | | train-quality / train-optimizer | Training entrypoints |

---

Project layout

promptModel/
├── src/promptforge/     # Package: scorer, optimizer, pipeline, CLI
├── configs/             # Training + local defaults
├── scripts/             # Train / eval / Hub export
├── demo/                # Gradio app
├── notebooks/
│   ├── colab/           # Self-contained experiments
│   └── package/         # Thin package drivers
├── docs/                # PRD + local setup
├── tests/
└── pyproject.toml

Notebooks: notebooks/README.md

---

Roadmap

| Phase | Deliverable | Status | |------:|-------------|--------| | 1 | Multi-dimension quality scorer | Done | | 2 | Intent-preserving prompt optimizer (LoRA) | Done | | 3 | Combined pipeline + eval + Gradio | Done | | 4 | Local package + CLI (tuneprompt) | Done | | 5 | VS Code / Cursor extension | Planned |

---

Contributing

Issues and PRs welcome. See CONTRIBUTING.md for setup, style, and PR expectations.

pip install -e ".[dev]"
pytest -q

---

License

MIT © PromptForge contributors