OpenPipe/OpenPipe

Note: we’ve temporarily stopped development on the open-source version of OpenPipe to integrate some proprietary third-party code. We hope to make the non-proprietary parts of the repository open again under an open core model once we have the bandwidth to do so!

OpenPipe

Open-source fine-tuning and model-hosting platform.

Demo - Running Locally - Docs

Use powerful but expensive LLMs to fine-tune smaller and cheaper models suited to your exact needs. Query your past requests and evaluate models against one another. Switch between OpenAI and fine-tuned models with one line of code.

Features

• Easy integration with OpenAI's SDK in both Python and TypeScript.
• Python SDK
• TypeScript SDK
• OpenAI-compatible chat completions endpoint.
• Fine-tune GPT 3.5, Mistral, and Llama 2 models. Host on-platform or download the weights.
• Model output is OpenAI-compatible.
• Switching from GPT 4 to a fine-tuned Mistral model only requires changing the model name.
• Query logs using powerful built-in filters.
• Import datasets in OpenAI-compatible JSONL files.
• Prune large chunks of duplicate text like system prompts.
• Compare output accuracy against base models like gpt-3.5-turbo.

Supported Base Models

• mistralai/Mixtral-8x7B-Instruct-v0.1
• OpenPipe/mistral-ft-optimized-1227
• meta-llama/Llama-3-8B
• meta-llama/Llama-3-70B
• gpt-3.5-turbo-0613
• gpt-3.5-turbo-1106
• gpt-3.5-turbo-0125

Documentation

• See docs

Running Locally

1. Install Postgresql. 2. Install NodeJS 20 (earlier versions will very likely work but aren't tested). 3. Install pnpm: npm i -g pnpm 4. Clone this repository: git clone https://github.com/openpipe/openpipe 5. Install the dependencies: cd openpipe && pnpm install 6. Create a .env file (cd app && cp .env.example .env) and enter your OPENAI_API_KEY. 7. If you just installed postgres and wish to use the default DATABASE_URL run the following commands:

psql postgres
CREATE ROLE postgres WITH LOGIN PASSWORD 'postgres';
ALTER ROLE postgres SUPERUSER;

8. Update DATABASE_URL if necessary to point to your Postgres instance and run pnpm prisma migrate dev in the app directory to create the database. 9. Create a GitHub OAuth App, set the callback URL to /api/auth/callback/github, e.g. http://localhost:3000/api/auth/callback/github. 10. Update the GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET values from the Github OAuth app (Note: a PR to make auth optional when running locally would be a great contribution!). 11. To start the app run pnpm dev in the app directory. 12. Navigate to http://localhost:3000

Using Locally

import os
from openpipe import OpenAI
client = OpenAI(
    api_key="Your API Key",
    openpipe={
        "api_key": "Your OpenPipe API Key",
        "base_url": "http://localhost:3000/api/v1", # Local OpenPipe instance
    }
)
completion = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "system", "content": "count to 10"}],
    openpipe={
        "tags": {"prompt_id": "counting"},
        "log_request": True
    },
)

Testing Locally

1. Copy your .env file to .env.test. 2. Update the DATABASE_URL to have a different database name than your development one 3. Run DATABASE_URL=[your new datatase url] pnpm prisma migrate dev --skip-seed --skip-generate 4. Run pnpm test