I prepared an internal LLM API Gateway for CompanyHouse to unify the use of different language models across the company infrastructure.
The solution provided one integration layer that could be used both as an SDK and through scalable HTTP requests. It supported local models as well as external providers such as OpenAI, Anthropic, and DeepInfra, making it easier to add or replace models when needed.
The gateway also tracked requests, token usage, and costs. It supported both streaming and batch processing, versioned prompts, guardrails, and multi-step LLM workflows with attached source documents.
This project created a more consistent and flexible foundation for using LLMs across internal systems.
Scope
- unified access to different models
- SDK and HTTP usage
- local models, OpenAI, Anthropic, DeepInfra
- token and cost tracking
- streaming and batch processing
- prompt versioning
- guardrails and multi-step workflows