← All work
AI Engineer (team project)2024

Multi-Agent CodeGen

A production-scale code-generation platform where five specialized CrewAI agents turn a natural-language request into tested, documented code — grounded by RAG over 2M+ embeddings and validated in an isolated Docker sandbox with iterative refinement.

GitHub
Multi-Agent CodeGen — RAG + CrewAI code generation platform
5
Specialized agents
6
Data sources
2M+
Vector embeddings
78.2%
Pass@1 accuracy
89.7%
Success rate
$0.16
Cost per request

The problem

Plausible is not working

  • No proof it runs

    A single model returns code that looks right and often is not — no tests, no execution

  • Ungrounded recall

    One prompt hallucinates APIs instead of drawing on real-world patterns

  • One overloaded role

    Analysis, coding, testing and docs crammed into a single prompt with no audit trail

The crew

5 agents, design-first

A request flows through five specialized agents. Test Designer and Programmer run in parallel — the tests are written before the code, so generation aims at a real spec rather than being graded after the fact.

  1. ANALYZE01

    Requirements Analyzer

    GPT-4

    Parses a natural-language request into a structured JSON specification the rest of the crew works from.

  2. SPEC02

    Test Designer

    GPT-4

    Writes the test suite from the spec, before any code exists. Design-first: the bar is set before the Programmer runs.

    parallel stage

  3. GENERATE03

    Programmer

    GPT-4 + RAG

    Generates production code grounded in retrieved real-world patterns, not free-form recall.

    parallel stage

  4. VERIFY04

    Test Executor

    Docker sandbox

    Runs the generated code against the test suite in an isolated container; failures trigger iterative refinement.

  5. DOCUMENT05

    Documentation Generator

    GPT-4

    Produces README and API docs for the accepted code, so the output ships explained.

Seven stages

Decisions, not descriptions

Arrow keys to step

Stage 01of 07

Ingest

Apache Airflow · ETL

  • 6 sources: GitHub, Stack Overflow, docs, issues/PRs, examples, blogs
  • Airflow DAGs schedule and orchestrate collection
  • 200+ high-quality GitHub repositories

Decision

Chose
Airflow-orchestrated multi-source ETL
Over
  • A single scraper
  • Manual dataset curation

Because

  • Six sources need dependency-aware scheduling and retries
  • Real code plus real Q&A plus real issues covers how developers actually write

Cost

Six connectors to maintain as each source's format drifts.

6
sources

Architecture

Ingest to serve
Multi-Agent CodeGen architecture: six sources feed an Airflow ETL that parses, cleans and deduplicates code, embeds it into a Pinecone vector store of 2M+ embeddings alongside GCS and BigQuery, served by a FastAPI RAG backend into a CrewAI crew of five agents, gated by a Docker sandbox and guardrails, and surfaced through a Streamlit dashboard.

Results

Measured, not asserted
Measured evaluation results
MetricResultNote
Pass@1 accuracy78.2%across 87 test cases
Overall success rate89.7%generations that pass
Average quality score7.8 / 10graded output
Average latency23.4send to end per request
Cost per request$0.16~18K tokens
Cache hit rate40%cost optimisation

Because every generation is grounded in retrieval and gated by sandboxed tests, the output is code you can trust to run — not just code that looks plausible.

Stack

Everything used

Agents

  • CrewAI
  • GPT-4
  • LangChain

Retrieval

  • Pinecone
  • text-embedding-3-large
  • RAG

Backend

  • FastAPI
  • Async Python

Data & orchestration

  • Apache Airflow
  • Google Cloud Storage
  • BigQuery

Testing & safety

  • Docker sandbox
  • pytest
  • Security scanning
  • HITL review

Interface

  • Streamlit
See these across every project