A scalable Retrieval-Augmented Generation (RAG) system designed to index, query, and synthesize information from foundational database research papers. By leveraging modern microservices, it bridges the gap between raw academic research and actionable engineering insights.
This project serves as a "living encyclopedia" for database engineering. Built on a multi-language microservices architecture, it processes dense academic PDFs using Docling, stores embeddings in Postgres via pgvector, and uses high-performance API gateways to serve a modern Angular frontend.
The project uses a containerized, decoupled microservices architecture designed for high precision and scalability.
- Angular UI (
/apps/ui): A modern, responsive web frontend built with Angular (v20). It serves as the primary interface for users to submit research queries, explore papers, and view AI-generated, citation-backed answers. It is served behind an Nginx reverse proxy in production.
- Go API Gateway (
/apps/api): A high-throughput REST API written in Go 1.23. It acts as the central orchestrator, handling:- User Authentication (JWT-based).
- Rate Limiting and Telemetry (Prometheus).
- Request orchestration between the UI, the PostgreSQL database, and the Python RAG Engine.
- PostgreSQL interaction for managing user metadata, chat history, and document states.
- FastAPI Service (
/apps/api/main.py): A Python-based AI service responsible for the heavy lifting of the RAG pipeline.- Ingestion & Parsing: Uses Docling for high-fidelity extraction of text, tables, and citations from academic PDFs.
- Embedding: Utilizes local Sentence-Transformers (e.g.,
BAAI/bge-small-en-v1.5) to generate dense semantic vectors, avoiding expensive API calls for high-volume ingestion. - Vector Storage: Stores and queries embeddings using pgvector on Postgres.
- LLM Integration: Combines retrieved contexts with powerful prompts to generate grounded answers using external LLM APIs (e.g., OpenAI, Anthropic).
- PostgreSQL (15): Relational database managing application state (users, auth, chat logs) and, via pgvector, the embedded document chunks used for semantic search.
- Local File System: Manages raw PDFs, parsed JSONs, and embedding artifacts in the
/datadirectory.
- Document Ingestion (
scripts/&services/ingestion/): Raw PDFs of research papers are downloaded into/data/raw_pdfs. The system uses Docling to accurately parse the structure (headings, paragraphs, tables) and convert them to parsed JSON representations. - Chunking & Embedding: The parsed documents are broken down into semantically meaningful chunks (Hierarchical Chunking). The Python engine uses a local Sentence-Transformer model to convert these text chunks into dense numeric vectors.
- Vector Storage: The vectors, along with metadata (paper title, chunk index), are stored in Postgres via pgvector for rapid similarity search.
- Query Execution:
- A user submits a query via the Angular UI.
- The Go API Gateway authenticates the user, logs the query in PostgreSQL, and forwards the request to the Python RAG Engine.
- The Python engine embeds the user's query, performs a semantic search against pgvector to find the most relevant document chunks, and optionally reranks them.
- The retrieved context is formatted into a strict prompt and sent to an LLM to generate a grounded, cited response.
- The answer is streamed back through the Go Gateway to the UI.
📖 For the long version — chunking strategy, reranking, hybrid retrieval, and the chat-memory model — see the deep-dive docs.
You can run the entire multi-service application locally using Docker Compose.
- Docker & Docker Compose
- Git
- (Optional) Python 3.9+ and Go 1.23+ if running services manually outside of Docker.
-
Clone the Repository:
git clone https://github.com/your-org/database-development.git cd database-development -
Configure Environment Variables: Copy the example environment file and adjust any keys (like external LLM API keys if required by the python service).
cp configs/service.env.example .env
-
Start the Infrastructure via Docker Compose: This command will build the Go API, Python RAG Engine, and Angular UI containers, and spin up PostgreSQL.
docker-compose up --build -d
-
Verify the Services: Check that all containers are running successfully:
docker-compose ps
Once the containers are up and running, you can access the different layers of the system:
- Frontend UI: http://localhost:4200
- Go API Gateway: http://localhost:8081
- Python RAG Engine: http://localhost:8000
- PostgreSQL Database:
localhost:5434(Credentials specified indocker-compose.yml)
To gracefully stop and remove the containers:
docker-compose down(Note: If you want to wipe the database and vector store volumes, run docker-compose down -v)
Long-form, plain-English walkthroughs of the system. Start the docs server and open the hub — each doc opens in its own maximized window, and F toggles true fullscreen inside any of them.
python3 -m http.server 8090 --directory docs
# then open http://localhost:8090/| Document | What it covers |
|---|---|
| How the Research Assistant Works | The full RAG pipeline: a question's journey, chunking in depth, reranking, hybrid retrieval, scaling, and a glossary. |
| The Database Design & How It Remembers a Conversation | The two-schema Postgres layout, the chat tables, what the model actually sees each turn, conversation reload, and guest vs. signed-in state. |
(These are HTML files — GitHub shows them as source in the browser. Clone the repo, or use the server above, to read them rendered.)
/ai: Contains system prompts, governance rules, and prompt engineering evaluation details./experiments: Scripts used for evaluating chunking, embedding models, and reranking strategies.