RAG prototype for structured configurations
RAG prototype translating a natural-language request into JSON or YAML configuration for professional software.
Business problem
The software concerned is used to create detailed technical configurations. This flexibility is useful, but it makes advanced use difficult for users who are not familiar with programming.
The challenge was to make this creation easier from natural language without ignoring the software’s constraints.
Mission
Over nine weeks, I designed and evaluated a pipeline from a user request to validated JSON or YAML configuration. I worked on the domain documentation, RAG architecture, evaluation dataset, and API delivery of the prototype.
Data & documentation
The existing documentation was written for users and was not well suited to semantic search. I restructured it in YAML to better represent the information and constraints required by the pipeline.
I also began building a set of 300 requests in Label Studio, with LLM-generated pre-annotations. I had not finished reviewing them by the end of the internship.
Architecture
The first approach retrieved documentation directly from the complete request. It sometimes confused features that were phrased similarly but behaved differently.
I therefore separated request element extraction, document retrieval, and generation. For retrieval, I used a parent-child strategy with re-ranking: short examples were used to retrieve and then select the associated documentation.
Evaluation
I monitored document retrieval, element extraction, and output compliance separately. Langfuse was used to trace executions and RAGAS to test metrics suited to RAG.
The protocol and pipeline evolved in parallel, without a complete independent dataset for the latest version. I therefore do not present intermediate scores as a measure of its final performance.
I also used t-SNE to identify documents that were too close in the embedding space or rarely retrieved.
Prototype structuring and deployment
The pipeline was exposed through FastAPI and a Streamlit interface supported testing. The application was containerised with Docker Compose.
Pydantic checked that outputs followed the expected format. This validation did not guarantee their domain correctness.
Results
The internship resulted in a working prototype connecting document retrieval, structured generation, format validation, and execution tracing.
The code, data, documentation, prompts, configuration schema, and internship report are not public.
Limits & next steps
The prototype processed one request and then produced one response. It did not yet support a conversation with the user to clarify an incomplete, ambiguous, or impossible request.
If I revisited this work, I would add this clarification loop. I would also finish annotating the 300 requests and reserve an independent dataset to compare pipeline versions.