An On-Premise Diffusion Deep Research Model & Evaluation Framework for Resource-Constrained Environments
Year: 2026
Type: dissertation
Abstract: State-of-the-art agentic large language model architectures for generating longform citation-grounded reports, called deep research models, are rapidly developing.These models are however mostly closed-source and rely heavily on cloudbased infrastructures, exceptionally large language models and web retrievers.These options are not acceptable for organizations with high data security demands and using a locally hosted alternative is therefore imperative.This master's thesis sets out to design, implement, and evaluate a deep research model that can operate entirely on-premise, employing a design science research methodology with two primary artifacts.The domain of application is requirements management.First, we present a novel deep research model, called Diffusion On-premise Research Agent (DORA), inspired by the test-timediffusion style of model presented by Google Cloud researchers.Second, we developed a novel evaluation framework, called Testing Local Deep Research (TLDR).It uses an LLM-as-a-judge paradigm along with dynamic claim extraction and retrieval for factual validation, scoring reports on four text quality metrics and correctness.The results demonstrate that it is practically viable to run deep Rrsearch architectures on-premise, provided that appropriate design choices are made and smaller language models are used.The proposed model DORA outperforms all evaluated state-of-the-art models when using a comparable LLM on the presented benchmark TLDR with a requirements management dataset.Employing an LLM viable for hosting locally for DORA shows results on par with SOTA alternatives.Furthermore, the thesis indicates that evaluation of long-form citation-grounded reports is emerging as an important but complex challenge, mainly due to the poor scalability of human expert evaluation and the lack of gold-standard reference texts.The results align well with literature in that reference-free absolute scoring is largely unreliable for this task and relative rankings need to be employed to get dependable evaluations.
Authors: Malm, Arvid, Boris-Möller, Wolmar
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W7166259213