Peer-reviewed study · arXiv:2605.05482
FinRAG‑12B: a production‑validated recipe for grounded question answering in banking
A 12-billion-parameter model tuned on 143M tokens that outperforms GPT-4.1 on answer quality and citation grounding in banking QA — at 20–50× lower cost. Deployed across 40+ financial institutions with a +7.1pp lift in query resolution.
Denys Katerenchuk¹ · Pablo Duboue²* · Keelan Evanini³* · David Gondek* · Nithin Govindugari¹ · Olivier Allauzen¹ · Joshua Baptiste¹ · David J More¹ · Joshua Schechter¹ — ¹Backbase (New York) · ²Textualization (Vancouver) · ³NBME (Philadelphia) · *work done while at Kasisto · arXiv:2605.05482v2 [cs.AI] · 21 Jul 2026
Answer quality (JudgeLM)
0.00
vs 5.72 GPT-4.1 · 5.70 base
Training tokens
0M
1,400 steps · 360 GPU-hours · $1,800
Query resolution lift
+0.0pp
p<0.001 · 7-month production A/B
Cost per query
$0.000
GPT-4.1: $0.02–0.05 (20–50×)
Abstract
Large language models (LLMs) are rapidly being adopted across various domains. However, their adoption in banking industry faces resistance due to demands for high accuracy, regulatory compliance, and the need for verifiable and grounded responses. We present a unified, data‑efficient framework for training grounded domain‑specific LLMs that optimizes answer quality, citation grounding, and calibrated refusal under real‑world deployment constraints. First, we describe a data generation pipeline that combines LLM‑as‑a‑Judge filtering, citation annotation, and curriculum learning with only 143M tokens. The resulting 12B model achieves high answer quality outperforming GPT‑4.1 on citation grounding, with a modest citation tradeoff versus the untuned base. Second, we propose a calibrated refusal mechanism: training on 22% unanswerable examples yields a 12% “I don’t know” rate, substantially improving over the base model’s unsafe 4.3% rate while avoiding GPT‑4.1’s over‑refusal (20.2%). Third, we present an end‑to‑end methodology spanning from data curation to quantized serving. The system is deployed at 40+ financial institutions, achieving a 7.1 percentage point improvement in query resolution (p < 0.001). Additionally, the model delivers 3–5× faster responses at 20–50× lower cost compared to GPT‑4.1.
Introduction
Large language models have transformed natural language processing across a wide range of applications, including customer support, content generation, and code synthesis. However, their adoption in the regulated banking industry remains limited due to their tendency to hallucinate, exhibit over‑agreeable behavior, and lack alignment with domain‑specific knowledge and constraints. In addition, banking applications operate over dynamic, frequently changing data, such as interest rates, account balances, and institutional policies, requiring models to ground their responses in up‑to‑date information retrieved from internal or external sources. This setting places strict demands on factual accuracy, traceability, and latency.
Consider a customer asking a bank’s virtual assistant about early mortgage payoff penalties. The system must retrieve the latest policy, interpret it correctly, and generate a response grounded in source documents within seconds. If the relevant information is unavailable, the model must explicitly refrain from answering rather than fabricate an answer, as incorrect responses may lead to regulatory and financial risk. Achieving this level of reliability and responsiveness requires moving beyond off‑the‑shelf LLMs toward domain‑specialized, grounded systems.
To address these challenges, we introduce FinRAG‑12B, a 12B‑parameter LLM optimized for retrieval‑augmented generation (RAG) in banking. We propose a unified, data‑efficient training framework that jointly improves answer quality, citation grounding, and calibrated refusal. Our approach centers on a multi‑stage data curation pipeline combining LLM‑as‑a‑Judge filtering, citation annotation, and curriculum learning, achieving 73% citation quality with only 143M tokens.
Through systematic ablations, we show that incorporating 22% unanswerable examples enables calibrated “I don’t know” responses, substantially reducing hallucinations while avoiding excessive refusal on answerable queries. Finally, we present a complete, production‑ready methodology from data curation to quantized serving that was validated through deployment across 40+ financial institutions. Our contributions are threefold: (1) a high‑quality, data‑efficient pipeline for generating grounded training data; (2) a calibrated refusal strategy based on controlled negative sampling to mitigate hallucinations; and (3) an end‑to‑end framework for training and deploying LLMs in production RAG systems for banking.
Related work
The field of financial NLP has advanced rapidly in recent years. BloombergGPT trained a 50B‑parameter model on 346B tokens of proprietary financial data, achieving strong performance on tasks such as sentiment analysis, named entity recognition, and question answering — but the model was not publicly released, and the work does not report deployment‑critical metrics such as latency, inference cost, or hallucination rates. FinGPT demonstrated that parameter‑efficient fine‑tuning (e.g., LoRA) can achieve competitive results on financial sentiment analysis at low cost, but focuses primarily on classification tasks rather than generative RAG with citation requirements. Earlier work such as FinBERT established the importance of domain adaptation, showing that continued pre‑training on financial corpora improves downstream performance.
Retrieval‑augmented generation (RAG) has emerged as a standard approach for improving factuality by grounding model outputs in external knowledge sources. While RAG reduces hallucinations, it introduces new failure modes: models may ignore retrieved context, rely on irrelevant passages, or generate unsupported claims. LLMs further exhibit positional bias in long contexts, attending disproportionately to tokens at the beginning and end of the input. We mitigate this issue through randomized context placement during training, sampled from a mixture of discrete right trapezoidal harmonic decay distribution.
A growing body of work emphasizes the importance of data quality over scale in LLM training. Phi‑3 demonstrates that smaller models trained on carefully filtered data can match or exceed the performance of larger models trained on noisier corpora; LIMA shows that a small number of high‑quality instruction examples can suffice for alignment. Subsequent work explores principled data selection strategies, including pairwise quality ranking, loss‑based example selection, and optimal data mixing. These findings motivate our multi‑stage training pipeline, which sequences data of varying quality and provenance rather than mixing all sources uniformly.
For parameter‑efficient training, we build on LoRA and DoRA, which learn low‑rank weight updates to adapt large models without full fine‑tuning. Curriculum learning has been shown to improve generalization by presenting training examples in order of increasing difficulty; in domain‑adaptation settings, continued pretraining on domain‑specific corpora improves downstream task performance. We adopt a two‑stage curriculum: Stage 1 adapts the model to general financial language using open‑source data, while Stage 2 specializes the model on proprietary banking interactions. This design also enhances reproducibility, as Stage 1 can be replicated using publicly available datasets. Unlike prior work, we integrate data curation, grounding, and refusal calibration into a unified framework optimized for real‑world deployment in regulated environments.
System architecture
3.1Base model selection
The choice of base model is critical for balancing performance and deployment constraints. We require strong instruction‑following capabilities, with low hallucination rates alongside efficiency in latency and cost. After surveying models such as Phi‑4 14B and Qwen3 14B, we select Gemma 3 12B‑IT as it provides competitive instruction‑following performance, a 128K context window, and permissive commercial licensing while remaining computationally efficient.
3.2Training data pipeline
Training data quality is a primary determinant of LLM performance, particularly in high‑stakes domains such as banking. While financial institutions possess large volumes of data, much of it is noisy, outdated, or contains personally identifiable information (PII), limiting its usability. To address this, we design a multi‑stage data curation pipeline that prioritizes quality over scale. The final corpus consists of 98,648 samples (143M tokens) drawn from a combination of open‑source and proprietary data, filtered and processed through multiple quality control stages.
| Source | Samples | License |
|---|---|---|
| RAG-v1 (Open) | 43,581 | Apache 2.0 |
| SEC Reports (Synthetic QA) | 16,773 | Public |
| CommonCrawl (Financial) | 20,499 | CC0 |
| Refusal Calibration (Proprietary) | 17,795 | Internal |
| Total | 98,648 | — |
Table 1 — Training data composition. Stage 1 uses open-source data (RAG-v1, CommonCrawl); Stage 2 adds proprietary banking conversations and synthetic QA from SEC filings.
3.2.1 RAG‑v1.We curate 43,581 samples from the glaiveai/RAG‑v1 dataset, a synthetic dataset designed for retrieval‑augmented generation. Each sample includes a question, supporting documents, and a cited answer. We apply JudgeLM to filter out low‑quality responses (score < 5), ensuring consistent supervision quality.
3.2.2 Synthetic SEC QA. Purely synthetic QA generation pipelines often suffer from distribution mismatch with real user queries, regurgitative artifacts, and limited control over difficulty and grounding. To address these limitations, we propose a style‑conditioned, multi‑stage QA generation pipeline grounded in financial SEC filings (10‑K and 10‑Q), producing 16,773 high‑quality training samples. The five‑stage pipeline:
- Segment each SEC filing into passage‑length chunks (400–600 tokens) using semantic boundaries when available and fixed windows otherwise — reflecting the granularity of retrieved context in real RAG systems while preserving local coherence.
- For each passage, generate questions at four difficulty levels (easy, medium, hard, expert) via prompt conditioning, operationalized through required reasoning depth. Easy and medium questions most closely match real user queries and receive the highest sampling weight.
- Rephrase generated questions using few‑shot prompting conditioned on empirical query distributions — controlling for (i) query style (fragment, “how‑do‑I”, “what‑is”), (ii) length (sampled from a log‑normal distribution), and (iii) formality. This transforms verbose, interrogative LLM outputs into concise, user‑like queries (e.g., “What is the minimum credit score required for mortgage approval?” → “min credit score for mortgage”).
- Generate answers using the original question and its corresponding gold passage, with explicit instructions to ground all claims in the provided context and enforce citation‑style outputs — each answer traceable to specific source spans.
- Inject 3–7 distractor passages sampled from topically similar documents, with the gold passage randomly interleaved at positions drawn from a mixture of discrete right trapezoidal harmonic decay distribution — training the model to identify and ground responses in the correct evidence rather than relying on positional heuristics.
Compared to single‑shot generation, this pipeline substantially improves alignment with real query distributions, reducing question‑type divergence by 10× (JS: 0.434 → 0.041) and matching average query length (8.85 vs. 9.91 words):
| Metric | Single-shot | Multi-step | Real queries |
|---|---|---|---|
| Avg. length (words) | 19.55 | 8.85 | 9.91 |
| Jaccard w/ real ↑ | 0.098 | 0.140 | — |
| Type entropy ↑ | 1.418 | 1.745 | 2.281 |
| Type JS divergence ↓ | 0.434 | 0.041 | — |
| Coverage, cos. ↑ | 0.464 | 0.520 | — |
| Distinct-2 ↑ | 0.295 | 0.451 | 0.721 |
| Financial term recall ↑ | 0.902 | 0.951 | — |
Table 2 — Single-shot vs. multi-step QA generation pipeline on lexical, question-type, semantic, diversity, and domain metrics. ↑ = higher is better, ↓ = lower is better.
3.2.3 CommonCrawl financial subset. Because user data is subject to strict privacy constraints, we adopt a hybrid approach: a random forest classifier identifies banking‑relevant content; real user questions containing no PII and appearing more than n times are extracted (preventing memorization of rare, potentially sensitive queries); and these are cross‑referenced with the classifier to retrieve relevant passages. The resulting 20,499 samples are grounded in real‑world usage while remaining privacy‑compliant, processed with the same multi‑stage strategy as §3.2.2.
3.2.4 Banking refusal calibration data.In production RAG systems, retrieved context may be incomplete or insufficient. Instruction‑tuned LLMs exhibit a well‑documented tendency toward over‑compliance (sycophancy), often producing plausible but ungrounded answers even absent supporting evidence — in regulated domains, a compliance risk. We construct a dedicated refusal calibration dataset from real banking conversations: each example pairs a user query with retrieved context that is topically relevant but lacks sufficient information, and an explicit “I don’t know” target. This subset comprises 17,795 examples (22% of the training mixture), shifting the model toward conservative, evidence‑based responses.
3.3Positional bias
RAG models exhibit positional bias, disproportionately attending to earlier or later context segments. To mitigate this, we randomize the placement of relevant passages among distractors during training, sampling positions from a mixture of discrete right trapezoidal harmonic decay distribution:
P(X = x) = 1 / (N − Kmin + 1) · ΣK = max(x, Kmin)N 1/KThis strategy reduces systematic bias toward fixed context positions and encourages more uniform attention over retrieved evidence.
3.4Model training
We adopt a two‑stage curriculum inspired by domain‑adaptive pretraining, progressively transitioning from grounded domain adaptation to task‑specific specialization. In Stage 1 (Domain Adaptation), the model is trained on 60,354 samples from RAG‑v1 and SEC‑derived synthetic QA using a conservative learning rate of 1×10⁻⁶ with cosine decay — establishing citation alignment, evidence‑based reasoning, and familiarity with financial text on largely reproducible data. In Stage 2 (Task Specialization), training continues from the Stage 1 checkpoint on 38,294 samples drawn from CommonCrawl financial data and proprietary banking conversations at a higher learning rate of 5×10⁻⁶ with linear decay — introducing real‑world query distributions and production‑specific behaviors, including calibrated refusal, robustness to noisy retrieval, and institution‑specific formatting.
We find that staged training is critical: jointly training on all data leads to degraded performance (§5.3), underscoring the importance of separating grounded domain learning from real‑world adaptation.
| Stage | Data source | Samples | LR |
|---|---|---|---|
| 1 | RAG-v1 + SEC Synth. QA | 60,354 | 1×10⁻⁶ |
| 2 | CC + Propriet. Banking | 38,294 | 5×10⁻⁶ |
Table 3 — Curriculum learning stages. Stage 1 uses grounded, reproducible data for domain adaptation; Stage 2 introduces real-world and proprietary data for task specialization.
3.5Refusal calibration
Calibrating a model to abstain when retrieved context is insufficient requires careful control of the negative‑example ratio. An insufficient proportion causes the model to default to its sycophantic prior, producing plausible but unsupported responses (high false‑positive rate). Conversely, an excessive proportion leads to over‑refusal (high false‑negative rate). Sweeping the negative ratio from 10% to 30% in 2‑percentage‑point increments, we find that a ratio of 22% yields the best trade‑off, minimizing unsupported answers while preserving recall on answerable queries. Beyond 26%, performance degrades due to excessive conservatism, with a sharp decline in recall.
3.6Training configuration
We fine‑tune the model using LoRA applied to all attention and MLP layers (r=64, α=256, dropout 0.05). Optimization uses 8‑bit AdamW at a learning rate of 2×10⁻⁵, per‑device batch size 4 with gradient accumulation over 4 steps (effective batch 16), and a maximum sequence length of 16,384 tokens. We reserve 1% of the training data as a validation set with early stopping (patience 5), restoring the best checkpoint on validation loss. Training completes in 1,400 steps, requiring approximately 360 GPU‑hours on 8× RTX A6000 GPUs at a total cost of $1,800 (360 GPU‑hours at $5/hour).
3.7Quantization
To meet production latency and memory constraints, we quantize the fine‑tuned model to W4A16 (4‑bit weights, 16‑bit activations) using SmoothQuant. This reduces the model footprint from 24GB to 8.4GB (2.86× compression), enabling efficient single‑GPU deployment. Despite aggressive compression, performance degradation is minimal: the quantized model retains over 99% of full‑precision citation quality, demonstrating that grounded generation remains robust under low‑bit quantization.
Evaluation
4.1Dataset
Proprietary banking test set. We construct a dataset of 258 RAG examples from three financial institutions spanning retail (account inquiries, loan information, branch hours) and commercial banking (treasury services, merchant processing). To reflect real‑world retrieval conditions, some examples are unanswerable — the provided context is topically related but does not contain sufficient information. Each example consists of a user query, five retrieved passages (1–2 relevant and 3–4 distractors), human‑annotated reference answers with citation labels, and a binary answerability label — enabling joint evaluation of grounding quality and refusal behavior under realistic RAG settings.
Public benchmark. To assess generalization beyond proprietary data, we additionally evaluate on FinanceBench, which consists of 150 questions over SEC filings requiring free‑form, grounded responses — evaluating both factual retrieval and citation grounding.
4.2Metrics
We evaluate along three dimensions: answer quality, refusal calibration, and latency. JudgeLM scores each response on a 1–10 scale using an LLM judge that assesses correctness, completeness, and coherence given the retrieved sources. Citation Quality (0–100) is a composite of faithfulness, source relevance, information synthesis, and source usage (evaluation prompts in Appendix E). QA F1 combines precision (fraction of generated answers that are correct) and recall (fraction of answerable queries that receive an answer) — capturing the tension between hallucination risk (low precision) and over‑caution (low recall). We additionally report the refusal rate and the fraction of abstentions that are true negatives.
Results
We compare FinRAG‑12B against Gemma 3 12B‑IT (no fine‑tuning) and GPT‑4.1 (API). BloombergGPT weights are unavailable; FinGPT has no citation evaluation.
5.1Answer quality and citation grounding
| Model | Jud.LM | Cit. Q | QA F1 | IDK % |
|---|---|---|---|---|
| Gemma 3 12B | 5.70 | 80.2 | 0.964 | 4.3 |
| GPT-4.1 | 5.72 | 70.8 | 0.900 | 20.2 |
| FinRAG-12B | 6.21 | 73.1 | 0.936 | 12.0 |
Table 4 — Main results on 258 banking QA examples (3 institutions). JudgeLM: answer quality (1–10); Cit. Q: citation quality (0–100); QA F1: precision–recall on answerable queries; IDK%: abstention rate.
FinRAG‑12B achieves the highest overall answer quality, with a JudgeLM score of 6.21, outperforming both GPT‑4.1 (+0.49) and the base Gemma 3 model (+0.51). It also improves citation grounding relative to GPT‑4.1, achieving a +2.3 point gain in citation quality (73.1 vs. 70.8). While the base model reports a higher citation quality score (80.2), this is driven by a broad citation strategy that references multiple retrieved passages regardless of relevance. In contrast, FinRAG‑12B is trained to cite selectively, grounding each claim in only the most relevant evidence. Importantly, FinRAG‑12B achieves a more balanced refusal behavior, with an abstention rate of 12.0%, compared to under‑refusal in the base model (4.3%) and over‑refusal in GPT‑4.1 (20.2%) — demonstrating effective calibration between answerability and abstention.
5.2Public benchmark results
FinanceBench (150 SEC-filing questions) — QA F1
Table 5 — Results on FinanceBench. FinRAG‑12B generalizes beyond proprietary data, achieving the highest F1 (0.284) with a 97.3% citation rate, outperforming both the base model (0.249) and GPT‑4.1 (0.238).
A 12B model trained for $1,800 on 143M tokens outperforms GPT‑4.1 on grounded banking QA — the recipe, not the scale, is what matters.
5.3Curriculum learning ablation
| Data strategy | Jud. | QA F1 | Cit.Q | IDK% | TN% |
|---|---|---|---|---|---|
| External only | 5.72 | 0.972 | 76.1 | 0.4 | 0 |
| Internal only | 5.62 | 0.913 | 69.2 | 17.4 | 53 |
| Combined (all) | 3.28 | 0.706 | 51.2 | 46.5 | 39 |
| Curriculum (staged) | 5.91 | 0.938 | 74.7 | 13.2 | 56 |
Table 6 — Data strategy ablation on 258 banking QA examples. TN%: share of refusals that are correct.
Table 6 isolates the contribution of each data source. Training on external data alone yields high QA F1 (0.972) but the model almost never refuses (0.4% IDK), making it unsafe for unanswerable queries. Internal data alone teaches refusal (17.4% IDK) but citation quality drops to 69.2. Mixing all data simultaneously collapses performance: JudgeLM falls to 3.28 and the model over‑refuses at 46.5%, with only 39% of refusals correct. The two‑stage curriculum resolves this conflict — external data first establishes citation conventions, then internal data calibrates refusal — achieving the best answer quality (5.91) and the highest true‑negative precision (56%) among fine‑tuned variants.
5.4Latency and cost
Figure 1 — Inference latency (seconds, lower is better)
RAG setting on a single RTX 6000 Ada GPU. FinRAG‑12B achieves 0.14s TTFT and 0.57s TTC, outperforming both proprietary APIs and the prior production model.
FinRAG‑12B achieves substantial latency improvements, operating 3–5× faster than GPT‑4.1 across both metrics. Compared to the prior production model (FinRAG‑v3, based on Mistral‑7B‑Instruct), FinRAG‑12B reduces TTC by 3.2× while maintaining comparable TTFT. In addition to latency gains, FinRAG‑12B offers significant cost advantages: running on a single RTX 6000 Ada GPU, inference costs approximately $0.001 per query at a throughput of 10K queries per day. In contrast, GPT‑4.1 API pricing ranges from $0.02 to $0.05 per query, making it 20–50× more expensive.
5.5Production impact
While the preceding sections evaluate performance on curated benchmarks, production deployment provides the ultimate test. We analyze 3,297 randomly sampled user queries collected over seven months (May–December 2025) from a large U.S. credit union serving millions of retail banking customers, comparing a legacy production model against FinRAG‑12B. Both models operate within the same RAG pipeline and share an identical knowledge base, ensuring that observed differences are attributable to the LLM rather than retrieval or data changes.
| Metric | Old (n=1,044) | FinRAG (n=2,253) | Δ |
|---|---|---|---|
| Resolution | 77.4% | 84.5% | +7.1pp *** |
| Unresolved | 20.7% | 13.7% | −7.0pp *** |
| Satisfaction (overall) | 59.5% | 62.9% | +3.4pp |
| — resolved queries | 65.0% | 66.7% | +1.7pp |
| — unresolved queries | 23.7% | 25.9% | +2.1pp |
Table 7 — Production metrics (3,297 queries, 7 months). Deltas in pp. *** p<0.001 (χ²). Satisfaction: p=0.26, 95% CI [−2.5, +9.3]pp.
FinRAG‑12B improves the resolution rate by 7.1 percentage points (χ²=24.4, p<0.001, Cramér’s V=0.09) — 7 additional queries per 100 being resolved without human escalation. The unresolved rate decreases by a similar margin (−7.0pp), indicating that the model converts previously unresolved queries into successful outcomes rather than redistributing errors across categories.
Overall user satisfaction increases from 59.5% to 62.9%, although this difference is not statistically significant (χ²=1.3, p=0.26). Decomposing satisfaction by outcome shows why: satisfaction remains nearly unchanged for both resolved (65.0% vs. 66.7%) and unresolved queries (23.7% vs. 25.9%), suggesting per‑response quality is largely comparable across models. Instead, the overall improvement is driven by a distributional shift: a larger proportion of queries are resolved, moving users into a higher‑satisfaction regime. In production RAG systems, increasing resolution rate rather than marginal improvements in response quality is the primary driver of user satisfaction.
Conclusion
We present FinRAG‑12B, a citation‑grounded LLM for banking RAG systems. We show that a data‑efficient training pipeline achieves the highest answer quality (JudgeLM 6.21) while improving citation grounding over GPT‑4.1 by 2.3 points. We further demonstrate that incorporating 22% unanswerable examples enables calibrated refusal behavior: FinRAG‑12B abstains on 12% of queries, balancing the under‑refusal of the base model (4.3%) and the over‑refusal of GPT‑4.1 (20.2%). These improvements translate directly to production. FinRAG‑12B increases query resolution by 7.1 percentage points (p<0.001), while achieving 3–5× lower latency and 20–50× lower cost than commercial APIs. Notably, gains in user satisfaction are driven by improved resolution rates rather than per‑response quality, highlighting resolution as the primary driver of user experience in RAG systems. Our results show that data quality, grounding, and training methodology are key to building reliable LLMs for regulated domains.
Limitations
Our evaluation focuses on banking RAG across three financial institutions (258 examples), and results may not generalize to other financial domains such as trading, insurance, or investment advisory. Additionally, the test set is skewed toward common retail banking queries, leaving rare edge cases underexplored.
The proprietary data used in Stage 2 training cannot be released due to privacy and regulatory constraints. However, Stage 1 relies entirely on an open‑source dataset (RAG‑v1), and we provide all training configurations — enabling researchers to reproduce our curriculum and adapt it to their own proprietary data.
Finally, our refusal evaluation captures explicit “I don’t know” responses and close variants, but may not fully account for hedged or partial uncertainty expressions (indirect or softened refusals), potentially underestimating nuanced abstention behavior.
Ethics statement
This work targets deployment in regulated financial services, where accuracy and reliability are critical. All proprietary training data was anonymized to remove personally identifiable information (PII) prior to use. To mitigate the risks of hallucination, the model is explicitly trained to abstain when sufficient evidence is not available, aligning with regulatory requirements for verifiable responses. As with all LLMs, there is a risk of inheriting biases present in training data. We monitor model behavior across user segments and continuously evaluate response quality to identify and mitigate potential disparities.
Appendix A — Training hyperparameters
| Parameter | Value |
|---|---|
| Base model | Gemma 3 12B-IT |
| LoRA rank (r) | 64 |
| LoRA alpha (α) | 256 |
| LoRA dropout | 0.05 |
| Target modules | q, k, v, o, gate, up, down |
| Learning rate | 2×10⁻⁵ |
| Optimizer | AdamW-8bit |
| Batch size | 4 |
| Gradient accumulation | 4 |
| Effective batch size | 16 |
| Max sequence length | 65,536 |
| Training steps | 1,402 |
| Final loss | 1.871 |
Table 8 — Training hyperparameters.
Appendix B — Quantization details
W4A16‑G128 quantization configuration:
- Weight quantization: 4‑bit, group size 128
- Activation precision: 16‑bit (FP16)
- Preprocessing: SmoothQuant, migration factor 0.5
- Calibration: 512 samples from training distribution
Size reduction: 24GB → 8.4GB (2.86× compression). Citation quality degrades marginally, retaining over 99% of full‑precision performance.
Appendix C — Data generation prompts
All synthetic data is generated by leveraging open‑weight models such as Llama‑3.3‑70B‑Instruct and Qwen‑3 32B.
C.1 Question generation. The system prompt shared across question generation strategies:
You are generating realistic financial customer questions. Customers ask questions to a financial assistant chatbot. Your questions must: - Sound like a real customer typed them into a chat or search box - Be answerable from the provided financial text - Match the specified style and constraints EXACTLY CRITICAL: Output ONLY the question text. No explanations, no numbering, no quotes.
Financial text: --- [FINANCIAL TEXT SEGMENT] --- Generate ONE question a [PERSONA] would ask about the information above. CONSTRAINTS: - Question style: [STYLE_DESCRIPTION] - Target length: approximately [WORD_COUNT] words - Formality: [FORMALITY] STYLE EXAMPLES for "[STYLE]": [FEW-SHOT EXAMPLES] Your question:
Dynamic fields are sampled per‑question: PERSONA from {retail customer, small business owner, financial advisor}; STYLE from the distribution in Table 2; WORD_COUNT from LogNormal(μ=2.1, σ=0.55); FORMALITY from {casual, neutral, formal}. Two additional strategies produce 23% of questions: contrastive pairs (two questions with different styles from the same passage, output as JSON) and template slot‑filling.
C.2 Single‑step baseline. The single‑step ablation baseline generates both question and answer in one API call, without the multi‑stage pipeline:
System: You are an expert financial Q&A generator.
Your task is to:
1. Read the numbered sources about financial topics
2. Generate a natural, standalone question based on
one of the sources
3. Provide a well-cited answer using the source
information
Important:
- The question should NOT refer to "the text",
"the passage", or "the document"
- The question should be specific and answerable
from the provided sources
- Include citations [1], [2], etc. when referencing
source content
- If the sources don't contain enough information,
respond with the unknown phrase
User: Given the following financial text sources,
generate a question-answer pair.
**Context Sources:**
[NUMBERED SOURCES]
**Instructions:**
1. Choose ONE source that contains interesting or
important financial information
2. Generate a natural question about that source's
content
3. Provide a concise answer citing the relevant
sources with [N] notation
4. If sources don't support a good question,
respond with: "I don't know."
**Output Format (JSON):**
{
"question": "Your generated question here",
"answer": "Your answer with [1], [2] citations"
}
Generate the question-answer pair:C.3 Answer generation with hint. The full pipeline generates answers with a hint identifying the gold source (Step 4 of the 5‑step process, §3.2.2). The no_hint ablation variant is identical except the hint line is omitted, requiring the model to identify the correct source independently.
You are a helpful assistant that provides answers to
questions based on the provided sources. You will be
given a set of sources and a question. Your task is
to generate an answer that accurately reflects the
information in the sources, while also including
citations for any specific details referenced.
Example:
Context: The following numbered sources are provided.
---------------------
Source [1]: Our savings account offers 2.5% APY for
balances over $5,000.
Source [2]: Withdrawals from savings accounts are
limited to 6 per month before fees apply.
---------------------
Question: How can I earn 2.5% APY on my savings?
Answer: You can earn 2.5% APY by maintaining a
balance of over $5,000 in your savings account[1],
but remember that withdrawals are limited to 6 per
month before fees apply[2].
Instructions:
1. Answer the Question using only the information
from the provided sources.
2. Include source citations using their
corresponding numbers (e.g., [1]).
3. Every answer must contain at least one citation.
4. Only cite a source if you directly reference it.
5. Keep the answer concise and focused.
6. Use bulleted lists for clarity if multiple points
are made.
7. If none of the sources are relevant, respond with
"I don't know." and stop.
Context: The following numbered sources are provided.
---------------------
{{context|numbered}}
---------------------
Hint: The correct answer should be found in the
source number {{hint}}.
Question: {{question}}
Answer:Appendix D — Inference prompt
The following prompt is used at both training and inference time. It uses Gemma 3’s native chat tokens:
<start_of_turn>user **Context:** --------------------- Source [1]: ... Source [2]: ... ... --------------------- **Instructions:** - Provide an answer based solely on the provided sources. - Use only the provided context to construct your answer. - Reference sources only when their information is explicitly used in your answer; include the corresponding source number as a citation (e.g., [1]). - Every answer must include at least one source citation. - If none of the provided sources are relevant, simply respond with 'I don't know.' and stop. **Question:** [QUESTION] **Answer:**<end_of_turn> <start_of_turn>model
Appendix E — Evaluation prompts
E.1 Citation Quality Judge (GPT‑4.1). System message: “You are an expert evaluator of Citations in RAG system responses. Return ONLY valid JSON that strictly follows the requested format.”
<rules>
- Evaluate the response solely based on the
provided sources.
- Focus on answer quality and the effective use
of citations.
</rules>
<format>
Return ONLY a JSON object that strictly adheres
to the schema below.
</format>
<structure>
---
QUESTION:
{question}
SYSTEM RESPONSE:
{response}
AVAILABLE SOURCES:
{sources}
1. Provide a brief evaluation summary.
2. Score the response (0-10) for:
- Source Relevance: How well the sources support
the answer.
- Answer Quality: Clarity, correctness, and
conciseness.
- Citation Usage: Appropriateness and accuracy of
citations.
- Information Synthesis: How well the information
is integrated.
- Faithfulness: Accuracy relative to the provided
sources.
3. List key strengths, weaknesses, and improvement
suggestions.
4. Provide an overall rating (0-10).
</structure>
Return ONLY a JSON object:
{ "scores": {...}, "analysis": {...},
"overall_rating": <0-10> }The five sub‑scores are averaged to produce the Citation Quality composite (0–100) reported in Tables 4 and 6.
E.2 Answer Quality Judge (JudgeLM 7B). JudgeLM uses a comparative evaluation format, scoring the model response (Assistant 2) against the gold reference (Assistant 1):
You are a helpful and precise assistant for
checking the quality of the answer.
[Question]
{question}
[The Start of Assistant 1's Answer]
{reference}
[The End of Assistant 1's Answer]
[The Start of Assistant 2's Answer]
{single answer}
[The End of Assistant 2's Answer]
[System]
We would like to request your feedback on the
performance of two AI assistants in response to the
user question displayed above. Please rate the
helpfulness, relevance, accuracy, level of details
of their responses. Each assistant receives an
overall score on a scale of 1 to 10, where a higher
score indicates better overall performance. Please
first output a single line containing only two
values indicating the scores for Assistant 1 and 2,
respectively. The two scores are separated by a
space. In the subsequent line, please provide a
comprehensive explanation of your evaluation,
avoiding any potential bias and ensuring that the
order in which the responses were presented does
not affect your judgment.
[Response]
10Acknowledgements
The authors thank David Gondek, Nithin Govindugari, Olivier Allauzen, Joshua Baptiste, DJ More, Binod Gyawali, Bob Stewart, George Trad, Jon Phillips, Priti Khanna, Victoria Povolotsky, Jared Kim, Samar Batra, Romain Lebolloch, Gerard Benavides, Martin Lopez, Allison Ray, Yeonju Lee‑Sikka, Max Schwartz, Tino Galizio, Rob Kassel, Jon Hopewell, Justin Arnoldi, Byron Wolff, Faisal Nematt, Robert Dugdale, Ben Ortega, Anand Venkatachalam, Mehul Pipalia, Raju Singh, Biswajit Singha, Michael Wasserfuhr, Helen Rosen and everyone on the team for their contributions to FinRAG‑12B and the broader project at Kasisto. We also thank the anonymous ACL reviewers and the meta‑reviewer for their valuable feedback. This work was conducted under the executive leadership of Zor Gorelov, Sasha Caskey, Joshua Schechter, and Lance Berks.
Full reference list (27 works) in the arXiv version: arxiv.org/abs/2605.05482