Digital Twin Evaluation: GPT-4.1 vs Gemini 2.5 Flash

Executive readout

GPT-4.1 behaved like the more expansive portfolio guide. It was longer on 44 of 58 questions, used more follow-up invitations, and more often turned retrieved facts into a fuller narrative answer.

Gemini 2.5 Flash behaved like the more concise answerer. It was shorter on most questions, much cheaper, and often serviceable on closed factual prompts. Its main weakness is not that it missed the fact; it more often under-developed the answer relative to the Digital Twin's intended voice and purpose.

The retrieval layer is not the explanation for the differences. Top-k and question set were held constant, and chunk similarity stats are essentially identical. The variation is in how each model used the same retrieved context.

Methodology

Two runs of the same 58-question evaluation set, held constant on retrieval (Chroma vector store, top-k = 4), system prompt, and temperature (0.6). Only the model varied. Metrics fall into three buckets: behavioral counts, lexical overlap, and cost.

The 15-question portfolio-critical subset

The full 58-question set is too large to score by hand after every change. These 15 questions are my first-pass quality check for whether the Twin can make a strong, grounded impression across career, project, and technical conversations.

1. What problems does Barbara solve?
2. Walk me through a project.
3. How was this Digital Twin built?
4. How does your cognitive science background shape your AI work?
5. What kinds of problems get you excited?
6. What are your biggest strengths?
7. How do you approach problem-solving?
8. How do you handle ambiguity?
9. How do you document your projects?
10. How do you approach ML model development?
11. Tell me about Resume Explorer.
12. Tell me about Concept Cartographer.
13. Tell me about the Beehive Photo Metadata Tracker.
14. What role are you looking for next?
15. Why should someone hire you?

Scoring rubric

Six dimensions, each scored from 1 (weak) to 5 (strong). Total possible score per response: 30. This rubric structure is the scaffold for the next round of human-scored evaluation. The dimensions are deliberately separable so a response can be accurate but bland, or specific but ungrounded, and the rubric will say so.

Run-level quantitative summary

Token counts are not directly comparable across providers because each tokenizer is different. Gemini reported more completion tokens despite shorter word counts. Word count and estimated dollar cost are the more interpretable comparisons.

Breakdown by question type

Higher-level partition of the 58 questions into three types: bounded-open (most of the set), closed-fact (biographical, credentials), and open-persona (the playful one).

Breakdown by intent

Finer-grained partition. Sorted by N descending; intents with only one question (the bottom rows) carry no statistical weight but are included for completeness.

Project and concept mention counts

How often each model named one of the portfolio projects or concepts across the full 58-question run.

Sample responses, lowest overlap

Paired snippets where the two models diverged the most. Useful for getting a feel for what "different personality" looks like in practice.

Limitations and next steps

What this evaluation can and cannot claim

This is a model behavior comparison built from text-level diagnostics. It is not a human-scored quality bakeoff. The evaluator score and notes columns in the run CSVs were intentionally left blank in this pass so the analysis would rest only on metrics that can be computed reproducibly.

The must-cover lexical coverage measure is a smoke test for grounding, not a correctness score. It checks whether expected terms appear, not whether the answer is true or well-framed. Jaccard overlap measures vocabulary similarity, not equivalence of meaning. Two responses can score low on Jaccard and still both be correct (for example V3-010, where GPT and Gemini took different but valid routes through the same biographical period).

Longer is not the same as better. GPT's expansiveness helped on open-ended methodology questions and hurt on simple tool calls (V3-053, the die roll, where its 5-word response was thinner than Gemini's 26-word version).

What comes next

Three concrete next steps for the evaluation infrastructure:

1. Human scoring on the 15-question subset. Apply the six-dimension rubric to both runs and report the scores. This is where qualitative quality lives.
2. Response-mode rules by question type. Closed factual questions should stay short. Project walkthroughs should carry a narrative arc. The current prompt does not differentiate.
3. Regression sweeps after knowledge-base edits. Use Gemini for these because the cost is roughly a third and the hallucination detection signal is comparable.

Practical recommendation

For the public-facing Digital Twin at twin.barbhs.com, the default is GPT-4.1. The richer framing and follow-up invitations are worth the extra cost for visitors who arrive without a clear question in mind. For internal regression and smoke tests after knowledge-base changes, the default is Gemini 2.5 Flash. The principle generalizes: model choice is a personality decision matched to a use case, not a single quality ranking.