Evaluating the Robustness of LLM-Generated Tests via Mutation Analysis

This thesis addresses whether LLMs can generate tests that catch bugs, not merely achieve high code coverage. The work evaluates LLM-generated tests against mutated Java programs using coverage deltas and test-failure behavior as evidence of mutant detection.

Evaluates whether LLM-generated test suites detect injected mutants instead of only improving line coverage.
Compares baseline generation from original code with targeted generation from mutated-code context.
Benchmarks proprietary and open models including GPT, Claude, Gemini, Llama, Qwen, and DeepSeek variants.
Uses Docker-isolated Python and Java environments with file-based triggers for repeatable experiment orchestration.

The system uses a Python agent container for LLM communication and orchestration, paired with a Java/Maven container that compiles subject code, runs generated JUnit tests, and produces coverage data.

Experiment outputs are organized by model and mutant so P0 baseline tests and P1 mutant-context tests can be compared across mutation operators and method types.