Despite recent advances in diffusion models, AI-generated images still often contain visual artifacts that compromise realism. Although more thorough pre-training and bigger models might reduce artifacts, there is no assurance that they can be completely eliminated, making artifact mitigation a highly crucial area of study.
Previous artifact-aware methodologies depend on human-labeled artifact datasets, which are costly and difficult to scale. We propose ArtiAgent, an agentic framework that efficiently creates pairs of real and artifact-injected images without human intervention. It comprises three agents: a perception agent that recognizes and grounds entities and subentities from real images; a synthesis agent that introduces artifacts via novel patch-wise embedding manipulation within a diffusion transformer; and a curation agent that filters synthesized artifacts and generates local and global explanations for each instance.
Using ArtiAgent, we synthesize 100K images with rich artifact annotations and demonstrate both efficacy and versatility across diverse applications, including fine-tuning open-source VLMs that consistently outperform proprietary systems (GPT-5, Gemini-2.5-Pro) on artifact detection, localization, and explanation.
ArtiAgent pipeline. (a) The perception agent detects entities and subentities using Grounded-SAM. (b) The synthesis agent injects artifacts via patch mapping tools and the inversion-injection paradigm. (c) The curation agent filters low-quality results and generates textual explanations.
Perception Agent
Decomposes an input image into a hierarchical vocabulary of entities (e.g., dog) and subentities (e.g., nose, leg) using out-of-the-box VLMs, then grounds them with Grounded-SAM for precise spatial localization.
Synthesis Agent
Uses grounding to inject artifacts via a toolbox of four tools (add, remove, distort, fuse) and a novel inversion-injection module that manipulates patch attention in diffusion transformers to generate realistic structural artifacts.
Curation Agent
Performs data filtering (LPIPS-based for distortions, VLM-based for structural artifacts) and generates local and global textual explanations for each artifact, producing training-ready supervision.
Visualization of each target-reference patch mapping and its resulting artifact.
The toolbox provides four patch-mapping tools:
Inversion-injection module: the right arm from the reference patches is added to the target patches below it.
We extend the inversion-restoration paradigm from image editing into an inversion-injection module that manipulates self-attention in diffusion transformers (FLUX.1-dev via FireFlow).
Two complementary mechanisms operate on the target patch region:
Their combination produces localized, realistic structural artifacts while keeping the background fully consistent with the original image.
We introduce ArtiBench, a benchmark of 1K images generated by five modern diffusion models (SD3.5, FLUX, Qwen-Image, Nano-Banana), annotated by 12 human labelers with binary labels, bounding boxes, and textual descriptions per artifact region.
Unlike prior benchmarks, ArtiBench covers all three tasks (detection, localization, explanation) and focuses on recent models where structural artifacts are subtle and diverse, making it a significantly harder and more representative.
| Benchmark | Samples | Det. | Loc. | Exp. |
|---|---|---|---|---|
| RichHF | 955 | ✓ | ||
| LOKI | 229 | ✓ | ✓ | |
| SynthScars | 1K | ✓ | ✓ | |
| ArtiBench | 1K | ✓ | ✓ | ✓ |
Artifact understanding performance across detection, localization, and explanation.
Open-source VLMs fine-tuned with ArtiAgent-generated data consistently outperform GPT-5 and Gemini-2.5-Pro on artifact detection, localization, and explanation across ArtiBench and three existing benchmarks (RichHF, LOKI, SynthScars).
Scaling effect with Qwen2.5-VL-7B; metrics averaged across all benchmarks per task.
Performance grows consistently as training data from ArtiAgent scales up:
ArtiAgent's paired data design (clean vs. artifact-injected) provides ideal supervision for a CLIP-based reward model. Combined with test-time scaling on FLUX-schnell:
Reward-guided generation: the artifact-free preference score increases across rounds.
The ArtiAgent-trained VLM guides FLUX inpainting to fix artifacts.
An artifact-aware VLM (Qwen2.5-VL-7B fine-tuned on ArtiAgent data) powers an iterative inpainting pipeline:
The pipeline reliably locates and corrects structural artifacts with natural, structurally consistent content.
@inproceedings{park2026artiagent,
title = {See and Fix the Flaws: Enabling VLMs and Diffusion Models
to Comprehend Visual Artifacts via Agentic Data Synthesis},
author = {Park, Jaehyun and Ahn, Minyoung and Kim, Minkyu and
Lee, Jonghyun and Lee, Jae-Gil and Park, Dongmin},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR)},
year = {2026},
}