Interactive GAN Facial Composite
Iterative, user-in-the-loop system: a pretrained GAN samples faces around a seed;
the user rates similarity to a person in mind; ratings are used to update the seed in latent space,
converging toward the target appearance. Built at the MISE Research Camp and
submitted to the International Science & Engineering Fair (ISEF).
* This research was done under the guidance and supervision of Karan Grewal.
Overview
The system starts from a latent seed z. Each iteration, we sample noisy variants around z and generate candidate faces. The user rates each candidate for resemblance. Those ratings become supervision to solve for a direction in latent space that increases resemblance, and we update z accordingly. Repeating this loop steers the GAN toward an image that better matches the person the user has in mind—an approach designed for facial composite creation.
Key Features
- 🧠 User-guided optimization: Ratings drive latent updates toward the target.
- 🎲 Seed + noise sampling: Diverse candidates around the current best.
- ⚙️ Latent update: Rating-weighted solve for direction; step-size scheduling.
Potential Impact
Assists witnesses and investigators by gradually converging on a useful facial composite without requiring drawing skills—interactive, fast, and repeatable.
My Contributions
- Prototyped the end-to-end loop: sampling, rating UI, latent updates, and convergence logic.
- Implemented rating aggregation and a robust update step (regularized, step-size control).
- Built iteration dashboards (seed preview, candidate grid, rating summaries).
- Prepared research materials for MISE and ISEF submission (method, results, limitations).
Tech Notes
Workflow
-
1
Seed & Sampling
Pick seed z; sample z + ε (noise) to get a batch of candidate faces.
-
2
User Ratings
User scores resemblance for each candidate (e.g., 0–5). Ratings form weak labels.
-
3
Latent Update
Solve for a direction in latent space correlated with higher ratings (regularized); update z with a scheduled step size.
-
4
Repeat & Converge
Iterate until resemblance plateaus; export best image and iteration history.
Future Work (In Progress)
- 🧩 Train a custom GAN: Tailored dataset & latent prior to improve identity control.
- 🗣️ LLM pre-prompting: Users describe high-level traits; system initializes seed and sampling to match those traits so the first iteration starts closer to target.
- 🛡️ Safeguards & bias checks: Monitor for mode collapse, artifacts, and demographic biases.
Screenshots
