Felix Brener

Filmstock Responsiveness

A full-factorial eval of how image models respond to filmstock, lighting and scan language: 288 images, 195 features, a 31-page write-up.

Role
Sole author (design, analysis, paper)
Year
May & September 2026
Status
Paper + replication
Outcome
ChatGPT Images 2.5: 24% less responsive than 2.0 to filmstock, 14% to lighting, 37% to scan language
felix-ab/filmstock-responsiveness-benchmarkA contact sheet of sixteen frames of a woman in a dark coat outside a corner store, labelled by model version, filmstock, lighting and processing.

Finding

Drop in prompt response, ChatGPT Images 2.0 to 2.5Holm-adjusted p < .001
  • Filmstock24% less
  • Lighting14% less
  • Development and scan37% less

filmstock-responsiveness-benchmark, README

Problem

Photographic prompts lean on film vocabulary. How sensitive are image models to it, and by how much?

What I shipped

Designed a full-factorial eval of how image models respond to filmstock, lighting and scan language: 288 images and 195 features per image, analysed with PCA, MANOVA and PERMANOVA.

A September replication found ChatGPT Images 2.5 24% less responsive than Images 2.0 to filmstock, 14% to lighting and 37% to scan language (Holm-adjusted p < .001). Repeats of the same prompt were 41.5% closer together.

Generated and normalised the image set, built the feature-extraction and stats pipeline, and wrote it up in a 31-page paper. Code is MIT; data and writing are CC BY 4.0.

Architecture

  1. Full-factorial prompt grid
  2. 288 normalised images
  3. 195 features per image
  4. PCA · MANOVA · PERMANOVA
  5. Paper + replication

Tradeoffs

  1. A full-factorial design with 432 blocked prompt-response contrasts, so each vocabulary's effect is isolated.
  2. Replicate on the next model (ChatGPT Images 2.0 vs 2.5) instead of stopping at one result.
  3. Scope the claim: three vocabularies measured, no claim of a general decline in quality.

Evals

  • 288normalised images in the first studyfilmstock-responsiveness-benchmark, README
  • 195features extracted per imagefilmstock-responsiveness-benchmark, README
  • 31pages in the replication paperstudies/2026-09-images-2-5

Figures

felix-ab/filmstock-responsiveness-benchmarkFour bar charts of standardised feature distance for filmstock, lighting and scan wording, comparing Images 2.0 and Images 2.5 across primary, colour, contrast and texture, and structure features.
fig. 1 Prompt-response distances by feature family, Images 2.0 against 2.5.
felix-ab/filmstock-responsiveness-benchmarkA PCA scatter plot of 195 standardised image features, with confidence ellipses for ChatGPT Image 2 and Grok Imagine that overlap heavily.
fig. 2 PCA of 195 standardised features by model, first study.
felix-ab/filmstock-responsiveness-benchmarkTwo bar charts: the ratio of factor distance to repeat distance by vocabulary, and mean same-prompt repeat distance for Images 2.0 and 2.5.
fig. 3 Repeat variation: how much results differ when the same prompt is run again.

Stack

Python, PCA, MANOVA, PERMANOVA, ComfyUI