README.md

Benchmarking PySceneDetect

This page contains the results of benchmarking PySceneDetect's detection accuracy againts public shot-boundary-detection datasets. Scoring follows the TRECVID-SBD convention (greedy 1-to-1 nearest-neighbor matching, with a configurable frame tolerance for hard cuts; point-in-interval matching for fade transitions; mean absolute frame offset on matched events) so numbers are comparable to published SBD results.

Supported datasets:

• BBC Planet Earth: 11 long-form broadcast clips; hard cuts only
• AutoShot: Short-form web clips; hard cuts only
• ClipShots: Short-form web clips; hard cuts and typed gradual transitions (fades/dissolves)

Usage

# Single detector x single dataset:
python -m benchmark --detector detect-content --dataset BBC

Pass --help for --dataset-root, --backend, --tolerance, and --out options.

Parameter sweeps

python -m benchmark.sweep runs a grid over detector parameters and reports the top cells by F1 plus the Pareto front across tolerances. One decode is shared by up to --workers parallel detectors via an internal fan-out wrapper, so the cost scales with ceil(cells / workers) decodes per video rather than cells decodes.

python -m benchmark.sweep \
  --detector detect-content --dataset BBC \
  --params "threshold=15:35:1;min_scene_len=0.0:1.0:0.1" \
  --tolerance 0,1 --workers 16 \
  --out sweep-content-bbc.json

--params: assignments joined by ;. Each is either key=v1,v2,v3 (enumerated values) or key=start:stop:step (numeric range, inclusive when stop lands on a step). Omitted keys use the detector's default.

Time-valued kwargs (min_scene_len, etc.) accept TimecodeLike - integers are frames, floats are seconds, and strings like "0.1s" / "00:00:00.500" also work. Prefer floats so the same sweep is comparable across datasets with different framerates. Use --quick N to limit to the first N samples for iteration; published numbers should always come from the full corpus.

Dataset Download

BBC

# annotations
wget -O BBC/fixed.zip https://zenodo.org/records/14873790/files/fixed.zip
unzip BBC/fixed.zip -d BBC
rm -rf BBC/fixed.zip

# videos
wget -O BBC/videos.zip https://zenodo.org/records/14873790/files/videos.zip
unzip BBC/videos.zip -d BBC
rm -rf BBC/videos.zip

AutoShot

Download AutoShot_test.tar.gz from Google Drive.

tar -zxvf AutoShot_test.tar.gz
rm AutoShot_test.tar.gz

ClipShots

ClipShots is gated behind a dataset request form; direct wget-style download links are not published. See the download instructions to obtain the annotations and videos. The expected on-disk layout is:

ClipShots/
  annotations/{train,test,only_gradual}.json
  video_lists/{train,test,only_gradual}.txt
  videos/*.mp4

The loader defaults to the test split (500 videos). The full corpus is ~46 GB.

Set --dataset-root /path/to/datasets to override. The default dataset location assumes they are all placed in the benchmark folder (e.g. benchmark/BBC, benchmark/AutoShot, benchmark/ClipShots).

Results (defaults)

Generated by scripts/benchmark_defaults.sh at tolerance=0 (frame-exact matching). Elapsed is mean wall-clock seconds per video.

BBC

Detector	Recall	Precision	F1	Mean s/video
AdaptiveDetector	87.12	96.55	91.59	36.12
ContentDetector	84.70	88.77	86.69	37.02
HashDetector	92.30	75.56	83.10	25.51
HistogramDetector	89.84	72.03	79.96	22.29
ThresholdDetector	0.06	0.70	0.11	16.05

AutoShot

Detector	Recall	Precision	F1	Mean s/video
AdaptiveDetector	70.59	77.46	73.86	3.52
ContentDetector	63.49	76.19	69.26	4.80
HashDetector	56.48	76.11	64.84	4.14
HistogramDetector	63.27	53.23	57.82	3.76
ThresholdDetector	0.75	38.64	1.47	3.28

ClipShots (hard cuts)

Detector	Recall	Precision	F1	Mean s/video
AdaptiveDetector	85.97	41.25	55.75	1.81
ContentDetector	81.93	42.36	55.84	2.52
HashDetector	81.34	30.14	43.98	1.04
HistogramDetector	72.20	11.47	19.80	0.71
ThresholdDetector	0.08	0.58	0.14	0.64

ClipShots (fades)

Detector	Recall	Precision	F1
AdaptiveDetector	13.65	98.12	23.96
ContentDetector	26.03	98.04	41.14
HashDetector	18.77	94.53	31.33
HistogramDetector	69.67	81.99	75.33
ThresholdDetector	5.69	99.24	10.77

Parameter sweep results

The tables above use each detector's v0.7 defaults. A grid sweep over the key parameters scored by hard-cut F1 at 1-frame tolerance, averaged across BBC / AutoShot / ClipShots gives the best single parameter set for this corpus mix:

Detector	Best mean F1	Best params	v0.7 default
ContentDetector	73.4	threshold=31, min_scene_len=0.6s	threshold=27
AdaptiveDetector	76.3	adaptive_threshold=3.5, window_width=3, min_scene_len=0.6s	adaptive_threshold=3.0, window_width=2
HashDetector	69.8	threshold=0.35, size=8	threshold=0.395, size=16
HistogramDetector	66.3	threshold=0.20, bins=128	threshold=0.05, bins=256
ThresholdDetector	--	detects fades, not hard cuts (validation only)	threshold=12

Full per-dataset breakdowns are in SWEEP_REPORT.md, and can be generated with python -m benchmark.report_sweep. The full grids (all detectors and datasets) are driven by scripts/benchmark_sweep.sh.

Citations

BBC

@InProceedings{bbc_dataset,
  author    = {Lorenzo Baraldi and Costantino Grana and Rita Cucchiara},
  title     = {A Deep Siamese Network for Scene Detection in Broadcast Videos},
  booktitle = {Proceedings of the 23rd ACM International Conference on Multimedia},
  year      = {2015},
}

AutoShot

@InProceedings{autoshot_dataset,
  author    = {Wentao Zhu and Yufang Huang and Xiufeng Xie and Wenxian Liu and Jincan Deng and Debing Zhang and Zhangyang Wang and Ji Liu},
  title     = {AutoShot: A Short Video Dataset and State-of-the-Art Shot Boundary Detection},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  year      = {2023},
}

ClipShots

@InProceedings{clipshots_dataset,
  author    = {Shitao Tang and Litong Feng and Zhanghui Kuang and Yimin Chen and Wei Zhang},
  title     = {Fast Video Shot Transition Localization with Deep Structured Models},
  booktitle = {Asian Conference on Computer Vision (ACCV)},
  year      = {2018},
}