SCOPE-Bench v1.0
Short-video recommendation benchmark

Content Depth Matters
in Short-Video Recommendation

Rethinking the attention economy through a benchmark that measures not only what people click, but how deeply recommended content develops information.

Liwei Deng1·Jing Jiang1·Zhiwei Li1·Yang Wang2·Guodong Long1

1 Australian Artificial Intelligence Institute, University of Technology Sydney

2 Evidence and Research, Department of Health, Disability and Ageing

University of Technology Sydney
⌘ Code▦ Dataset▶ Get started▤ Paper
150K+short videos in the full benchmark collection
1M+interactions in the full source dataset
13representative paper-facing baselines
54model implementations across five settings

SCOPE-Bench is open to researchers working on recommendation beyond engagement-only objectives.

Abstract

Driven by the attention economy, short-video Recommender Systems (RSs) are primarily optimized to maximize user engagement by promoting videos that capture attention within seconds. These systems inherently favor shallow-content videos that are effective at attracting immediate attention. However, growing evidence suggests that prolonged exposure to such content may negatively affect users' cognitive engagement and mental well-being, raising concerns about the long-term societal impact of the short-video platform.

To tackle this challenge, this paper introduces a new metric, the Content Depth Score (CDS), to quantify the content depth of short videos. CDS measures the extent to which a video is expected to stimulate higher-order cognitive processes, using a seven-level scale grounded in established theories of cognitive psychology and learning. As an initial step toward this vision, we present SCOPE-Bench, the first benchmark for content-depth evaluation in short-video recommendation. Built upon a large-scale open-source short-video dataset, SCOPE-Bench provides CDS annotations for 150K videos, enabling systematic evaluation of RSs from a cognitive-content perspective.

Leveraging SCOPE-Bench, we evaluate 13 representative RSs and reveal a consistent preference for shallow-content videos. Moreover, we find that these algorithms recommending cognitively deep content are only marginally better than random selection, highlighting a previously overlooked limitation of existing recommendation objectives.

Three ways in

SCOPE-Bench is a paper project, a runnable benchmark and an extensible model library. Pick the path that matches what you need.

For buildersRun SCOPE-Bench

Install the benchmark and evaluate one model with unified relevance and depth metrics.

→ Get startedFor understandingExplore the benchmark

See how CDS and LCDS connect video-level content analysis to top-k recommendation.

→ Read the overviewFor comparisonBrowse the models

Centralized, multimodal, federated and sequential implementations in one registry.

→ View model coverage
①

Overview of the scalable CDS annotation workflow.

At the item level, SCOPE-Bench assesses the content depth of individual short videos from the user perspective.

SCOPE-Bench item-level Content Depth Score annotation workflow
Item-level CDS annotation workflow. CDS captures informational and reasoning depth; factual correctness and safety remain independent dimensions.
②

Overview of the content-depth-aware RS evaluation framework

At the recommendation-list level, it evaluates the content depth of recommendation lists from the platform perspective

SCOPE-Bench list-level cognitive-aware recommender evaluation workflow
Recommendation-list-level cognitive-aware evaluation, reporting Recall, NDCG, Precision, A-LCDS and E-LCDS together.

One benchmark, five settings

The repository separates centralized and federated learning by input modality, alongside sequential recommendation, under one evaluation contract.

01

Centralized ID

ID-based recommendation methods trained from centralized user–item interactions.

02

Centralized Multimodal

Methods trained centrally with user–item interactions and multimodal content features.

03

Federated ID

ID-based recommendation methods optimized across distributed clients.

04

Federated Multimodal

Federated methods combining distributed interactions with multimodal content features.

05

Sequential

Sequence-aware recommendation methods that model ordered user interaction histories.

③

From clone to benchmark

Select a model and dataset, then receive engagement and content-depth metrics in the same result files.

Read the complete tutorial →
QUICK STARTv1.0
# install
$ git clone https://github.com/LiweiDengDavid/SCOPE-Bench.git
$ cd SCOPE-Bench
$ pip install -e ".[torch,multimodal,hpo]"

# train and evaluate
$ python main.py \\
    --model LightGCN \\
    --dataset ShortVideoSampled \\
    --gpu_id 0

A living research project

More work is on the way.

SCOPE-Bench will continue to grow with new research, artifacts and evaluation capabilities. Follow the repository for future updates.

Watch on GitHub

Cite this work

@misc{deng2026contentdepthmattersshortvideo,
      title={Content Depth Matters in Short-Video Recommendation: Rethinking the Attention Economy},
      author={Liwei Deng and Jing Jiang and Zhiwei Li and Yang Wang and Guodong Long},
      year={2026},
      eprint={2608.13990},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2608.13990},
}