ICCV 2025 Workshop on Human-inspired Computer Vision (Oral Presentation)

Human Vision Constrained Super-Resolution

Paper (IEEE) arXiv Code
Human Vision Constrained Super-Resolution Teaser
Our perceptually driven framework dynamically and locally maps visual contrast sensitivity thresholds to super-resolution quality bands.

Abstract

Modern deep-learning super-resolution (SR) techniques process images and videos independently of the underlying content and viewing conditions. However, the sensitivity of the human visual system (HVS) to image details changes depending on the underlying image characteristics, such as spatial frequency, luminance, color, contrast, or motion; as well viewing condition aspects such as ambient lighting and distance to the display. This observation suggests that computational resources spent on up-sampling images/videos may be wasted whenever a viewer cannot resolve the synthesized details i.e the resolution of details exceeds the resolving capability of human vision. Motivated by this observation, we propose a human vision inspired and architecture-agnostic approach for controlling SR techniques to deliver visually optimal results while limiting computational complexity. Its core is an explicit Human Visual Processing Framework (HVPF) that dynamically and locally guides SR methods according to human sensitivity to specific image details and viewing conditions. We demonstrate the application of our framework in combination with network branching to improve the computational efficiency of SR methods. Quantitative and qualitative evaluations, including user studies, demonstrate the effectiveness of our approach in reducing FLOPS by factors of 2× and greater, without sacrificing perceived quality.

Model Attenuation Curves

We use attenuation curves to characterize the reconstruction capability of different super-resolution models. By analyzing the radially averaged 2D Fourier transform of the up-scaled outputs against ground-truth images, we express the reconstruction power of each model or branch as a spatial-frequency response.

Attenuation curves for EDSR models of varying depth
Network Depth/Channels (EDSR): Attenuation curves computed for independently trained EDSR networks with varying channel counts (256, 128, 64, 16, 8) per layer.
Attenuation curves for Transformer-based models
Transformer Models (SwinIR): Attenuation curves computed for SwinIR models of different sizes (SwinIR-L, SwinIR-M, SwinIR-S) alongside the Bicubic interpolation baseline.

Method Predictions

Method predictions and quality maps
Visual results of our method compared to the original networks. On the right, we can observe the maps produced by our perceptual model.

VR Application

VR eccentricity and quality maps under foveated rendering
Our model predictions based on gaze position with X4 super-resolution. In the first column, we have the original image and the corresponding quality map. In the other columns we have on top the eccentricity map expressed in degrees and bottom we have the corresponding quality map.

Video Super-Resolution (Temporal CSF)

To extend our Human Visual Processing Framework (HVPF) to video content, we model temporal contrast sensitivity by estimating the optical flow between frames. Since human visual sensitivity to spatial details decreases with motion velocity, we dynamically scale the spatial quality threshold in the foveation and contrast-sensitivity calculations. This allows for significant computational savings in high-motion regions without any human-perceptible degradation.

Subjective study preference results for video network branching
The result of our video subjective quality study (14 participants) evaluating network branching on 7 natural videos from the Inter4K dataset. The 50% preference indicates that our HVPF-guided video SR is visually indistinguishable from full network execution, while saving 51.3% of the computational cost on average.

Quantitative Results (FLOPS Reduction)

We evaluate the computational efficiency of our method using average Floating Point Operations (FLOPS) as a machine-independent metric. By deploying simpler networks or shallower branches in visually insensitive regions, our perceptual model achieves massive speedups.

Predictor Model Overhead: The HVPF predictor model is designed to be extremely lightweight, introducing negligible overhead. Running the predictor costs only 39 KFLOPs for a \(10 \times 10\) patch and 477 KFLOPs for a \(35 \times 35\) patch. In real-time VR/AR applications, a full shader implementation of HVPF runs in under 1 ms at 2K resolution on standalone headsets like the Meta Quest 2.

Application & Model Dataset Avg. FLOPS Savings Subjective Study Size Perceived Quality
VDSR Branching (Images) Set5 & Set14 (24 natural scenes) 30% - 80% 15 participants Indistinguishable (~50% pref)
EDSR Channel Depth (Images) Set5 & Set14 (24 natural scenes) 76.4% 9 participants Indistinguishable (~50% pref)
VDSR Branching (Videos) Inter4K (7 natural videos) 51.3% 14 participants Indistinguishable (~50% pref)

Subjective Quality Studies (Images)

User study network branching application
Network Branching (VDSR): The result of our user study (15 subjects) for the network branching application with 24 natural images.
Subjective study network channel depth application
Channel Depth Selection (EDSR): The result of our subjective study (9 participants) for the network channel depth selection application.

Additional Qualitative Results

Additional model predictions and quality maps
Further qualitative maps generated by our method, demonstrating selective deployment of higher capacity networks in regions of greater detail and contrast.

Citation

@inproceedings{karpenko2025human, author = {Karpenko, Volodymyr and Tariq, Taimoor and Condor, Jorge and Didyk, Piotr}, title = {Human Vision Constrained Super-Resolution}, booktitle = {2025 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)}, year = {2025}, pages = {4782-4790}, doi = {10.1109/ICCVW69036.2025.00498}, publisher = {IEEE}, address = {Honolulu, HI, USA}, url = {https://ieeexplore.ieee.org/document/11375594} }