Inconsistent input and disparity in Kitti2015Stereo (and others)

[tpet]

🐛 Describe the bug

In the official archive for Kitti2015Stereo, there are twice as input images for each camera than there are disparities.
Matching inputs and disparities now seems to depend solely on sorting the file names in respective directories which is not enough and inconsistent input image and disparities are served as a result.
The dataset also reports size 400 but iteration ends after 200 examples due to missing disparities.

A workaround is to get rid of the extra images in extracted archive. This would be good to add into docs.
A better solution would be to use the file names to get matching counterparts, e.g., to use the file names intersection from the relevant directories as examples to serve.

It affects any stereo dataset if sorting the files is not enough.

How to check/reproduce:

import matplotlib.pyplot as plt
import torch
import torchvision
import torchvision.transforms.functional as F

# Using https://s3.eu-central-1.amazonaws.com/avg-kitti/data_scene_flow.zip
dataset = torchvision.datasets.Kitti2015Stereo('data')
print(dataset)

for i, (left, right, disparity, _) in enumerate(dataset):
    print(f'{i} / {len(dataset)}')
    left = F.to_tensor(left).mean(dim=0, keepdim=True)
    right = F.to_tensor(right).mean(dim=0, keepdim=True)
    disparity = torch.as_tensor(disparity)
    grid = torchvision.utils.make_grid([left, right, disparity])
    plt.imshow(grid.numpy().transpose((1, 2, 0)))

Versions

Collecting environment information...
PyTorch version: 1.13.0+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] (64-bit runtime)
Python platform: Linux-5.15.0-58-generic-x86_64-with-glibc2.35
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.4
[pip3] torch==1.13.0
[pip3] torchvision==0.14.0
[conda] Could not collect

cc @pmeier

[pmeier]

Can confirm that this is happening for split="train". For split="test" however this is not an issue, since we don't depend on the disparities there:

vision/torchvision/datasets/_stereo_matching.py

Lines 341 to 346 in dabb6d5

	if split == "train":
	left_disparity_pattern = str(root / "disp_occ_0" / "*.png")
	right_disparity_pattern = str(root / "disp_occ_1" / "*.png")
	self._disparities = self._scan_pairs(left_disparity_pattern, right_disparity_pattern)
	else:
	self._disparities = list((None, None) for _ in self._images)

We definitely need to fix this for split="train" though. Naming scheme regex for the images is \d{6}_1[0,1].png. For example, we have 000194_10.png and 000194_11.png. For the disparities it is \d{6}_10.png. So we can get away by just injecting _10 the file name globs:

vision/torchvision/datasets/_stereo_matching.py

Lines 337 to 343 in dabb6d5

	left_img_pattern = str(root / "image_2" / "*.png")
	right_img_pattern = str(root / "image_3" / "*.png")
	self._images = self._scan_pairs(left_img_pattern, right_img_pattern)
	if split == "train":
	left_disparity_pattern = str(root / "disp_occ_0" / "*.png")
	right_disparity_pattern = str(root / "disp_occ_1" / "*.png")

Doing that will effectively remove the *_11.png images and thus eliminate the disconnect between the images and disparities.

However, we also need to decide what we do for split="test". Do we continue to return the *_11.png images or do we leave them out there as well? Looking at them, it seems both images are either taken from different cameras or at different times. In any case, they both show basically the same scene with slight differences. @tpet Any insights if it would be useful to keep both images for testing?

[tpet]

I guess, the pairs are to be used in optical flow, so I would purge them out for stereo dataset.