Caltech-101 doesn't load bboxes but it should.

[Swarzkopf314]

🐛 Describe the bug

Annotation files in Caltech-101 have two keys: 'box_coord' and 'obj_contour'. Caltech101 PyTorch class appends just the contours, explicitly skipping 'box_coord'. This would be ok if coordinates of obj_contour were relative to the entire image. Unfortunately they are relative to the bboxes and thus are useless without loading box_coord.

There are two ways to fix this - load box_coord or transform obj_contour coordinates to be relative to the image.

Versions

My env is irrelevant as this is a clear bug in code on main:

torchvision/datasets/caltech.py#L95C9-L110C9

        target: Any = []
        for t in self.target_type:
            if t == "category":
                target.append(self.y[index])
            elif t == "annotation":
                data = scipy.io.loadmat(
                    os.path.join(
                        self.root,
                        "Annotations",
                        self.annotation_categories[self.y[index]],
                        f"annotation_{self.index[index]:04d}.mat",
                    )
                )
                target.append(data["obj_contour"])
        target = tuple(target) if len(target) > 1 else target[0]

cc @pmeier

[pmeier]

Thanks @Swarzkopf314 for the report. Can confirm that the contour coordinates are relative:

from torchvision import datasets

from PIL import ImageDraw

def draw_and_save(image, contour, title):
    image = image.convert("RGB")
    draw = ImageDraw.Draw(image)
    draw.polygon(list(map(tuple, contour.T.astype("int64").tolist())), outline="red", width=2)

    image.save(f"{title}.png")

IDX = 3562

###################################

dataset = datasets.Caltech101(..., target_type="annotation")

image, contour = dataset[IDX]

draw_and_save(image, contour, "without_bbox")

###################################

# this target type only exist locally for me
dataset = datasets.Caltech101(..., target_type="annotation_fixed")

image, target = dataset[IDX]
contour, bbox = target["obj_contour"], target["box_coord"]

# Bounding boxes are in format y1, y2, x1, x2
# We need to add x1 and y1 here
contour += bbox.T[[2, 0], :]

draw_and_save(image, contour, "with_bbox")

Since the contours that we currently return are indeed unusable, I'm inclined to treat a patch as a bug fix. For example, we could just return a dictionary with keys "obj_contour" and "box_coord" for target_type="annotation". This is what I did in the snippet above for the target_type="annotation_fixed".

As bug fix, it would be less bad breaking BC here. Given the fact that Caltech101 is a really old dataset (#775, > 4 years) and no one reported this until now, my guess is that the amount of affected users is really low.

However, since the bounding box comes in an awkward format (y1, y2, x1, x2) and without deeper knowledge of the dataset it is also unclear that the contour coordinates are relative to the bounding box, I would also add target_type="bounding_box" and target_type="contour" for convenience. The former would return the bounding box in XYXY (x1, y1, x2, y2) format and the latter is the contour in coordinates relative to the image rather the bounding box.

[NicolasHug]

Since the contours that we currently return are indeed unusable, I'm inclined to treat a patch as a bug fix

I agree, thanks for the report @Swarzkopf314

The former would return the bounding box in XYXY (x1, y1, x2, y2)

Do we usually preserve the original format?

Re "box_coord" and target_type="bounding_box" names, we should definitely keep both names the same (and ideally, consistent with the rest of the datasets, and or default consistent with the original dataset)

[pmeier]

Do we usually preserve the original format?

Yes. However, so far all bounding boxes were mostly in XYXY or at least in a format that is supported by torchvision.ops.box_convert (XYXY, XYWH, and CXCYWH). These are the same formats that the v2 transforms API supports as well. Given that I have never seen Y1Y2X1X2 before and the extra target_type would be for convenience, I would reorder them to X1Y1X2Y2 aka XYXY. Note that we don't need any computation here as would be the case for all other format changes.

Re "box_coord" and target_type="bounding_box" names, we should definitely keep both names the same

Fair enough.

ideally, consistent with the rest of the datasets

Not possible since we are not standardizing names so far. Meaning, every dataset uses the key given by said dataset.