How to transform a quickdraw image to 84 by 84 in pytorch using the learn2learn library?

Question

I was trying to use learn2learn's QuickDraw but it seems like I'm getting errors when I attempt to apply transformations (resizing to 84x84, random crop) on it. The issue stems from when l2l's quickdraw library attempts to apply transformations onto a quickdraw image (that is apparently in the form of a np.memmap/.npy record that PIL can't understand) and so I get the following error:

Traceback (most recent call last):
  File "/home/pzy2/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/dataloaders/maml_patricks_l2l.py", line 2300, in <module>
    loop_through_l2l_indexable_benchmark_with_model_test()
  File "/home/pzy2/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/dataloaders/maml_patricks_l2l.py", line 2259, in loop_through_l2l_indexable_benchmark_with_model_test
    for benchmark in [quickdraw_l2l_tasksets()]: #hdb8_l2l_tasksets(),hdb9_l2l_tasksets(), delaunay_l2l_tasksets()]:#[dtd_l2l_tasksets(), cu_birds_l2l_tasksets(), fc100_l2l_tasksets()]:
  File "/home/pzy2/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/dataloaders/maml_patricks_l2l.py", line 2216, in quickdraw_l2l_tasksets
    _transforms: tuple[TaskTransform, TaskTransform, TaskTransform] = get_task_transforms_quickdraw(_datasets,
  File "/home/pzy2/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/dataloaders/maml_patricks_l2l.py", line 2184, in get_task_transforms_quickdraw
    train_transforms: TaskTransform = DifferentTaskTransformIndexableForEachDataset(train_dataset,
  File "/home/pzy2/diversity-for-predictive-success-of-meta-learning/div_src/diversity_src/dataloaders/common.py", line 130, in __init__
    self.indexable_dataset = MetaDataset(indexable_dataset)
  File "learn2learn/data/meta_dataset.pyx", line 59, in learn2learn.data.meta_dataset.MetaDataset.__init__
  File "learn2learn/data/meta_dataset.pyx", line 96, in learn2learn.data.meta_dataset.MetaDataset.create_bookkeeping
  File "learn2learn/data/meta_dataset.pyx", line 65, in learn2learn.data.meta_dataset.MetaDataset.__getitem__
  File "/home/pzy2/miniconda3/envs/metalearning3.9/lib/python3.9/site-packages/learn2learn/vision/datasets/quickdraw.py", line 511, in __getitem__
    image = self.transform(image)
  File "/home/pzy2/miniconda3/envs/metalearning3.9/lib/python3.9/site-packages/torchvision/transforms/transforms.py", line 60, in __call__
    img = t(img)
  File "/home/pzy2/miniconda3/envs/metalearning3.9/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/pzy2/miniconda3/envs/metalearning3.9/lib/python3.9/site-packages/torchvision/transforms/transforms.py", line 900, in forward
    i, j, h, w = self.get_params(img, self.scale, self.ratio)
  File "/home/pzy2/miniconda3/envs/metalearning3.9/lib/python3.9/site-packages/torchvision/transforms/transforms.py", line 859, in get_params
    width, height = F._get_image_size(img)
  File "/home/pzy2/miniconda3/envs/metalearning3.9/lib/python3.9/site-packages/torchvision/transforms/functional.py", line 67, in _get_image_size
    return F_pil._get_image_size(img)
  File "/home/pzy2/miniconda3/envs/metalearning3.9/lib/python3.9/site-packages/torchvision/transforms/functional_pil.py", line 26, in _get_image_size
    raise TypeError("Unexpected type {}".format(type(img)))
TypeError: Unexpected type <class 'numpy.memmap'>

original:

• https://github.com/learnables/learn2learn/issues/392
• https://discuss.pytorch.org/t/how-to-transform-a-quickdraw-image-to-84-by-84-in-pytorch-using-the-learn2learn-library/178292

Answer 1

If you are using torchvision.transforms.ToTensor this is not possible directly, because as stated in the documentation

this transformation should not be used when transforming target image masks

check out this instead to see the proper way of resizing

And here is an example quoted from this answer:

>>> import numpy as np
>>> import torch
>>> np_arr = np.ones((5289, 38))
>>> torch_tensor = torch.from_numpy(np_arr).long()
>>> type(np_arr)
<class 'numpy.ndarray'>
>>> type(torch_tensor)
<class 'torch.Tensor'>

Answer 2

try this code out and let me know how it goes

import torch
import torchvision.transforms as transforms
from PIL import Image
from learn2learn.data import QuickDraw

quickdraw = QuickDraw(root='path/to/quickdraw/data', train=True, transform=None)
transform = transforms.Compose([
    transforms.Resize(84),
    transforms.CenterCrop(84),
    transforms.ToTensor(),
])

# Get an example QuickDraw image
img, _ = quickdraw[0]

# Convert to PIL image
img = Image.fromarray(img)

# Apply transform
img = transform(img)

# Add a batch dimension
img = img.unsqueeze(0)

# Display the transformed image
print(img.shape)  # torch.Size([1, 1, 84, 84])

Answer 3

Based on the lack of code I will get some inspiration from your issue #333 one year ago. I guess it changed a lot and you are not working on the same thing. Although, I think you kept the basic data structure.

You want to create your MetaDataset based on your QuickDraw dataset. I do not think QuickDraw can be indexed directly by MetaDataset because MetaDataset calls each element. If you apply a data transform on a bad data format -> you get incompatibility.

Actually, if you look at quickdraw on the repo, the MetaDataset is not instantiate anywhere from it. This is because it does not work: this is your bug.

The problem is the following: how to do the book keeping (indexing) on quickdraw which seems to be very big compared to the other dataset?

I suggest you the following:

1. Perform the book keeping yourself based on the annotations file from the quickdraw repo. Then you can put it has an attribute to your new QuickDraw class. You won't have the (very long) book keeping in MetaDataset. This indexing can be stored in .pkl file so you can do it only once.
2. Then add a pre-transform only for QuickDraw. Actually, if you really do not want to touch your transform in your class, let's say, I do not know your constraints, you can monkey patch the object, in worst case. But the more reasonable fix would be to insert into your data transform a function to convert the np.memmap to PIL or Tensor object.

Answer 4

Try converting the numpy.memmap object to a NumPy array and then to a PIL image.