[FIXED] DeepSORT's Feature extractor cannot be used for Person ReIdentification

Issue

I am using this repo for DeepSORT - https://github.com/nwojke/deep_sort

I am trying to build a Multi Camera Person Tracking system. I want to save and utilize the features extracted by one camera in footage from other cameras.

The Feature extractor which is trained on Mars dataset, doesn't seem to help in differentiating between two different people.

I wrote the below snippet to check the Cosine Distance between images from same person and different person.

extr = Extractor("./deep_sort_pytorch/deep_sort/deep/checkpoint/ckpt.t7")
list = glob.glob("mars/*.jpg")
features_list = []

for i in list:
    im = cv2.imread(i)
    im_crops = [im]
    features = extr(im_crops)
    features_list.append(features)

for f in features_list:
    print(_cosine_distance(f, features_list[0]),"<<")

def _cosine_distance(a, b, data_is_normalized=False):
    cos = nn.CosineSimilarity(dim=1, eps=1e-6)
    return (1. - cos(torch.from_numpy(a), torch.from_numpy(b)))

As expected, the cosine distance between images of same person is very low. But unexpectedly the cosine distance between crops of two different people is also similarly low.

I thought the Feature extractor will help me in differentiating.

Shall I increase the latent space dimensions from 512 to a bigger size?

Or maybe I am mistaking the role of Feature extractor.

Solution

A slightly larger feature space may help. But your main issue is the architecture of the feature extractor. In order to match people and distinguish them from impostors, features corresponding small local regions (e.g. shoes, glasses) and global whole body regions are equally important. This is not captured by the simple feature extractor provided by https://github.com/nwojke/deep_sort. For more information on this check: https://arxiv.org/pdf/1905.00953.pdf. I recommend you to try any of the OSNet models provided here: https://kaiyangzhou.github.io/deep-person-reid/MODEL_ZOO

I can also recommend you to check out my repository: https://github.com/mikel-brostrom/Yolov5_DeepSort_Pytorch. It seems to provide all you need: multi-camera multi-object tracking and OSNet models

Answered By - Mike B