[FIXED] Faster Alternative to numpy.unique

Issue

I'm trying to find unique pixels in a numpy ndarray of shape 400, 800, 3. I can get exactly what I want with np.unique(im.reshape(-1, 3), axis=0) however np.unique is too slow for my application because it sorts the array. I found this answer which seems to offer a faster alternative. However, because it flattens the array, it gives unique single integers, rather than unique RGB color values. How can I get unique rgb values with numpy?

Solution

Standard set will be the fastest:

from timeit import timeit

import numpy as np
import pandas as pd

np.random.seed(42)
x = np.random.randint(0, 256, (400, 800, 3))

t1 = timeit("u = np.unique(x.reshape(-1, 3), axis=0)", number=1, globals=globals())
t2 = timeit(
    "xt = x.reshape(-1, 3).T;pd.Series(zip(xt[0], xt[1], xt[2])).unique()",
    number=1,
    globals=globals(),
)
t3 = timeit(
    "u = set(zip(x.flat[::3], x.flat[1::3], x.flat[2::3]))", number=1, globals=globals()
)
t4 = timeit(
    "l1, l2, l3 = x.flat[::3].tolist(), x.flat[1::3].tolist(), x.flat[2::3].tolist();u = set(zip(l1, l2, l3))",
    number=1,
    globals=globals(),
)

print(f"np.unique      {t1:>10.3f}")
print(f"pd.Series      {t2:>10.3f}")
print(f"set            {t3:>10.3f}")
print(f"tolist() + set {t4:>10.3f}")

Prints on my computer (AMD 5700x):

np.unique           0.233
pd.Series           0.117
set                 0.058
tolist() + set      0.040

Answered By - Andrej Kesely