[FIXED] Numba TypingError List of optional np.ndarray

Issue

Good afternoon,

I'm facing an issue when trying to jit this function:

from typing import Optional
import numpy as np

from numba import njit
from numba.typed import List

@njit
def check_something(list_arrays: List[Optional[np.ndarray]], mask_array: np.ndarray):
   for array in list_arrays:
      if (
         array is not None and
         (np.bitwise_and(array, mask_array)).any()
      ):
         # do something

if __name__=='__main__':
   check_something([None, np.random.randint(0, 10, 11)], np.random.randint(0, 10, 11))

Here is the error:

Traceback (most recent call last):
  File "C:\Users\ll\git\test-numba-error.py", line 17, in <module>
    check_something([None, np.random.randint(0, 100, 1837)])
TypeError: not enough arguments: expected 2, got 1
PS C:\Users\ll\git> python .\test-numba-error.py
Traceback (most recent call last):
  File "C:\Users\ll\git\test-numba-error.py", line 17, in <module>
    check_something([None, np.random.randint(0, 100, 1837)], np.random.randint(0, 100, 1837))
  File "C:\Python39\lib\site-packages\numba\core\dispatcher.py", line 468, in _compile_for_args
    error_rewrite(e, 'typing')
  File "C:\Python39\lib\site-packages\numba\core\dispatcher.py", line 409, in error_rewrite
    raise e.with_traceback(None)
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
No implementation of function Function(<ufunc 'bitwise_and'>) found for signature:

 >>> bitwise_and(none, array(int32, 1d, C))

There are 2 candidate implementations:
  - Of which 2 did not match due to:
  Overload in function 'Numpy_rules_ufunc.generic': File: numba\core\typing\npydecl.py: Line 98.
    With argument(s): '(none, array(int32, 1d, C))':
   Rejected as the implementation raised a specific error:
     TypingError: can't resolve ufunc bitwise_and for types [none, array(int32, 1d, C)]
  raised from C:\Python39\lib\site-packages\numba\core\typing\npydecl.py:107

During: resolving callee type: Function(<ufunc 'bitwise_and'>)
During: typing of call at C:\Users\ll\test-numba-error.py (11)


File "test-numba-error.py", line 11:
def check_something(list_arrays: List[Optional[np.ndarray]], mask_array: np.ndarray):
    <source elided>
    for array in list_arrays:
        if array is not None and (np.bitwise_and(array, mask_array)).any():
        ^

It seems like numba doesn't get the array is not None check when variable comes from list. Is there any way to make it understand i have checked that the value is not None ? Thanks for your time !

(Numba 0.56.0, Python 3.9.5)

Solution

Numba only support homogeneous typed lists. A list with None is heterogeneous so it is unsupported. Optional is not supported as a valid type for typed lists. AFAIK, it is only meant to be used for optional parameters. In fact, this is clearly reported by nb.typed.typedlist.ListType(nb.types.Optional(nb.int32)). Note that typed lists are not directly compatible with pure-Python lists called reflected list so you cannot pass your basic CPython list to Numba the way you do. The error reported by Numba are confusing because of the order in which Numba does the checks. Put it shortly, what you try to do is not supported. Besides, note that the dimensionality of an array and its contiguity are a part of a Numba type so you cannot mix them in a list.

Answered By - Jérôme Richard