[FIXED] Python 3: "TypeError: cannot astype a datetimelike from [datetime64[ns]] to [int32]" - Jupyter Notebook

Issue

Running into an issue with a jupyter notebook project I'm trying to get working on my Windows 10 machine, running Python 3. I get the mentioned error from this function:

buy_per_min = (buy
               .groupby([pd.Grouper(key='timestamp', freq='Min'), 'price'])
               .shares
               .sum()
               .apply(np.log)
               .to_frame('shares')
               .reset_index('price')
               .between_time(market_open, market_close)
               .groupby(level='timestamp', as_index=False, group_keys=False)
               .apply(lambda x: x.nlargest(columns='price', n=depth))
               .reset_index())
buy_per_min.timestamp = buy_per_min.timestamp.add(utc_offset).astype(int)
buy_per_min.info()

The issue is in the buy_per_min.timestamp = buy_per_min.timestamp.add(utc_offset).astype(int) line, but I don't fully understand why I'm getting it. This is the full traceback:

TypeError                                 Traceback (most recent call last)
<ipython-input-28-396768b710c8> in <module>()
     10                .apply(lambda x: x.nlargest(columns='price', n=depth))
     11                .reset_index())
---> 12 buy_per_min.timestamp = buy_per_min.timestamp.add(utc_offset).astype(int)
     13 buy_per_min.info()

~\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\generic.py in astype(self, dtype, copy, errors, **kwargs)
   5689             # else, only a single dtype is given
   5690             new_data = self._data.astype(dtype=dtype, copy=copy, errors=errors,
-> 5691                                          **kwargs)
   5692             return self._constructor(new_data).__finalize__(self)
   5693 

~\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\internals\managers.py in astype(self, dtype, **kwargs)
    529 
    530     def astype(self, dtype, **kwargs):
--> 531         return self.apply('astype', dtype=dtype, **kwargs)
    532 
    533     def convert(self, **kwargs):

~\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\internals\managers.py in apply(self, f, axes, filter, do_integrity_check, consolidate, **kwargs)
    393                                             copy=align_copy)
    394 
--> 395             applied = getattr(b, f)(**kwargs)
    396             result_blocks = _extend_blocks(applied, result_blocks)
    397 

~\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\internals\blocks.py in astype(self, dtype, copy, errors, values, **kwargs)
    532     def astype(self, dtype, copy=False, errors='raise', values=None, **kwargs):
    533         return self._astype(dtype, copy=copy, errors=errors, values=values,
--> 534                             **kwargs)
    535 
    536     def _astype(self, dtype, copy=False, errors='raise', values=None,

~\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\internals\blocks.py in _astype(self, dtype, **kwargs)
   2137 
   2138         # delegate
-> 2139         return super(DatetimeBlock, self)._astype(dtype=dtype, **kwargs)
   2140 
   2141     def _can_hold_element(self, element):

~\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\internals\blocks.py in _astype(self, dtype, copy, errors, values, **kwargs)
    631 
    632                     # _astype_nansafe works fine with 1-d only
--> 633                     values = astype_nansafe(values.ravel(), dtype, copy=True)
    634 
    635                 # TODO(extension)

~\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\dtypes\cast.py in astype_nansafe(arr, dtype, copy, skipna)
    644         raise TypeError("cannot astype a datetimelike from [{from_dtype}] "
    645                         "to [{to_dtype}]".format(from_dtype=arr.dtype,
--> 646                                                  to_dtype=dtype))
    647 
    648     elif is_timedelta64_dtype(arr):

TypeError: cannot astype a datetimelike from [datetime64[ns]] to [int32]

Is there some kind of conversion I need to do to the timestamp info, and what might it look like? Thanks!

UPDATE

There has been a similar question asked before, which I already read, but fail to see how that can be applied to my issue and would love an explanation if someone else knew. It can be found here:

Pandas DataFrame - 'cannot astype a datetimelike from [datetime64[ns]] to [float64]' when using ols/linear regression

Solution

Pandas cannot convert datetimes to int32, so raised error. If convert to np.int64 it working, also working convert numpy array converted to int with wrong values or convert to int64 - then get datetimes in native format in nanoseconds:

rng = pd.date_range('2017-04-03 12:00:45', periods=3)
buy_per_min = pd.DataFrame({'timestamp': rng})  

from datetime import timedelta
utc_offset = timedelta(hours=4)

print (buy_per_min.timestamp.add(utc_offset))
0   2017-04-03 16:00:45
1   2017-04-04 16:00:45
2   2017-04-05 16:00:45
Name: timestamp, dtype: datetime64[ns]

print (buy_per_min.timestamp.add(utc_offset).values)
['2017-04-03T16:00:45.000000000' '2017-04-04T16:00:45.000000000'
 '2017-04-05T16:00:45.000000000']
print (buy_per_min.timestamp.add(utc_offset).values.astype(np.int64))
[1491235245000000000 1491321645000000000 1491408045000000000]

print (buy_per_min.timestamp.add(utc_offset).astype(np.int64))
0    1491235245000000000
1    1491321645000000000
2    1491408045000000000
Name: timestamp, dtype: int64

#https://stackoverflow.com/a/12716674
print (buy_per_min.timestamp.add(utc_offset).values.astype(int))
[ -289111552 -2146205184   291668480]

Answered By - jezrael