[FIXED] python numpy - convert Timestamps to datetime

Issue

I have a np array such as follow:

example = np.array([[Timestamp('2005-03-06 17:00:00'), 1225.75, 1226.25, 1225.0, 1225.5,
        1668.0],
       [Timestamp('2005-03-06 17:30:00'), 1225.75, 1227.5, 1225.75, 1227.0,
        1603.0],
       [Timestamp('2005-03-06 18:00:00'), 1227.0, 1227.5, 1226.75, 1227.25,
        590.0]], dtype=object)

The first column is a Timestamp type value. How do i convert those values to datetime? I know there are few similar questions on the topic, but i couldnt manage to form a clear understanding of it and figure out a clean neat solution based on them.

I can convert the timestamp of a single value with example[0,0].to_datetime() but how to do it on all the Timestamps at once? ideally something like example[:,0]....

Solution

If I define Timestamp as the numpy datetime dtype:

In [43]: Timestamp=np.datetime64

Then I can copy-n-paste your example:

In [44]: example = np.array([[Timestamp('2005-03-06 17:00:00'), 1225.75, 1226.25 , 1225.0, 1225.5, 1668.0],
    ...:        [Timestamp('2005-03-06 17:30:00'), 1225.75, 1227.5, 1225.75, 1227.0,         1603.0],
    ...:        [Timestamp('2005-03-06 18:00:00'), 1227.0, 1227.5, 1226.75, 1227.25, 590.0]], dtype=object)

Note that this array is dtype object

In [45]: example
Out[45]: 
array([[numpy.datetime64('2005-03-06T17:00:00'), 1225.75, 1226.25, 1225.0,1225.5, 1668.0],
       [numpy.datetime64('2005-03-06T17:30:00'), 1225.75, 1227.5, 1225.75, 1227.0, 1603.0],
       [numpy.datetime64('2005-03-06T18:00:00'), 1227.0, 1227.5, 1226.75, 1227.25, 590.0]], dtype=object)

The 1st column is:

In [46]: example[:,0]
Out[46]: 
array([numpy.datetime64('2005-03-06T17:00:00'),
       numpy.datetime64('2005-03-06T17:30:00'),
       numpy.datetime64('2005-03-06T18:00:00')], dtype=object)

which can be converted to an array of datetime64 elements:

In [47]: example[:,0].astype(np.datetime64)
Out[47]: array(['2005-03-06T17:00:00', '2005-03-06T17:30:00', '2005-03-06T18:00:00'], dtype='datetime64[s]')

tolist for this type of array converts the elements to datetime objects:

In [48]: example[:,0].astype(np.datetime64).tolist()
Out[48]: 
[datetime.datetime(2005, 3, 6, 17, 0),
 datetime.datetime(2005, 3, 6, 17, 30),
 datetime.datetime(2005, 3, 6, 18, 0)]

Alternatively, grabing the pandas.Timestamp function

In [50]: Timestamp = pd.Timestamp

In [52]: example
Out[52]: 
array([[Timestamp('2005-03-06 17:00:00'), 1225.75, 1226.25, 1225.0, 1225.5, 1668.0],
       [Timestamp('2005-03-06 17:30:00'), 1225.75, 1227.5, 1225.75, 1227.0,  1603.0],
       [Timestamp('2005-03-06 18:00:00'), 1227.0, 1227.5, 1226.75, 1227.25,  590.0]], dtype=object)

In [64]: ts = example[:,0]
In [65]: ts
Out[65]: 
array([Timestamp('2005-03-06 17:00:00'), Timestamp('2005-03-06 17:30:00'), Timestamp('2005-03-06 18:00:00')], dtype=object)

Iterative conversion of the Timestamp objects

In [67]: np.array([t.to_datetime() for t in ts])
Out[67]: 
array([datetime.datetime(2005, 3, 6, 17, 0),
       datetime.datetime(2005, 3, 6, 17, 30),
       datetime.datetime(2005, 3, 6, 18, 0)], dtype=object)

But I discovered that astype works with Timestamp objects:

In [73]: ts = example[:,0]
In [74]: ts.astype('datetime64[s]')
Out[74]: array(['2005-03-06T17:00:00', '2005-03-06T17:30:00', '2005-03-06T18:00:00'], dtype='datetime64[s]')

So I can use that tolist to do the conversion in one line:

In [75]: ts.astype('datetime64[s]').tolist()
Out[75]: 
[datetime.datetime(2005, 3, 6, 17, 0),
 datetime.datetime(2005, 3, 6, 17, 30),
 datetime.datetime(2005, 3, 6, 18, 0)]

I wouldn't describe this as a final solution, but it gives you an idea of how numpy deals with dates.

For array math I'd stick with the datetime64 dtype. To keep in one array along with the example[:,1:] floats you have to use a structured array.

=================

Experimenting with a copy:

In [80]: ex1 = example.copy()

In [82]: ex1[:,0] = example[:,0].astype('datetime64[s]').tolist()
In [83]: ex1
Out[83]: 
array([[datetime.datetime(2005, 3, 6, 17, 0), 1225.75, 1226.25, 1225.0, 1225.5, 1668.0],
       [datetime.datetime(2005, 3, 6, 17, 30), 1225.75, 1227.5, 1225.75, 1227.0, 1603.0],
       [datetime.datetime(2005, 3, 6, 18, 0), 1227.0, 1227.5, 1226.75, 1227.25, 590.0]], 
      dtype=object)

Answered By - hpaulj