Issue
Having the file TEST.txt structured as following:
a 45
b 45 55
c 66
When I try to open it:
import numpy as np
a= np.loadtxt(r'TEST.txt',delimiter='\t',dtype=str)
I have got the following error:
ValueError: Wrong number of columns at line 2
It's clearly due to the fact that the second line has three columns instead of two, but I can't find an answer to my problem using the documentation.
Is there anyway I can fix it keeping all the data into an array?
In Matlab I can do something like:
a=textscan(fopen('TEST.txt'),'%s%s%s');
Something similar in Python would be apreciated.
Solution
Try np.genfromtxt
. It handles missing values; loadtxt
does not. Compare their docs.
Missing values can be tricky when the delimiter is white space, but with tabs it should be ok. If there still are problems, test it with a ,
delimiter.
oops - you still need the extra delimiter
eg.
a, 34,
b, 43, 34
c, 34
Both loadtxt
and genfromtxt
accept any iterable that delivers the txt line by line. So a simple thing is to readlines
, tweak the lines that have missing values and delimiters, and pass that list of lines to the loader. Or you can write this a 'filter' or generator. This approach has been described in a number of previous SO questions.
In [36]: txt=b"""a\t45\t\nb\t45\t55\nc\t66\t""".splitlines()
In [37]: txt
Out[37]: [b'a\t45\t', b'b\t45\t55', b'c\t66\t']
In [38]: np.genfromtxt(txt,delimiter='\t',dtype=str)
Out[38]:
array([['a', '45', ''],
['b', '45', '55'],
['c', '66', '']],
dtype='<U2')
I'm using Python3 so the byte strings are marked with a 'b' (for baby and me).
For strings, this is overkill; but genfromtxt
makes it easy to construct a structured array with different dtypes for each column. Note that such array is 1d, with named fields - not numbered columns.
In [50]: np.genfromtxt(txt,delimiter='\t',dtype=None)
Out[50]:
array([(b'a', 45, -1), (b'b', 45, 55), (b'c', 66, -1)],
dtype=[('f0', 'S1'), ('f1', '<i4'), ('f2', '<i4')])
to pad the lines I could define a function like:
def foo(astr,delimiter=b',',cnt=3,fill=b' '):
c = astr.strip().split(delimiter)
c.extend([fill]*cnt)
return delimiter.join(c[:cnt])
and use it as:
In [85]: txt=b"""a\t45\nb\t45\t55\nc\t66""".splitlines()
In [87]: txt1=[foo(txt[0],b'\t',3,b'0') for t in txt]
In [88]: txt1
Out[88]: [b'a\t45\t0', b'a\t45\t0', b'a\t45\t0']
In [89]: np.genfromtxt(txt1,delimiter='\t',dtype=None)
Out[89]:
array([(b'a', 45, 0), (b'a', 45, 0), (b'a', 45, 0)],
dtype=[('f0', 'S1'), ('f1', '<i4'), ('f2', '<i4')])
Answered By - hpaulj
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.