[FIXED] pickle dependency failure

Issue

I am experiencing a module dependency failure when unpickling an object created on Linux and (attempting) to unpickle on Windows. The pickle correctly unpickles in Linux, but fails in Windows. Both systems are running Python 2.6.

I have studied the pickle manual page (with particular focus on having the unpickle environment the same as the pickle environment), and several pieces of great advice here - yet I remain stumped. Most are advising to make sure the sys.modules are correct and loaded. Here are some snippets that show what I'm attempting:

The pickling code:

...
pickle_fp = self.getPickleFile('wb')
pickler = Pickler(pickle_fp, protocol=2)
pickler.dump(archive)
pickle_fp.close()
...

In the unpickling code I have added a line to print out the sys.modules dictionary so we can see the modules present:

...
pickle_fp = self.getPickleFile('rb')
unpickler = Unpickler(pickle_fp)
pprint.pprint(sys.modules)
package = unpickler.load()
pickle_fp.close()
...

when I run the unpickle in Linux it works great. When I attempt to unpack the pickle generated on Linux on Windows I get:

...
ImportError: No module named photo_data

as for the environment, the pprint.pprint(sys.modules) produces, on Linux

...
'photo_data': <module 'photo_data' from 
'/home/xxx/Desktop/PythonPhoto/photo_data.pyc'>,
...

and on Windows

...
'photo_data': <module 'photo_data' from                       
'C:\Users\xxx\git\PhotoManagement\Photo\src\photo_data.pyc'>,
...

So it appears to me that I've got the photo_data module in the environment. I tried using pickle without the protocol (defaulting to 0) and I tried running unix2dos to strip characters. I'm officially stumped.

Thanks for your help!

---

Based on suggestions in the comments I have generated the simplest case that does not work. The classes I'm pickling look like this:

class photo_data:
    def __init__(self):
        self.isdir = False
        self.size = 0
        self.mtime = -(sys.maxint - 1) #Set default time to very old
        self.timestamp = datetime.datetime.strptime('1700:1:1 00:00:00', '%Y:%m:%d %H:%M:%S')
        self.gotTags = False
        self.signature = ''
        self.fileMD5 = ''
        self.userTags = ''
        self.inArchive = False
        self.candidates = []
        self.dirpaths = []
        self.filepaths = []      

class photo_collection:  #This class should be data only

    def __init__(self):
        self.host = ''
        self.path = ''
        self.photo = dict()
        self.pickle = None
        self.datasetChanged = False

    def __getitem__(self, key):
        return self.photo[key]

    def __setitem__(self, key, value):
        self.photo[key] = value

In my use case, the photo_collection object is instantiated, and the dictionary self.photo is filled with instances of photo_data. The simplest case that works between systems is a directory with one photo in it, and arbitrarily complex cases work within a system. The simplest case that doesn't work between systems is a directory with one photo and one subdirectory which also contains a photo.

Per request, I have attached two pickle files saved in format = 0. If you compare them I see that the program has descended the file tree in a different order (maybe not a big surprise since I copied the directory between systems and OSes), but otherwise they seem to open and close with identical structure other than file-specific data. I don't see how to upload files to a separate place so I include them in-line here.

This is the Windows-generated pickle:

(iphoto_data
photo_collection
p0
(dp1
S'path'
p2
S'C:\\Users\\scott_jackson\\Desktop\\phototest'
p3
sS'host'
p4
S'4DAA1001312'
p5
sS'pickle'
p6
NsS'datasetChanged'
p7
I01
sS'photo'
p8
(dp9
S'C:\\Users\\scott_jackson\\Desktop\\phototest\\img_4697.jpg'
p10
(iphoto_data
photo_data
p11
(dp12
S'isdir'
p13
I00
sS'dirpaths'
p14
(lp15
sS'filepaths'
p16
(lp17
sS'timestamp'
p18
cdatetime
datetime
p19
(S'\x07\xda\x04\x12\x124&\x00\x00\x00'
p20
tp21
Rp22
sS'gotTags'
p23
I01
sS'signature'
p24
S'9b2ca527b2bf0865d9b87ecd2a68d417'
p25
sS'fileMD5'
p26
S''
p27
sS'candidates'
p28
(lp29
sS'mtime'
p30
F1347576558.0
sS'inArchive'
p31
I00
sS'userTags'
p32
S'NA'
p33
sS'size'
p34
L6489323L
sbsg3
(iphoto_data
photo_data
p35
(dp36
g13
I01
sg14
(lp37
S'C:\\Users\\scott_jackson\\Desktop\\phototest\\060101 Nags Head'
p38
asg16
(lp39
g10
asg18
g19
(S'\x06\xa4\x01\x01\x00\x00\x00\x00\x00\x00'
p40
tp41
Rp42
sg23
I00
sg24
g27
sg26
g27
sg28
(lp43
sg30
I-2147483646
sg31
I00
sg32
g27
sg34
I0
sbsS'C:\\Users\\scott_jackson\\Desktop\\phototest\\060101 Nags Head\\img_1150.jpg'
p44
(iphoto_data
photo_data
p45
(dp46
g13
I00
sg14
(lp47
sg16
(lp48
sg18
g19
(S'\x07\xd6\x01\x01\x11\t#\x00\x00\x00'
p49
tp50
Rp51
sg23
I01
sg24
S'5925063685af0d741a23fe6d75523741'
p52
sg26
g27
sg28
(lp53
sg30
F1347751812.0
sg31
I00
sg32
g33
sg34
L538233L
sbsS'C:\\Users\\scott_jackson\\Desktop\\phototest\\060101 Nags Head'
p54
(iphoto_data
photo_data
p55
(dp56
g13
I01
sg14
(lp57
sg16
(lp58
g44
asg18
g19
(S'\x06\xa4\x01\x01\x00\x00\x00\x00\x00\x00'
p59
tp60
Rp61
sg23
I00
sg24
g27
sg26
g27
sg28
(lp62
sg30
I-2147483646
sg31
I00
sg32
g27
sg34
I0
sbssb.

And this is the Linux-generated pickle:

(iphoto_data
photo_collection
p0
(dp1
S'path'
p2
S'/home/scott/phototest'
p3
sS'host'
p4
S'barney'
p5
sS'pickle'
p6
NsS'datasetChanged'
p7
I01
sS'photo'
p8
(dp9
S'/home/scott/phototest/060101 Nags Head/img_1150.jpg'
p10
(iphoto_data
photo_data
p11
(dp12
S'isdir'
p13
I00
sS'dirpaths'
p14
(lp15
sS'filepaths'
p16
(lp17
sS'timestamp'
p18
cdatetime
datetime
p19
(S'\x07\xd6\x01\x01\x11\t#\x00\x00\x00'
p20
tp21
Rp22
sS'gotTags'
p23
I01
sS'signature'
p24
S'5925063685af0d741a23fe6d75523741'
p25
sS'fileMD5'
p26
S''
p27
sS'candidates'
p28
(lp29
sS'mtime'
p30
F1347751812.0842018
sS'inArchive'
p31
I00
sS'userTags'
p32
S'NA'
p33
sS'size'
p34
I538233
sbsS'/home/scott/phototest/060101 Nags Head'
p35
(iphoto_data
photo_data
p36
(dp37
g13
I01
sg14
(lp38
sg16
(lp39
g10
asg18
g19
(S'\x06\xa4\x01\x01\x00\x00\x00\x00\x00\x00'
p40
tp41
Rp42
sg23
I00
sg24
g27
sg26
g27
sg28
(lp43
sg30
I-9223372036854775806
sg31
I00
sg32
g27
sg34
I0
sbsS'/home/scott/phototest/img_4697.jpg'
p44
(iphoto_data
photo_data
p45
(dp46
g13
I00
sg14
(lp47
sg16
(lp48
sg18
g19
(S'\x07\xda\x04\x12\x124&\x00\x00\x00'
p49
tp50
Rp51
sg23
I01
sg24
S'9b2ca527b2bf0865d9b87ecd2a68d417'
p52
sg26
g27
sg28
(lp53
sg30
F1347576558.5344362
sg31
I00
sg32
g33
sg34
I6489323
sbsg3
(iphoto_data
photo_data
p54
(dp55
g13
I01
sg14
(lp56
S'/home/scott/phototest/060101 Nags Head'
p57
asg16
(lp58
g44
asg18
g19
(S'\x06\xa4\x01\x01\x00\x00\x00\x00\x00\x00'
p59
tp60
Rp61
sg23
I00
sg24
g27
sg26
g27
sg28
(lp62
sg30
I-9223372036854775806
sg31
I00
sg32
g27
sg34
I0
sbssb.

Please feel free to paste these into your favorite 'compare' editor; I don't know enough about pickles to detect the problem.

Thanks in advance for your help!!

Solution

I've reproduced your problem by saving the pickle with CRLF (Windows) line endings on my Mac OS X machine.

The pickle machinery is quite particular about newlines. If the pickle is saved or copied using a non-binary transfer mode (e.g. resaved with a text editor on Windows, copied using ASCII FTP transfer, saved as a text file from a website, etc.), the pickle will be corrupted with the addition of CR characters.

Now, the problem is this line in pickle.py:

module = self.readline()[:-1]

If the file you give the Unpickler is opened 'rb' but contains CRLF, then these lines will read module = "photo_data\r", which isn't a valid module name. Upon importing, the error will appear as

ImportError: No module named photo_data

with an unprinted carriage return after the photo_data (dreadfully sneaky!).

The solution is to ensure you transfer files in a binary fashion, and don't run unix2dos or any similar utilities on the pickle. Alternately, if using protocol 0 (text) pickles, it is safe to use 'rU' (universal newline mode) to open the pickled files instead.

See also Pickled file won't load on Mac/Linux.

Answered By - nneonneo