How Do I Save A 3d Python/numpy Array As A Text File?

I have to launch a great number of calculations, and I have to save a 2D file text each time, so I would like to store results in 'real-time' as a 3D text file with each slice corr

Solution 1:

Here's an example of cPickle:

import cPickle

# write to cPickle
cPickle.dump( thing_to_save, open( "filename.pkl", "wb" ) )

# read from cPickle
thing_to_save = cPickle.load( open( "filename.pkl", "rb" ) )

The "wb" and "rb" parameters to the open() function are important. CPickle writes objects in a binary format, so using just "w" and "r" won't work.

Solution 2:

If the save file needs to be a 'csv' style text, you could use multiple savetxt and loadtxt. The key is knowning that both of these can take an open file as input.

Writing example:

In [31]: A=np.arange(3*2*4).reshape(3,2,4)    
In [32]: A    # normal display as 3 blocks of 2d array
Out[32]: 
array([[[ 0,  1,  2,  3],
        [ 4,  5,  6,  7]],

       [[ 8,  9, 10, 11],
        [12, 13, 14, 15]],

       [[16, 17, 18, 19],
        [20, 21, 22, 23]]])

In [36]: for a in A:print a, '\n'   # or iterate on the 1st dimension
[[0 1 2 3]
 [4 5 6 7]][[ 8  9 10 11]
 [12 13 14 15]][[16 17 18 19]
 [20 21 22 23]]

Following that example, I can iterate on the file, using savetxt for each subarray:

In [37]: with open('3dcsv.txt','wb') as f:
    for a in A:
        np.savetxt(f, a, fmt='%10d')
        f.write('\n')
   ....:         

Confirm the file write with a system cat (via ipython):

In [38]: cat 3dcsv.txt
         0          1          2          3
         4          5          6          7
         8          9         10         11
        12         13         14         15
        16         17         18         19
        20         21         22         23

For simple read, loadtxt apparently ignores the empty lines, returning a 6 x 4 array. So I know that it is supposed to be (2,3,4) I can easily reshape the result.

In [39]: np.loadtxt('3dcsv.txt')
Out[39]: 
array([[  0.,   1.,   2.,   3.],
       [  4.,   5.,   6.,   7.],
       [  8.,   9.,  10.,  11.],
       [ 12.,  13.,  14.,  15.],
       [ 16.,  17.,  18.,  19.],
       [ 20.,  21.,  22.,  23.]])

After a bit of debugging I got this multiple loadtxt to work. loadtxt (and genfromtxt) works with a list of lines.

In [53]: A1=[]     # list to collect blocks

In [54]: with open('3dcsv.txt') as f:
    lines = []     # list to collect lineswhile1:
        aline = f.readline()
        if aline.strip():
            lines.append(aline)     # nonempty line
        else:              # empty line
            iflen(lines)==0: break
            A1.append(np.loadtxt(lines, dtype=int))
            lines = []
   ....:             

In [55]: A1 = np.array(A1)

In [56]: A1
Out[56]: 
array([[[ 0,  1,  2,  3],
        [ 4,  5,  6,  7]],

       [[ 8,  9, 10, 11],
        [12, 13, 14, 15]],

       [[16, 17, 18, 19],
        [20, 21, 22, 23]]])

This may not be the most robust pairing of of save/load but it gives a framework for building something better.

But if it doesn't need to be text then pickle is fine, as is the native numpy 'save/load'

In [57]: np.save('3dsave.npy',A)

In [58]: np.load('3dsave.npy')
Out[58]: 
array([[[ 0,  1,  2,  3],
        [ 4,  5,  6,  7]],

       [[ 8,  9, 10, 11],
        [12, 13, 14, 15]],

       [[16, 17, 18, 19],
        [20, 21, 22, 23]]])