[FIXED] arranging text files side by side using python

Issue

I have 3000 text files in a directory and each .txt file contain single column data. i want to arrange them side by side to make it a mxn matrix file.

For example: paste 1.txt 2.txt 3.txt 4.txt .............3000.txt in linux

For this i tried

printf "%s\n" *.txt | sort -n | xargs -d '\n' paste

However it gives error paste: filename.txt: Too many open files

please suggest a better solution for the same using python.

Solution

Based on the inputs received, please follow the below steps.

# Change ulimit to increase the no of open files at a time
$ ulimit -n 4096

# Remove blank lines from all the files
$ sed -i '/^[[:space:]]*$/d' *.txt

# Join all files side by side to form a matrix view
$ paste $(ls -v *.txt) > matrix.txt

# Fill the blank values in the matrix view with 0's using awk inplace
$ awk -i inplace 'BEGIN { FS = OFS = "\t" } { for(i=1; i<=NF; i++) if($i ~ /^ *$/) $i = 0 }; 1' matrix.txt

Answered By - BarathVutukuri