bash - Sort lots of large compressed files -


i have lot of large compressed files called xaa.gz, xab.gz, xac.gz etc. unfortunately not sorted. equivalent of following.

zcat x*|sort > largefile  split -l 1000000 largefile

then gzip split files , throw away other files made before well.

the problem makes massive uncompressed file , lots of smaller uncompressed split files before compressing them. possible whole thing without making huge file in middle of process , ideally without saving split files before compressing them either?

i have 8 cores take advantage of them (i don't have coreutils 8.20 can't take advantage of sort --parallel).

not full code, ideas on can here.

1) partition input files process them in parallel:

num_cores=8 i=0 while read f;   part_name=part$i   set $part_name="${!part_name} $f"   (( i=(i+1)%num_cores )) done < <(ls x*.gz)

2) decompress , sort part of files in different processes:

sort -m <(zcat $part0 | sort) <(zcat $part1 | sort) ...

3) tell split compress files immediately:

... | split -l 1000000 --filter='gzip > $file.gz'

Comments

Post a Comment

Popular posts from this blog

matlab - Deleting rows with specific rules -

i got 20*3 cell array , need delete rows contains "137", "2" , "n:t" origin data: 't' '' '' 'np(*)' '' '' [ 137] '' '' [ 2] '' '' 'are' 'and' 'np(fcc_a1#1)' '' '' '1:t' [ 1200] [0.7052] '' [1.2051e+03] [0.7076] '' 'are' 'and' 'np(fcc_a1#3)' '' '' '2:t' [ 1200] [0.0673] '' [1.2051e+03] [0.0671] '' 'are' 'and' 'np(m23c6)' '' '' '3:t' [ 1200] [0.227...
Read more

php - MySQLi multi_query results for later use -

i using mysqli multi_query work several select statemets @ time. know how handle results, able use them later in code. example: <?php //connection stuff $query = "select name, surname database1;"; $query.= "select car, year, type database2 carid='1';"; $query.= "select product, price database3;"; if ($mysqli->multi_query($query)) { if($result = $mysqli->store_result()) { while($row = $result->fetch_row()) { --> here? } } } ?> <html> <div id='persona'> <?php foreach() { --> print name + surname } ?> </div> <div id='cars'> <?php foreach() { --> print car + year + type } ?> </div> <div id='product'> <?php foreach() { --> print product + price } ?> </div> </html> one more thing, prepared statements not possible when using multiple_query, right? there no benefit in putting unrelated queries in 1 multi ...
Read more