Linux Ask! is a Q & A web site specific for Linux related questions. Questions are collected, answered and audited by experienced Linux users.
Parallel the execution of a job that read from stdin
Answer:
Assume you have a huge list of file, e.g.
line1 line2 line3 ..
You want to "feed every line" of this file to an external program that read from stdin, to parallel it, you can use the gnu parallel, e.g.
cat input.txt | parallel -j16 -N1 --pipe myprogram
Executing jobs in parallel using GNU parallel
Answer:
GNU parallel is a tool for executing jobs in parallel. It is not part of standard Linux distribution yet, so it is so worth to install it.
You can download the latest version from: http://ftp.gnu.org/gnu/parallel/
E.g.
# wget http://ftp.gnu.org/gnu/parallel/parallel-20101202.tar.bz2
# tar -jxvf parallel-20101202.tar.bz2
# cd parallel-20101202
# ./configure
# make && sudo make install
Example usages:
# echo -e "a\nb\nc" | parallel echo "foo"
foo a
foo b
foo c
