UNIX命令行队列工具：nq
这些实用的工具类能够创建非常轻量级的作业队列系统，不无需安装，维护，监督，或任何长期运行的进程。
nq should run on any POSIX.1-2008 compliant system which also provides a working flock(2). Tested on Linux 2.6.37, Linux 4.1, OpenBSD 5.7, FreeBSD 10.1, and Mac OS X 10.3.
The intended purpose is ad-hoc queuing of command lines (e.g. for building several targets of a Makefile, downloading multiple files one at a time, running benchmarks in several configurations, or simply as a glorifiednohup), but as any good Unix tool, it can be abused for whatever you like.
Job order is enforced by a timestampnqgets immediately when started. Synchronization happens on file-system level. Timer resolution is milliseconds. No sub-second file system time stamps are required. Polling is not used. Exclusive execution is maintained strictly.
Enforcing job order works like this:
- every job has a flock(2)ed output file ala,TIMESTAMP.PID
- every job starts only after all earlier flock(2)ed files are unlocked
- the lock is released by the kernel after the job terminates
You enqueue (get it?) new jobs usingnq CMDLINE.... The job id is output (unless suppressed using-q) andnqdetaches immediately, running the job in the background. STDOUT and STDERR are redirected into the log file.
nqtries hard (but does not guarantee) to ensure the log file of the currently running job has +x bit set. Thus you can usels -Fto get a quick overview of the state of your queue.
The "file extension" of the log file is actually the PID, so you can kill jobs easily. Before the job is started, it is the PID ofnq, so you can cancel a queued job by killing it as well.
Due to the initialexecline in the log files, you can resubmit a job by executing it as a shell command file, i.e. runningsh $jobid.
You can wait for jobs to finish usingnq -w, possibly listing job ids you want to wait for; the default is all of them. Likewise, you can test if there are jobs which need to be waited upon using-t.
By default, job ids are per-directory, but you can set$NQDIRto put them elsewhere. Creatingnqwrappers setting$NQDIRto provide different queues for different purposes is encouraged.
All these operations take worst-case quadratic time in the amount of lock files produced, so you should clean them regularly.
Examples
Build targetsclean,depends,all, without occupying the terminal:
% nq make clean % nq make depends % nq make all % fq ... look at output, can interrupt with C-c any time without stopping the build ...
Simple download queue, accessible from multiple terminals:
% mkdir -p /tmp/downloads % alias qget='NQDIR=/tmp/downloads nq wget' % alias qwait='NQDIR=/tmp/downloads fq -q' window1% qget http://mymirror/big1.iso window2% qget http://mymirror/big2.iso window3% qget http://mymirror/big3.iso % qwait ... wait for all downloads to finish ...
As nohup replacement (The benchmark will run in background, every run gets a different output file, and the command line you ran is logged too.):
% ssh remote remote% nq ./run-benchmark ,14f6f3034f8.17035 remote% ^D % ssh remote remote% fq ... see output, fq exits when job finished ...
Assumptions
nqwill only work correctly when:
- $NQDIR(respectively.) is writable.
- flock(2)works in$NQDIR(respectively.).
- gettimeofdaybehaves monotonic (usingCLOCK_MONOTONICwould create confusing file names). Else job order can be confused and multiple tasks can run at once due to race conditions.
- No other programs put files matching,*into$NQDIR(respectively.).
nq helpers
Two helper programs are provided:
fqoutputs the log of the currently running jobs, exiting when the jobs are done. If no job is running, the output of the last job is shown.fq -ashows the output of all jobs,fq -qonly shows one line per job.fqusesinotifyon Linux and falls back to polling for size change else. (fq.shis a similar tool, not quite as robust, implemented as shell-script callingtail.)
tqwrapsnqand displays thefqoutput in a new tmux or screen window.
(A pure shell implementation ofnqis provided asnq.sh. It needsflockfrom util-linux, and only has a timer resolution of 1s. Lock files fromnqandnq.shshould not be mixed.)
Installation
Usemake allto build,make installto install relative toPREFIX(/usr/localby default). TheDESTDIRconvention is respected. You can also just copy the binaries into yourPATH.
You can usemake checkto run a simple test suite, if you have Perl'sproveinstalled.