Python 并行处理框架,Jug 0.9.1 发布
openkk 12年前
<p>jug 0.9.1 发布,该版本主要是小改进和 bug 修复。</p> <p>Jug 是一个基于任务的并行处理框架,采用 Python 编写,可用来在不同的机器上运行同一个任务,使用 NFS 做文件系统的通讯。</p> <ul class="simple"> <li>Persistent data across runs</li> <li>Re-use partial results if you change the algorithms (for example, if you search over a few more parameters for the best, then it will reuse the pre-computed values). Normally, I have a main computation script and then write a second visualisation script to plot out the results or compute some summary statistics and it's good if the second script is easy to write, easy to change, and reuses all computational results seamlessly.</li> <li>Supports concurrency with a very flexible system: CPUs can join the computation at any time. This allows it to be used in batch processing systems.</li> <li>You can check up on the status of the computation at any time (<cite>jug status</cite>)</li> <li>Two backends: file-based if all the processors share a filesystem (works over NFS too) or <a class="reference external" href="/misc/goto?guid=4958343814961817005">redis</a> based if they can all connect to the same redis server.</li> </ul> <p>示例代码:</p> <pre class="brush:python; toolbar: true; auto-links: false;">from jug import TaskGenerator from time import sleep @TaskGenerator def is_prime(n): sleep(1.) for j in xrange(2,n-1): if (n % j) == 0: return False return True primes100 = map(is_prime, xrange(2,101))</pre> <p></p>