digarlab@uw.edu

Denny Hall 457

© 2015 Marcos Llobera. Design by Claire Kohler

Research > Submission creator

HTCondor Submission Creator

The purpose of this project was to create a user interface that would allow the user to quickly create a submission file of the format needed to submit jobs to an instance of HTCondor. High throughput computing systems like HTCondor can allow archaeologists to ask questions of large datasets that are impractical (or impossible) on a single laptop or desktop computer. This is especially true of computationally intensive analyses involving large geographic datasets. The user can break a big job into chunks and use a system like HTCondor to distribute the work across many different machines. In order to do this, however, HTCondor requires a specially-formatted text file (the submission file) that contains system parameters and the details of each small job that it will distribute to the various machines.

This submission creator is a simple Python script that initializes a user interface to help the user create a simple submission file. It was designed for a very basic implementation of HTCondor and thus won’t suit all purposes, but it can definitely eliminate wasted time in just setting up the jobs. It is often the case that submission files will contain hundreds or even thousands of separate jobs, so entering each one manually is out of the question and often requires writing a separate script just to create the submission file. This submission creator simply generalizes that process so that it can be used with many different projects.

 

If you’d like to try out the submission creator, contact Jake Deppen.

Project Team