htsworkflow.git
16 years ago[project @ Method 2 for monitoring pipeline run (based on config_pipeline.py]
Brandon King [Fri, 16 Nov 2007 22:13:21 +0000 (22:13 +0000)]
[project @ Method 2 for monitoring pipeline run (based on config_pipeline.py]
 * Since the first version stalled in a mysterious way,
   I am trying a new method.
 * Now writes out standard out and standard error to file using
   file descriptors.
 * Also uses inotify to detect "finished" files.
   * FIXME: Need to do something about all other files (currently
     printed to stdout).
 * FIXME: Double check the run_dir exists in configure step.
 * FIXME: Upon run completion, read and process stderr output.

16 years ago[project @ Handling of missing app errors with running pipeline]
Brandon King [Fri, 16 Nov 2007 18:46:22 +0000 (18:46 +0000)]
[project @ Handling of missing app errors with running pipeline]
 WARNING: Still not fully functional, making progress through.
 * Prints out success or failure of run
   * FIXME: There is a weird BUG where towards the end of a run when
     the summary files are being generated that the pipeline code just
     stops processing. It's just sitting in memory not doing anything.
     The config_pipeline.py gets stuck on line = pipe.stdout.readline()
     at this point.
 * Checks for lack of ghostscript.

16 years ago[project @ Work toward pipeline monitor code (WARNING: testing version)]
Brandon King [Thu, 15 Nov 2007 21:16:47 +0000 (21:16 +0000)]
[project @ Work toward pipeline monitor code (WARNING: testing version)]
 * Warning, this code is a steping stone to monitoring a running pipeline.
   * It only runs with tiles=s_4_0100,s_4_0101,s_4_0102,s_4_0103,s_4_0104
   * There are also important FIXME that should be looked at before using this version of the code.
 * This patch is needed for all future patches to work.

16 years ago[project @ Runs pipeline on successful config + traceback]
Brandon King [Thu, 15 Nov 2007 04:04:07 +0000 (04:04 +0000)]
[project @ Runs pipeline on successful config + traceback]
 * Now runs the pipeline upon successful configuration step.
   * Currently no monitoring of output, just writing to files so
     I can figure out how to code the monitoring.
 * Now detects if goat crashes and displays a traceback.
 * Figured out another goat_pipeline.py call that triggers a
   traceback (included as commented out code) -> Test case for future.
 * FIXME: pipeline monitoring code needs to be written.

16 years ago[project @ Now handles stderr output.]
Brandon King [Wed, 14 Nov 2007 21:19:49 +0000 (21:19 +0000)]
[project @ Now handles stderr output.]

16 years ago[project @ Better handling of configuring pipeline w/ clear success or failure. Loads...
Brandon King [Wed, 14 Nov 2007 05:09:00 +0000 (05:09 +0000)]
[project @ Better handling of configuring pipeline w/ clear success or failure. Loads run_path info ConfigInfo object as well.]

16 years ago[project @ First attempt at script for configuring pipeline automatically.]
Brandon King [Wed, 14 Nov 2007 03:33:04 +0000 (03:33 +0000)]
[project @ First attempt at script for configuring pipeline automatically.]
 * More of prof of concept.
 * FIXME: hardcoded config script path.
 * Uses logging module
   * Info for everything going fine.
   * Error when it breaks in a bad way.

16 years ago[project @ Updated eland_config app to generate config file from fctracker app.]
Brandon King [Tue, 13 Nov 2007 02:14:12 +0000 (02:14 +0000)]
[project @ Updated eland_config app to generate config file from fctracker app.]
 * WARNING DB CHANGE: Added read_length, and advanced_run to flowcell
   model in fctracker app.
 * Updated eland_config app to generate config files from db instead of form.
   * Currently regenerates the config file on every request. Simple flag will
     cause it to re-read saved copy on disk.
 * Changed URL to http://<host>/eland_config/.

16 years ago[project @ Url should be <host>/eland_config/ in this case.]
Brandon King [Mon, 12 Nov 2007 22:43:35 +0000 (22:43 +0000)]
[project @ Url should be <host>/eland_config/ in this case.]

16 years ago[project @ The rest of the rename fix.]
Brandon King [Mon, 12 Nov 2007 22:36:44 +0000 (22:36 +0000)]
[project @ The rest of the rename fix.]

16 years ago[project @ Small fixes to changed directory name]
Lorian Schaeffer [Mon, 12 Nov 2007 22:17:34 +0000 (22:17 +0000)]
[project @ Small fixes to changed directory name]

16 years ago[project @ Renamed main django project dir to ga_frontend + updated settings.]
Brandon King [Mon, 12 Nov 2007 22:16:09 +0000 (22:16 +0000)]
[project @ Renamed main django project dir to ga_frontend + updated settings.]

16 years ago[project @ Massive change to DB structure; complete library table]
Lorian Schaeffer [Fri, 9 Nov 2007 23:20:37 +0000 (23:20 +0000)]
[project @ Massive change to DB structure; complete library table]
Primary change to the DB is the library table and supporting changes to the flowcell table.
They should both be properly linked now; you'll have to pull species information from the
linked library field. In addition, I added a common name to the Species table. Most interface
changes are via the meta and admin classes in each model, and are fairly straightforward. I
also added databrowse support; go to /databrowse instead of /admin to play with it.

16 years ago[project @ Finished statement + additional error message]
Brandon King [Sat, 10 Nov 2007 01:27:44 +0000 (01:27 +0000)]
[project @ Finished statement + additional error message]
 * Now says where it wrote the config file to upon success
 * Now puts more meaningful error message when user enters invalid domain/ip address.

16 years ago[project @ Updated to give more user friendly error:]
Brandon King [Sat, 10 Nov 2007 01:20:19 +0000 (01:20 +0000)]
[project @ Updated to give more user friendly error:]
 * Gives a more user friendly error when connection is refused.
 * FIXME: Should include other errors as well, such as host lookup errors.

16 years ago[project @ Improved help output:]
Brandon King [Sat, 10 Nov 2007 01:05:14 +0000 (01:05 +0000)]
[project @ Improved help output:]
 * Automattically displays help if called with no arguments.
 * Includes description and location of config files.
 * Include description of command line program.

16 years ago[project @ First working version of script which retrieves eland config file from...
Brandon King [Fri, 9 Nov 2007 23:45:20 +0000 (23:45 +0000)]
[project @ First working version of script which retrieves eland config file from ga_frontend.]

16 years ago[project @ Basic Library Table]
Lorian Schaeffer [Tue, 6 Nov 2007 16:34:29 +0000 (16:34 +0000)]
[project @ Basic Library Table]

16 years ago[project @ Eland Config Saver + FlowCell Tracker DB update]
Brandon King [Thu, 18 Oct 2007 19:22:00 +0000 (19:22 +0000)]
[project @ Eland Config Saver + FlowCell Tracker DB update]
 * Now saves a copy of eland config file for later retrieval.
   /elandifier/config/ <-- Index of available config files
   /elandifier/config/FC#####/ <-- Eland config for specific FC#
 * Save record of flow cell to database.
   * Form is also saved to database... which can be viewed
     in /admin/

16 years ago[project @ Prototype Web-based Eland Config generator]
Brandon King [Mon, 15 Oct 2007 22:36:48 +0000 (22:36 +0000)]
[project @ Prototype Web-based Eland Config generator]
 * Functional, but hard coded elandifier setting.py (needs updating before use).
 * Using Django SVN (will be Django 0.97).
 * Form currently returns to browser, will change in future.

16 years ago[project @ add notes about what we need to do to more fully automate the pipelien]
Diane Trout [Fri, 9 Nov 2007 23:18:46 +0000 (23:18 +0000)]
[project @ add notes about what we need to do to more fully automate the pipelien]

16 years ago[project @ handle IOErrors from xsend]
Diane Trout [Tue, 16 Oct 2007 22:53:43 +0000 (22:53 +0000)]
[project @ handle IOErrors from xsend]

16 years ago[project @ add script to launch copier daemon]
Diane Trout [Fri, 10 Aug 2007 01:19:23 +0000 (01:19 +0000)]
[project @ add script to launch copier daemon]

16 years ago[project @ make copier a bot with logging abilities]
Diane Trout [Fri, 10 Aug 2007 00:15:18 +0000 (00:15 +0000)]
[project @ make copier a bot with logging abilities]
this now imports benderjab and uses pythons logging module to keep
track of what rsync commands have been executed.

16 years ago[project @ start the copier script]
Diane Trout [Thu, 9 Aug 2007 01:30:34 +0000 (01:30 +0000)]
[project @ start the copier script]
this contains code to deal with launching rsync against the
rsync config over on jumpgate. it checks for directories matching
stuff the sequencer tends to generate and will launch rsync processes
against directories that look like that name. (Which might be excessive)

I still need to add the bot portion

16 years ago[project @ add __init__.py]
Diane Trout [Thu, 9 Aug 2007 01:29:19 +0000 (01:29 +0000)]
[project @ add __init__.py]
actually to make setup.py work you do really need the __init__.py
package flag

16 years ago[project @ make a spoolwatcher script]
Diane Trout [Thu, 9 Aug 2007 01:27:02 +0000 (01:27 +0000)]
[project @ make a spoolwatcher script]
and put most of the configuration information in the script
(i really should load it from a config file at some point)
Also teach setup.py about said script

16 years ago[project @ add a basic setup.py file]
Diane Trout [Wed, 8 Aug 2007 23:56:28 +0000 (23:56 +0000)]
[project @ add a basic setup.py file]

16 years ago[project @ Sequencer spool drive watcher]
Diane Trout [Wed, 8 Aug 2007 23:51:43 +0000 (23:51 +0000)]
[project @ Sequencer spool drive watcher]
spoolwatcher checks to see if a file named Run.completed has been
written to /gec/jumpgate/ext0 and if it has sends a message to
runner_jid. Additionally after (currently 10 seconds have passed without
data being written, after some data has been written, it'll send a
message top copy_jid)

16 years ago[project @ tools to handle the illumina sequencer workflow]
Diane Trout [Wed, 8 Aug 2007 21:41:25 +0000 (21:41 +0000)]
[project @ tools to handle the illumina sequencer workflow]

16 years agoThis directory will host the upstream sources
Diane Trout [Fri, 21 Dec 2007 01:53:12 +0000 (01:53 +0000)]
This directory will host the upstream sources