Brandon King [Tue, 11 Aug 2009 19:53:30 +0000 (19:53 +0000)]
Script for conversion of db; start of fixes brought on by changes to the db.
Brandon King [Mon, 10 Aug 2009 22:33:58 +0000 (22:33 +0000)]
print killing wsgi; minor fix.
Brandon King [Fri, 7 Aug 2009 22:43:06 +0000 (22:43 +0000)]
New Lane Table migration progress.
Brandon King [Thu, 6 Aug 2009 22:35:56 +0000 (22:35 +0000)]
Barcode Magic Initial Data
Brandon King [Tue, 4 Aug 2009 19:09:21 +0000 (19:09 +0000)]
Minor update.
Brandon King [Fri, 24 Jul 2009 22:22:03 +0000 (22:22 +0000)]
Lorian supplied quick fix for 32 char UUIDs.
Brandon King [Thu, 23 Jul 2009 21:30:54 +0000 (21:30 +0000)]
Lorian requested item search box
Diane Trout [Thu, 23 Jul 2009 20:01:32 +0000 (20:01 +0000)]
ReadPreps can also use the netcopy complete flags which causes problems
so I needed to force the "completion file" to be only checked in the root
of the runfolder.
Brandon King [Tue, 21 Jul 2009 22:53:42 +0000 (22:53 +0000)]
Moved from ext-3.0-rc2 to ext-3.0.0.
Diane Trout [Tue, 21 Jul 2009 22:23:21 +0000 (22:23 +0000)]
Allow specifying the we're finished sequencing file in the config file.
(Currently the default is netcopy_complete.txt, but that might not be a good
choice).
While I was in there I changed it to catch the current event, and set a
completed flag in the event instead of just directly sending the
sequencingFinished message.
I also put some effort into trying to only send startCopy messages for
things that look like runfolders, both by not sending a copy message
for the root of the watch directory and by checking to see if the
top level directory that contains the event looks kind of like a runfolder
name. (AKA "[0-9]{6}_" )
Brandon King [Mon, 20 Jul 2009 18:36:29 +0000 (18:36 +0000)]
Lorian requested save_as feature in admin interface.
Brandon King [Mon, 13 Jul 2009 23:37:10 +0000 (23:37 +0000)]
Prototype of bcmagic search plugin.
* Works for inventory item object.
Brandon King [Sat, 11 Jul 2009 00:03:01 +0000 (00:03 +0000)]
Adding ReagentLibrary & ReagentFlowcell tables in Inventory App.
Brandon King [Tue, 7 Jul 2009 00:23:31 +0000 (00:23 +0000)]
Attempted linking to archived hard drive.
Brandon King [Tue, 7 Jul 2009 00:20:56 +0000 (00:20 +0000)]
Attempted linking to archived hard drive.
Brandon King [Mon, 6 Jul 2009 23:46:14 +0000 (23:46 +0000)]
Update to item summary now displays Long Term Storage information if exists.
Brandon King [Mon, 6 Jul 2009 22:12:53 +0000 (22:12 +0000)]
Updated inventory to support display of inventory items based on uuid/barcode_id scans.
Brandon King [Thu, 2 Jul 2009 01:29:34 +0000 (01:29 +0000)]
Made Diane's suggest json problem fix for python 2.5/2.6 support... now using simplejson for <= python 2.5.
Brandon King [Thu, 2 Jul 2009 00:31:25 +0000 (00:31 +0000)]
Added a subtle but useful | to the toolbar
Brandon King [Thu, 2 Jul 2009 00:27:17 +0000 (00:27 +0000)]
Added a subtle but useful | to the toolbar
Brandon King [Thu, 2 Jul 2009 00:22:52 +0000 (00:22 +0000)]
NOTE: settings.py update:
Will need to include a section as follows to ini file:
[bcprinter]
printer1_host=<ip_address>
printer1_port=9100
printer2_host=<ip_address>
printer2_port=9100
Added a simple way to print out a label on inventory item summary
page. Needs major improvement on this page, but went for getting
labels printed to start with.
Brandon King [Wed, 1 Jul 2009 23:10:00 +0000 (23:10 +0000)]
Disabled menu selection when not logged in.
Brandon King [Wed, 1 Jul 2009 19:43:02 +0000 (19:43 +0000)]
Added start of inventory summary view and cleaned up inventory template location
Diane Trout [Wed, 1 Jul 2009 00:35:07 +0000 (00:35 +0000)]
Add LibraryType option, (this will probably require a dump/reload)
And you'll probably need to do it before updating to this version of the
database
Diane Trout [Wed, 1 Jul 2009 00:33:56 +0000 (00:33 +0000)]
Change the cluster station default to the current working cluster station.
Diane Trout [Tue, 30 Jun 2009 23:45:19 +0000 (23:45 +0000)]
Allow cellline and conditition to be blank [ticket:115] [ticket:105]
However this requires a dump and reload of the database to work.
This saves the data as a django 'fixture'
python manage.py dumpdata samples > /tmp/sampledata.json
python manage.py sqlreset samples | sqlite3 [path to db]
python manage.py loadddata /tmp/sampledata.json
Brandon King [Tue, 30 Jun 2009 22:43:48 +0000 (22:43 +0000)]
Auto resizing for grids based on collapsing and resizing of other components
Diane Trout [Tue, 30 Jun 2009 22:29:05 +0000 (22:29 +0000)]
Provide a dummy eland result lane class if there's nothing appropriate in the xml
files. Though this produces blank records in the library page, at least the
program doesn't crash.
Brandon King [Tue, 30 Jun 2009 20:03:16 +0000 (20:03 +0000)]
Behold the power of ExtJS grids! Muhahaha! =o)
Brandon King [Tue, 30 Jun 2009 19:06:56 +0000 (19:06 +0000)]
auto height test.
Brandon King [Tue, 30 Jun 2009 19:04:26 +0000 (19:04 +0000)]
two json modules... one for 2.5, one for 2.6... util that just works.
Brandon King [Tue, 30 Jun 2009 18:26:29 +0000 (18:26 +0000)]
Start of inventory app page.
Brandon King [Mon, 29 Jun 2009 21:38:44 +0000 (21:38 +0000)]
Activating bcmagic model & initial data.
Brandon King [Sat, 27 Jun 2009 01:59:13 +0000 (01:59 +0000)]
Pulling in barcode magic from sample tracker and starting to update it to use ExtJS
Brandon King [Fri, 26 Jun 2009 22:23:18 +0000 (22:23 +0000)]
Libraries Detail View integrated.
Brandon King [Fri, 26 Jun 2009 19:57:07 +0000 (19:57 +0000)]
Fixed dynamic east panel code
Brandon King [Fri, 26 Jun 2009 19:35:03 +0000 (19:35 +0000)]
CSS fix broke non-library index views; fix for fix.
Brandon King [Fri, 26 Jun 2009 19:31:47 +0000 (19:31 +0000)]
CSS fixes for library index
Brandon King [Fri, 26 Jun 2009 19:13:23 +0000 (19:13 +0000)]
Migrating the west panel menu to a 2nd toolbar menu.
Brandon King [Thu, 25 Jun 2009 18:32:15 +0000 (18:32 +0000)]
CSS fixes
Brandon King [Wed, 24 Jun 2009 22:55:55 +0000 (22:55 +0000)]
Made a dynamic way of adding div dynamically to the east panel.
* see app_base.html template for example.
Brandon King [Wed, 24 Jun 2009 21:51:59 +0000 (21:51 +0000)]
Sample Tracker EXTJS frontend merge part 1
Diane Trout [Tue, 23 Jun 2009 23:46:49 +0000 (23:46 +0000)]
The url validation code wasn't working on python2.4
the problem where rpc exception are being eaten by something is getting
really annoying.
Brandon King [Mon, 22 Jun 2009 23:18:30 +0000 (23:18 +0000)]
Schema update to inventory
Diane Trout [Sat, 20 Jun 2009 00:06:44 +0000 (00:06 +0000)]
Send the specific directory that needs to be copied in the startCopy message.
Also I changed it to send "urls" which need to match in both spoolwatcher
and copier. (as security feature it checks to make sure the rsync
directory it got over the network matches one of its internal list)
Diane Trout [Fri, 19 Jun 2009 22:49:16 +0000 (22:49 +0000)]
I started using ResultLane, so stop treating it like an abstract class.
I needed to initialize the ELAND class with something for the older
run xml files which don't have eland results initialized, So I initialized
some default values so the website wouldn't crash when accessing
these older files.
Diane Trout [Wed, 17 Jun 2009 19:36:00 +0000 (19:36 +0000)]
Add support for extracting data out of Illumina's new RTA runfolder.
I dropped extraction of matrix & phasing for the moment, no one was
looking for them, and I need to go back to one matrix per lane
for RTA Intensity directories. I'm also not sure where the phasing
information might be being stored.
Another weakness is if there is a firecrest directory the run scanning
wasn't finding the intensities directory.
Diane Trout [Wed, 17 Jun 2009 19:25:34 +0000 (19:25 +0000)]
Add a hack to add a frontend section if not present
This will allow the django test code to run on systems that aren't
setup to run the web site.
Diane Trout [Wed, 17 Jun 2009 19:24:29 +0000 (19:24 +0000)]
Loading really old run xml files caused the website to crash
with an attribute error.
frontend/samples/views.py line 220 had
eland_summary = run.gerald.eland_results.results[end][lane_id]
However really old run.xmls lacked any eland_results so
I added some code to add dummy entries if the Eland class
wasn't initialized.
Also this adds a bit of test code which at least tries to load
a couple run xml files.
Diane Trout [Wed, 17 Jun 2009 19:10:39 +0000 (19:10 +0000)]
Ignore *.py[co~] files to make svn status happier
Diane Trout [Wed, 17 Jun 2009 19:09:50 +0000 (19:09 +0000)]
Add stub uuid function to make this module import when running under python2.4
Diane Trout [Wed, 17 Jun 2009 19:07:23 +0000 (19:07 +0000)]
remove spurious debug print statements
Brandon King [Wed, 17 Jun 2009 18:58:12 +0000 (18:58 +0000)]
Arg... spelling!
Brandon King [Wed, 17 Jun 2009 18:57:31 +0000 (18:57 +0000)]
Once again, assumed I had a flowcell object when I actually had a dictionary!
Brandon King [Wed, 17 Jun 2009 18:39:42 +0000 (18:39 +0000)]
Apparently we might be receiving None, so this check should fix it.
Brandon King [Wed, 17 Jun 2009 18:14:47 +0000 (18:14 +0000)]
* Fixed urls for result_fc_cnm_eland_lane and summaryhtm_fc_cnm functions.
* Added listing of hard drives the results have been archived to on the "Raw Result Files" table.
Brandon King [Tue, 16 Jun 2009 22:54:43 +0000 (22:54 +0000)]
Updated Library view to show an icon if a record exists showing that the library has been archived.
Diane Trout [Tue, 16 Jun 2009 22:20:35 +0000 (22:20 +0000)]
The error being thrown by benderjab changed slightly.
Diane Trout [Tue, 16 Jun 2009 22:19:05 +0000 (22:19 +0000)]
Split rsync initialization out of run so we can test
it better. (needed to support the previous changes to test/test_copier.py
Diane Trout [Tue, 16 Jun 2009 21:55:27 +0000 (21:55 +0000)]
Fixed a couple of problems.
The name/description wasn't being updated.
there was a typo in the name of the outstream.
Diane Trout [Tue, 16 Jun 2009 21:36:59 +0000 (21:36 +0000)]
Update test copier code for the changes in the copier config file.
(Support for multiple source destinations)
Brandon King [Tue, 16 Jun 2009 19:46:02 +0000 (19:46 +0000)]
Icon added.
Removed unneed urllib module import (since we switched to urllib2).
Brandon King [Tue, 16 Jun 2009 18:58:13 +0000 (18:58 +0000)]
Managed to use a urllib python 2.6 feature... converting code to work in 2.5
Brandon King [Tue, 16 Jun 2009 18:25:07 +0000 (18:25 +0000)]
Minor package update.
Brandon King [Tue, 16 Jun 2009 18:22:39 +0000 (18:22 +0000)]
Updated installtion script:
* Was missing some newer django apps
* Was missing mark_archived_data script
Diane Trout [Mon, 15 Jun 2009 19:48:20 +0000 (19:48 +0000)]
Allow searching by library name on the experiments/flowcell page
Brandon King [Wed, 10 Jun 2009 20:13:13 +0000 (20:13 +0000)]
Improved debugging output.
* added a -v/--verbose option
* Report if a record already existed and therefore did not need to be updated
Fixed a minor logic bug... wasn't actually causing problems.
Brandon King [Wed, 10 Jun 2009 17:53:51 +0000 (17:53 +0000)]
Moved hardcoded URL to it's proper place in the settings.py.
* Supports Diane's config file feature. =o)
Brandon King [Wed, 10 Jun 2009 17:44:47 +0000 (17:44 +0000)]
Track which flowcells are archived on which long term storage devices
* Added a command_line script called mark_archived_data
* Storage Device must be in DB as well as flowcell before using
* If you supply a device (i.e. /dev/sda), it will extract the serial number of the drive for you.
* Uses HTTP REST request to link Flowcell, Libraries and Storage Devices.
* Supports archives across storage devices.
* device query requires py_sg python package (easy_install py_sg)
Diane Trout [Wed, 10 Jun 2009 00:51:49 +0000 (00:51 +0000)]
Scan more than one rsync repository for directory trees to copy.
Also don't configure the rsync class until the bot.run function
to avoid problems with logging while daemonized.
Also add some code to make sure that we have a trailing /
Diane Trout [Tue, 9 Jun 2009 00:44:18 +0000 (00:44 +0000)]
Add code to construct a copy url using a mapping in the config file and
the top level directory that had writes in it.
Brandon King [Fri, 5 Jun 2009 18:00:50 +0000 (18:00 +0000)]
Minor updates to inventory schema based on input from Lorian.
Diane Trout [Fri, 5 Jun 2009 00:32:00 +0000 (00:32 +0000)]
Pyinotify behaves oddly when the stdio file descriptors are closed.
so don't initialize it until after the daemonize code has been
called. this means after BenderJab.start has been called.
(So I changed from SpoolWatcher.start to SpoolWatcher.run
and moved the watch manager configuration into the
start of run)
Brandon King [Thu, 4 Jun 2009 22:55:37 +0000 (22:55 +0000)]
A simple command to request the hard drive serial number from a device.
* Used py_sg (easy_install py_sg)
Diane Trout [Mon, 1 Jun 2009 17:57:12 +0000 (17:57 +0000)]
Consistently use 'watchdir' instead of 'watch_dir' now.
Diane Trout [Sat, 30 May 2009 01:07:08 +0000 (01:07 +0000)]
Watch more than one directory tree for modification.
I also added code to record which directory watch, and what entry in that
directory was being created (AKA whatever element of the runfolder is being
touched, I record the root of the runfolder directory).
To support this I changed the config option in the benderjab file from
watchdir to watchdirs to make it a bit clearer that things have changed.
I still need to work out the communication protocol to copier so it
can figure out what to start copying. Also I'm recording the
watchdirectory, but copier needs access to a different url. So there's
some question about where the right place to map watchdir to copy url source
might be.
Diane Trout [Sat, 30 May 2009 01:01:49 +0000 (01:01 +0000)]
Don't print the discovered mount point.
Brandon King [Fri, 29 May 2009 23:00:04 +0000 (23:00 +0000)]
First attempt at an inventory tracking database
* Includes a LongTermStorage object which links flowcells/libraries to storage items.
Diane Trout [Wed, 27 May 2009 23:21:42 +0000 (23:21 +0000)]
Add a minimal test case to test a small bit of the django app.
(I needed to figure out how to do testing for django).
there's a tool nosedjango from http://hg.assembla.com/nosedjango
that adds auto-detection of the django settings file to
nose.
I needed to modify it for our app though.
Diane Trout [Mon, 11 May 2009 20:06:31 +0000 (20:06 +0000)]
More consistently split the flowcell id and status messages
some flowcells weren't being found because their names were
"flowcell (status)" which meant the code that was trying to find
the run.xml file in the results/flowcells directory was looking
for a directory named "results/flowcell_id (status)" which
wasn't so effective.
We really need a status field
Diane Trout [Mon, 11 May 2009 20:03:16 +0000 (20:03 +0000)]
Fix parameter list for base class.
I was passing in the ElementTree xml into the base class constructor instead of
an integer representing which 'end' the sample is from.
Which when working with single ended ElandLanes ended up with the ElementTree node
instead of none (which showed up in the UI).
Diane Trout [Wed, 6 May 2009 22:56:29 +0000 (22:56 +0000)]
Update htsworkflow.util.fctracker module to work with the newer django db
this was needed to support the make-library-tree script
Diane Trout [Tue, 5 May 2009 19:21:30 +0000 (19:21 +0000)]
Update the django app to report "raw reads" for a sequence lane.
I still need to figure out how to allow downloading of the
"result" files.
Diane Trout [Tue, 5 May 2009 18:02:20 +0000 (18:02 +0000)]
Handle lanes that were only sequenced.
So the report needs to be a bit smaller, and I need to archive a different file
Also this version only counts the number of records in the "sequence" file
which is raw sequence that passed the QC filter, I don't have a uniform way
of determining how many total sequences thre were.
Diane Trout [Tue, 5 May 2009 17:59:03 +0000 (17:59 +0000)]
Return NaN, NaN if a mean +/- deviation value was undefined in Summary.htm
Diane Trout [Tue, 5 May 2009 17:56:11 +0000 (17:56 +0000)]
Allow overriding the queued commands environment.
Diane Trout [Fri, 1 May 2009 00:22:17 +0000 (00:22 +0000)]
Add wait_for_ipar option to the spool watcher config which will wait until
"IPAR_Netcopy_Complete.txt" is written before sending the "sequencing finished
message.
Diane Trout [Thu, 30 Apr 2009 22:44:34 +0000 (22:44 +0000)]
Allow mapping library IDs to the library primary key
Diane Trout [Fri, 24 Apr 2009 23:37:35 +0000 (23:37 +0000)]
add --clean option to runfolder
this deletes the various log files as well as the images and calibration
directories.
Also it runs make clean_intermediate in the firecrest/ipar directory
(if the makefile exists)
Perhaps it should delete the bustard & gerald directories?
Or should I delete them after I've archived the useful parts.
Diane Trout [Fri, 24 Apr 2009 23:34:13 +0000 (23:34 +0000)]
Allow picking runfolder variant we should be trying to extract srf files from.
adds an option -r to chose which version of the illumina2srf or solexa2srf
programs
0 is solexa2srf
1 is illumina2srf -V1
2 is illumina2srf (current pipeline 1.3.2 version)
Diane Trout [Wed, 8 Apr 2009 00:47:34 +0000 (00:47 +0000)]
Stop assuming that library_id == pk.
I needed to fix the get_absolute_url to lookup by library_id,
we needed a redirect function to get to the right admin page.
And for sequence runs I don't have any eland results so the get result files
failed because it was assuming that the eland_result from the run xml would
always have all 8 lanes defined in its dictionary. Now I check to see if its
defined and if its not I don't display anything.
Diane Trout [Sat, 4 Apr 2009 00:41:40 +0000 (00:41 +0000)]
Don't crash if we can't find the flowcell run xml file
Diane Trout [Sat, 4 Apr 2009 00:36:43 +0000 (00:36 +0000)]
Display the affiliations on the public library detail page.
This involved cleaning up some of the view code to push all of the html
generation into the template and none in the view function, by modifying
the view function to feed everything to the template in the form of
variables.
Diane Trout [Wed, 1 Apr 2009 00:49:19 +0000 (00:49 +0000)]
Show the "Amplified from sample" as the parent library id and allow
searching on that on the public library index.
Additionally detect if we have an eland_multi or eland extended file
and make the url to download those different files. (and change the
link name slightly to reflect you're getting a different file.)
finally don't bother showing the bed downloader for newer file types
as the bed convertor doesn't work on them.
Diane Trout [Tue, 31 Mar 2009 01:33:27 +0000 (01:33 +0000)]
Use the image directory name to detect what type of image dir we have,
firecrest or ipar.
Also remove the pesky return that was preventing --extract-results from
actually generating the compressed eland files.
And finally change some of the logging to logging.info instead of debug
as there wasn't currently a way to generate logging.debug messages.
Diane Trout [Tue, 31 Mar 2009 01:30:40 +0000 (01:30 +0000)]
Don't attempt to save xml for a matrix in firecrest if it wasn't created.
since we're now creating it in bustard for greater than pipeleine 1.1
Diane Trout [Tue, 31 Mar 2009 01:29:12 +0000 (01:29 +0000)]
remove some useless debugging print statements
Diane Trout [Tue, 31 Mar 2009 01:28:14 +0000 (01:28 +0000)]
provide an error message if no run is found with the --use-run option
Diane Trout [Mon, 30 Mar 2009 23:14:26 +0000 (23:14 +0000)]
Add unit test to check that the crosstalk matrix was being saved and reloaded
Aparently if your unit test fails to check something there's no guarantee
that it happens.
Also actually add an implementation for saving the matrix.
Diane Trout [Mon, 30 Mar 2009 22:57:41 +0000 (22:57 +0000)]
IPAR detection is more reliable than firecrest so do it first, and then
check for firecrest second