Diane Trout [Mon, 9 May 2011 22:05:49 +0000 (15:05 -0700)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Mon, 9 May 2011 22:05:32 +0000 (15:05 -0700)]
Move default_pm setting to a little more predictable of a location
Diane Trout [Mon, 9 May 2011 21:54:47 +0000 (14:54 -0700)]
Keep moving settings around until I can find a place that tests on multiple machines
Diane Trout [Mon, 9 May 2011 21:54:13 +0000 (14:54 -0700)]
Add utility program to help find what exists in georgi's submission dirs
Diane Trout [Mon, 9 May 2011 21:53:17 +0000 (14:53 -0700)]
Report missing directory names to help diagnose sequence scanning problems
Diane Trout [Mon, 9 May 2011 21:52:04 +0000 (14:52 -0700)]
Validate fastq files in both phred33 & phred64 versions
Do not validate some imaginary fasta/fastq hybrid that I imagined
late some night.
I needed to add a parameter to pick which fastq format is in use.
Diane Trout [Fri, 6 May 2011 22:44:43 +0000 (15:44 -0700)]
Make a slightly less confusing 404 error message
Diane Trout [Mon, 18 Apr 2011 22:54:44 +0000 (15:54 -0700)]
Be more flexible with fastq filenames.
Allow pass/nopass, all, or no filter flag.
Also add more information to exception text
Diane Trout [Wed, 6 Apr 2011 20:49:26 +0000 (13:49 -0700)]
django-nose didn't work with django 1.1, so I went back to the nose plugin NoseDjango
I needed to move the settings file around a bit so the default build of
nosedjango will find it (instead of my previously hacked copy).
I got tired of modifying the location of the settings files and
discovered that django has a standardized way of grabbign the config
file. with "from django.conf import settings", though that will
only show uppercased config entries.
Diane Trout [Tue, 5 Apr 2011 17:10:17 +0000 (10:10 -0700)]
Adjust django driver script to test the whole package.
I moved settings & manage up a bit in the tree, and added a dependency
on python-django-nose which uses nose for the django test runner.
This version is using the class based runner, which appears
to require django 1.2. I'm not sure what's on the cluster machines
right now.
Diane Trout [Tue, 5 Apr 2011 05:37:41 +0000 (22:37 -0700)]
Revert a bad change. Apparently past me knew what I was doing,
and I needed to manually construct the dictionary for feeding
a json object, instead of just handing the json.dumps function a
random django object.
unlike the RDFa template which can just operate on the django ORM object.
Diane Trout [Tue, 5 Apr 2011 05:36:57 +0000 (22:36 -0700)]
remove spurious debugging print
Diane Trout [Sat, 2 Apr 2011 01:20:59 +0000 (18:20 -0700)]
Remove spurious debug print statements
Diane Trout [Sat, 2 Apr 2011 01:18:37 +0000 (18:18 -0700)]
Fix a typo inserted into the template.
Apparently a {% trans "value" %} needs to be on one line.
This didn't show up until I'd provided the get_absolute_url reference
for flowcells.
Diane Trout [Sat, 2 Apr 2011 01:18:12 +0000 (18:18 -0700)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Sat, 2 Apr 2011 01:13:59 +0000 (18:13 -0700)]
Add pages to show information about a particular flowcell.
-- in both human and RDFa formats.
Also I implemented get_absolute_url for flowcells & lanes, which
means "view on site" pops up on the flowcell admin page.
Also with modifying the templates for RDFa, many more elements
on the library, flowcell, and flowcell lane detail pages link to
each other.
I moved the table formatting css out of the library page
and into app.css.
I also switched experiments/views.py from using Context to RequestContext
which provides the default media url to the template.
There were a few changes to samples/views.py to get the more informative
object to the template instead of our on-the-fly created lists.
Diane Trout [Wed, 30 Mar 2011 21:31:22 +0000 (14:31 -0700)]
Update single vs paired guessing code
Add in new guesses for Georgi files to view names
Diane Trout [Wed, 9 Mar 2011 00:14:59 +0000 (16:14 -0800)]
Update spoolwatcher to 0.2 benderjab's logging api
Diane Trout [Mon, 7 Mar 2011 23:33:19 +0000 (15:33 -0800)]
Move some common runfolder path management code into its own module
I discovered I had two different functions for checking if something
was a runfolder name. I put them in once place and linked the
two together.
Then I cleaned up the test code that was calling them.
Diane Trout [Mon, 7 Mar 2011 23:31:51 +0000 (15:31 -0800)]
Add a place for "breadcrumbs" back into the base.html page so
navigation links appear on the started email page
Diane Trout [Sat, 26 Feb 2011 00:10:26 +0000 (16:10 -0800)]
Ignore emacs backup files for things other than python files
Diane Trout [Sat, 26 Feb 2011 00:07:41 +0000 (16:07 -0800)]
Save the overrides to the RDF model.
They're designed to be loaded into the model before we start pulling data
from UCSC and our website.
Currently the file just defines submission to library URN/URL mappings
for the submissions whose submission name doesn't include the correct
library ID.
Diane Trout [Fri, 25 Feb 2011 23:46:15 +0000 (15:46 -0800)]
Only pull web data when its missing from the rdf model.
This involved alterning the RDF model.
I changed the namespace to something a bit more discriptive.
The information grabbed from the ddf is now attached to the submission
status node, as changing the ddf will change the status.
Diane Trout [Thu, 17 Feb 2011 23:47:17 +0000 (15:47 -0800)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Thu, 17 Feb 2011 23:44:11 +0000 (15:44 -0800)]
Major updateds to encode_find for extracting encode cell line life cycle.
Its still not done, but I'm going to rip out more and wanted to save
my current progress.
Also I abstracted out the optparse options for setting up htswapi
access into htsworkflow.util.api
Diane Trout [Thu, 17 Feb 2011 23:38:19 +0000 (15:38 -0800)]
Only use one name for dates on the page.
Also the django database uses a date field for creation date
and wont let you format it to an isoformatted timestamp
so I hacked in the time portion into the RDFa field.
Diane Trout [Thu, 17 Feb 2011 19:51:06 +0000 (11:51 -0800)]
Minor RDFa cleanups
I changed the XMLSchema prefix to the example used by the W3c "xsd"
instead of the random other sites example "xmls".
More importantly the content tag is content and not contents so
the places I was trying to replace the human readable value with
a standardized number or date were being ignored.
Lastly I figured out how to get django 1.1 to spit out a
XMLSchema#dateTime value. (which is not XMLSchema#datetime value)
Diane Trout [Thu, 17 Feb 2011 19:00:00 +0000 (11:00 -0800)]
Change default font size to 14pt
So many old
Diane Trout [Thu, 17 Feb 2011 08:39:54 +0000 (00:39 -0800)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Thu, 17 Feb 2011 08:35:44 +0000 (00:35 -0800)]
Add RDFa data to our library pages.
While doing that I discovered that the library_index page was spitting up
seriously bad HTML, with two DOCTYPE & html tags.
It looks like it was a combination of my simplifications with the
leftovers of brandon's "app" templates I ended up putting in two
base.html templates.
I stripped out the app templates and tried to simplify the template
inheritance.
As a side effect the library detail page, now has a header and is using
a css based font.
There was also a minor issue where I discovered the paginator was showing
"X libraries libraries" because the views was adding "libraries" to
the changelist result count. (And the change list also adds in the name).
I needed to replace the default admin pagination template, as
RDFa uses xhtml, and xhtml doesn't support the entity.
(The solution is to use a unicode character for nbsp -- such as
  -- unicode for non-breaking space.
Diane Trout [Thu, 17 Feb 2011 08:14:51 +0000 (00:14 -0800)]
Add a small comment defining "results_dir"
because I confused myself
Diane Trout [Wed, 16 Feb 2011 19:06:34 +0000 (11:06 -0800)]
Add in patterns for Encode Elements submission
Diane Trout [Tue, 15 Feb 2011 07:28:58 +0000 (23:28 -0800)]
Starting component to try and figure out what we've submitted.
Diane Trout [Tue, 15 Feb 2011 00:52:35 +0000 (16:52 -0800)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Tue, 15 Feb 2011 00:49:46 +0000 (16:49 -0800)]
Make plus/minus bigwig filename extension a bit more lax.
*plus.bigwig vs *.plus.bigwig
Diane Trout [Mon, 14 Feb 2011 18:42:51 +0000 (10:42 -0800)]
Add in support for generating md5sums for files
Diane Trout [Mon, 7 Feb 2011 23:52:21 +0000 (15:52 -0800)]
Add in a little bit of qseq2fastq testing code. More is still needed.
This one just tests the slice parsing code
Diane Trout [Mon, 7 Feb 2011 23:51:05 +0000 (15:51 -0800)]
Add documentation about which Phred format we're generating
Diane Trout [Thu, 3 Feb 2011 19:50:12 +0000 (11:50 -0800)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Thu, 3 Feb 2011 02:05:29 +0000 (18:05 -0800)]
First cycle can be 2 digit.
Diane Trout [Tue, 25 Jan 2011 01:16:09 +0000 (17:16 -0800)]
Fix typo in a function name.
(Silly character bleed through from a different application)
Diane Trout [Tue, 25 Jan 2011 00:52:39 +0000 (16:52 -0800)]
Add in 12x / 18x PCR protocol end points from Lorian
Diane Trout [Tue, 25 Jan 2011 00:49:22 +0000 (16:49 -0800)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
root [Tue, 25 Jan 2011 00:48:46 +0000 (16:48 -0800)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Tue, 25 Jan 2011 00:48:10 +0000 (16:48 -0800)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Fri, 21 Jan 2011 23:23:29 +0000 (15:23 -0800)]
Update view names as per Cricket for the Jan 21st encode submission.
Also we're avoiding submitting the GENCODE v4 models for this
submission so I commented them out
Diane Trout [Fri, 21 Jan 2011 22:10:36 +0000 (14:10 -0800)]
Merge branch 'master' of gitosis@mus.cacr.caltech.edu:htsworkflow
Diane Trout [Fri, 21 Jan 2011 22:07:49 +0000 (14:07 -0800)]
Add function to ucsc submission tool to make symlink tree.
I needed a way to read data files from one tree and build archives in
another.
This also changed the condor scripts to tell tar to read the contents
of the file being pointed at by the symlink.
In addition there were slight additions and loosenings of the
file name to ucsc view matching patterns. (As Georgi slightly
changed his names.)
root [Fri, 14 Jan 2011 23:14:15 +0000 (15:14 -0800)]
Make replicate field visable on the public page.
We might want to change the default value for replicate to "None"
Diane Trout [Wed, 12 Jan 2011 22:40:25 +0000 (14:40 -0800)]
Include version.py in source distribution for setup.py
Diane Trout [Sat, 8 Jan 2011 01:22:36 +0000 (17:22 -0800)]
Add validation script
Current version just validates illumina phred64 fastq files.
But I can add more later.
Diane Trout [Tue, 4 Jan 2011 21:29:27 +0000 (13:29 -0800)]
Rename all the scripts to start with htsw-
In preparation for making a debian package I thought
I should try to limit my pollution of the command namespace.
so now everything that's actually an installed script starts
with htsw-. There's one left-over that still might be useful
in the future but isn't currently that was left with its old
name.
Diane Trout [Tue, 21 Dec 2010 23:53:59 +0000 (15:53 -0800)]
Add utility to fix the off-by-3 error in the amplified from field.
The bug appeared because the library name and library primary key
drifted apart because of some accidental deletions.
I eventually moved us back to using raw primary keys as that
made constructing django urls simplier & easier. However
I'd apparently forgotten to adjust the amplified from field.
Diane Trout [Thu, 2 Dec 2010 01:19:07 +0000 (17:19 -0800)]
Make the inventory login page look a bit nicer.
This involved adjusting some of the base django templates.
Whenever given a choice I try to make our pages look more like the
admin site.
Diane Trout [Thu, 2 Dec 2010 01:18:05 +0000 (17:18 -0800)]
Adjust some whitespace
Diane Trout [Wed, 1 Dec 2010 00:26:51 +0000 (16:26 -0800)]
This mildly adjusts the css to be a bit prettier
Diane Trout [Wed, 1 Dec 2010 00:15:43 +0000 (16:15 -0800)]
Add a new ini-file option to point to the flowcell repository.
It's in the [frontend] section called "results_dir".
Diane Trout [Tue, 30 Nov 2010 19:45:47 +0000 (11:45 -0800)]
Add config option notification_bcc for always attaching to notify emails.
Its called notification_bcc in the htsworkflow.ini file and
NOTIFICATION_BCC in the django settings.py file.
It ignores the "send bcc" option on the email_started form.
Diane Trout [Wed, 17 Nov 2010 05:39:25 +0000 (21:39 -0800)]
Cache the attributes for each type of fastq file.
Previously it was just using whatever was last as the
set of attributes. Which led to everything being flagged as
FastqRd2
Diane Trout [Tue, 16 Nov 2010 01:41:04 +0000 (17:41 -0800)]
Map junctions.bed to the Junctions view.
Hopefully it'll pass validation and not be treated as a bed file,
even though it ends in the .bed extension.
(There were lots of problems with a previous subission because the
cufflinks bed file was almost but not quite a match to ucsc's bed
file validator. Or maybe they caved on the definition of a score.)
Diane Trout [Tue, 16 Nov 2010 01:40:11 +0000 (17:40 -0800)]
Autodetect location of *2fastq scripts
Uses the .__file__ variable of the python modules.
Diane Trout [Mon, 15 Nov 2010 22:59:57 +0000 (14:59 -0800)]
Don't count failed flowcells when guessing if a library is paired or single ended
In addition the NameToView map was extended to cache the is_paired flag
so I don't have to keep searching through the dictionary of library information.
Diane Trout [Sun, 7 Nov 2010 23:58:07 +0000 (15:58 -0800)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Sun, 7 Nov 2010 23:54:22 +0000 (15:54 -0800)]
Very basic non-ExtJS version of the inventory page
It needs to group by object type. Possible with an index of types
followed by index of objects before getting to the detail page.
The component where it forces you to login before viewing the
inventory page is broken. It shows a super-plain form, but
fails on submit with some kind of cross site forgery prevention
cookie fails
Diane Trout [Sun, 7 Nov 2010 23:46:49 +0000 (15:46 -0800)]
Remove another ExtJS panel reference.
This version has a working two column independantly scrollable
library browse with no ExtJS dependency.
Diane Trout [Fri, 5 Nov 2010 23:27:02 +0000 (16:27 -0700)]
Update to the new UCSC DAF file.
We are submitting 3 different types of expression data,
one for Gencode v3c, one for Gencode v4, and one for the
de novo cufflinks assembly.
As a result I needed to update the file name to ucsc view map
to describe these new, more specific datasets.
Also I modified the name of the archival and upload condor scripts
to include a username.
Diane Trout [Wed, 3 Nov 2010 23:59:07 +0000 (16:59 -0700)]
Create a class to handle mapping extension to ucsc view attributes.
This will attempt to guess if a library is paired end by seeing
if there are more paired end lanes than single read lanes.
The file exention matching was changed to use fnmatch instead of
ends with.
I still haven't added the ability to define extensions to ucsc DAF
view maps in a config file, as I don't know how to handle the case
of the .bam file that goes to a different view depending on if its
a paired end vs single end.
Also the ucsc_gather script is too long and parts of it need
to migrate into the rest of the htsworkflow tree.
Diane Trout [Tue, 2 Nov 2010 00:16:03 +0000 (17:16 -0700)]
Two lines between functions (trivial change)
Diane Trout [Mon, 1 Nov 2010 21:55:42 +0000 (14:55 -0700)]
Only complane about missing genomes if we _wanted_ to align the lane.
If it was flagged for sequencing, don't complain.
Diane Trout [Mon, 1 Nov 2010 21:52:45 +0000 (14:52 -0700)]
Check for s_${lane}_02_matrix.txt as well as s_${lane}_1_matrix.txt
It still could use more test coverage
Diane Trout [Thu, 28 Oct 2010 00:25:02 +0000 (17:25 -0700)]
Remove dependency on ExtJS for library & lanes_for pages.
The inventory page still needs extjs.
In the process I cleaned up some of the CSS as well.
The downside is you'll need to link
htsworkflow/static/js/jquery.min.js
to a valid jquery sourcefile.
Diane Trout [Wed, 27 Oct 2010 22:51:20 +0000 (15:51 -0700)]
The WITH_SEQUENCE flag needs to be lower-case "true".
Diane Trout [Fri, 22 Oct 2010 23:40:13 +0000 (16:40 -0700)]
Report version number derived from git tag.
This patch includes the necessary infrastructure to support that
feature and its been added to qseq2fastq and srf2fastq.
Additionally to improve testability of qseq2fastq and srf2fastq, the
original standalone module was moved into htsworkflow.pipelines and a
small stub module was placed in scripts.
Diane Trout [Tue, 19 Oct 2010 19:35:57 +0000 (19:35 +0000)]
Set WITH_SEQUENCE as both a per-lane AND global parameter
the per-lane version set $(lane)_WITH_SEQUENCE := TRUE in the makefiles
the gerald WITH_SEQUENCE TRUE changed the setting in the config.txt file
So lets just set both versions and hopefully we'll get sequence files
out the other end.
Let me show you my hammer...
Diane Trout [Fri, 8 Oct 2010 17:32:46 +0000 (17:32 +0000)]
WITH_SEQUENCE is actually a global GERALD option, remove the lane specifier.
Diane Trout [Thu, 7 Oct 2010 22:32:45 +0000 (22:32 +0000)]
Update gerald config file generation.
Include ELAND_SET_SIZE, WITH_SEQUENCE options
Stop commenting SEQUENCE_FORMAT out
Lorian Schaeffer [Tue, 5 Oct 2010 18:12:42 +0000 (18:12 +0000)]
Minor changes to display
Lorian Schaeffer [Fri, 1 Oct 2010 19:01:58 +0000 (19:01 +0000)]
Removed some models from the admin index view
Lorian Schaeffer [Fri, 1 Oct 2010 18:57:01 +0000 (18:57 +0000)]
Changed Frequently Used links to be properly relative
Lorian Schaeffer [Fri, 1 Oct 2010 01:39:37 +0000 (01:39 +0000)]
Removed from admin index page
Lorian Schaeffer [Fri, 1 Oct 2010 01:25:11 +0000 (01:25 +0000)]
Removed unused DataRun model from admin index page
Lorian Schaeffer [Fri, 1 Oct 2010 01:21:39 +0000 (01:21 +0000)]
Removed all analysis models from main admin index
Lorian Schaeffer [Fri, 1 Oct 2010 01:19:44 +0000 (01:19 +0000)]
Limited which models show up on admin index page
Removed some fields from view in the Library Change/Add page
Rearranged the remaining Library fields
Diane Trout [Thu, 30 Sep 2010 21:58:28 +0000 (21:58 +0000)]
Ignore some files generated by python
.coverage
*.egg-info
Diane Trout [Fri, 24 Sep 2010 22:34:17 +0000 (22:34 +0000)]
Adjust spoolwatchers use of inotify to correspond to pyinotify 0.8.9
Diane Trout [Fri, 24 Sep 2010 22:34:16 +0000 (22:34 +0000)]
Handle the case when a sequencing lane lacks any yield information.
(For instance if we only run one lane of analysis on a flowcell)
Diane Trout [Fri, 24 Sep 2010 22:34:15 +0000 (22:34 +0000)]
Remind myself that dry-run doesn't work with the extract results code
Diane Trout [Fri, 10 Sep 2010 22:35:20 +0000 (22:35 +0000)]
Add support for CASAVA 1.7
They stopped generating eland_extended files so I needed to parse
the export files instead.
Also more carefully test how I'm computing the U0-2 and R0-2
scores.
Diane Trout [Tue, 24 Aug 2010 21:32:06 +0000 (21:32 +0000)]
Better detect which program ids generated CNF4 srf files.
apparently illumina2srf v1.11.5Illumina.1.3 also genrated CNF4s.
It's only illumina2srf v1.11.6... that generated CNF1s.
Diane Trout [Tue, 24 Aug 2010 00:31:55 +0000 (00:31 +0000)]
Split parts of build_fastqs into a seperate function.
Use the database to flag if a lane is bad instead of hard coding it.
Support comments and spaces in the library_id to target directory file.
Diane Trout [Mon, 23 Aug 2010 22:54:20 +0000 (22:54 +0000)]
Add a "All Lanes" option to the control-lane field for the cases
where we didn't set a control lane, but instead let the pipeline
estimate from the entire flowcell.
Diane Trout [Sat, 14 Aug 2010 00:23:06 +0000 (00:23 +0000)]
Shrink flowcell notes field.
See [ticket:167]
Diane Trout [Sat, 14 Aug 2010 00:07:16 +0000 (00:07 +0000)]
Don't offer to track flowcell config files in svn
Diane Trout [Fri, 13 Aug 2010 23:57:06 +0000 (23:57 +0000)]
Remove debug print statement
Diane Trout [Fri, 13 Aug 2010 23:51:32 +0000 (23:51 +0000)]
Reduce the size of the bioanalyzer summary text box
to 3 lines.
Diane Trout [Fri, 13 Aug 2010 22:49:10 +0000 (22:49 +0000)]
Tell git to ignore *.py[co]~ files.
(Useful if you're using git-svn)
Diane Trout [Fri, 13 Aug 2010 22:49:09 +0000 (22:49 +0000)]
This patch adds in fields to support storing bioanalyzer results. [ticket:166]
Diane Trout [Wed, 14 Jul 2010 22:32:16 +0000 (22:32 +0000)]
Georgi used slightly different extensions for the second batch
of data
Diane Trout [Wed, 14 Jul 2010 22:31:57 +0000 (22:31 +0000)]
Add search field to the lanes admin page
Diane Trout [Mon, 12 Jul 2010 19:02:22 +0000 (19:02 +0000)]
Use the htsworkflow API to determine if a flowcell is paired end or not.