Diane Trout [Tue, 27 Nov 2012 18:30:36 +0000 (10:30 -0800)]
Add dependencies to the setup.py
I do need to make benderjab public before other people
could install this. Or maybe figure out how to use the optional
dependency mode.
Diane Trout [Tue, 20 Nov 2012 22:37:49 +0000 (14:37 -0800)]
Test presence of species & species name on library index page.
Diane Trout [Tue, 20 Nov 2012 22:36:49 +0000 (14:36 -0800)]
Correctly implement merging Notification & Manager sets
Apparently I was off imagining functiosn that don't exist.
Diane Trout [Tue, 20 Nov 2012 22:31:05 +0000 (14:31 -0800)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Tue, 20 Nov 2012 22:29:20 +0000 (14:29 -0800)]
Renamed various django tests.py to test_module.py
It appears by default py.test was looking for packages named test_,
since it wasn't finding the tests.py modules. This seemed like
a reasonable alternative convention.
Diane Trout [Tue, 20 Nov 2012 20:51:39 +0000 (12:51 -0800)]
Ignore .tox directory
I was experimenting with the virtualenv testing tool tox.testrun.org
and I might as well ignore its directory
Diane Trout [Tue, 20 Nov 2012 20:46:41 +0000 (12:46 -0800)]
Fix RDF schema problems with lane_number and species.
My RDF schema was using the term "species" both for the species
name and a species class -- which doesn't make sense. This
version of the schema introduces a species_name which can
attach either to the Library object or the Species object.
Its still a little inconsistent as I'm using it for both
"common name" and "scientific name". But hey its an improvement.
Also there's a tweak to the library_number type on the library detail
page setting the type to string instead of number, as I decided
it should be treated internally as an opaque identifier.
In theory someone might start naming lanes A,B,C,D or 1T, 1B
(for the top and bottom of a flowcell slide).
Finally I decided that the gel_cut should be of type integer,
yes decimal is the "more general type" but I'm using integer in
my sql schema so its only going to return integers.
Diane Trout [Tue, 20 Nov 2012 20:41:57 +0000 (12:41 -0800)]
Run the library detail page through RDF validation.
Also ignore the missing type error message for http://localhost/
as that resource really shouldn't have a type.
This improved test does catch a few new model inconsistencies
which I'll fix in my next patch.
Diane Trout [Tue, 20 Nov 2012 06:16:08 +0000 (22:16 -0800)]
Change add_default_schema to use pkg_resources feature to find schemas.
I was trying to get py.test to work and it really wants to install
things, and my previous method to find the schema files wasn't working
very well with the egg distribution.
Diane Trout [Tue, 20 Nov 2012 01:09:47 +0000 (17:09 -0800)]
Add dependency information to the setup.py script
Though its still missing a bit as I don't have benderjab
hosted and librdf needs to be installed seperately.
Diane Trout [Tue, 20 Nov 2012 01:04:12 +0000 (17:04 -0800)]
rdf:Resource can be either a resource or a blank node.
Thus we should only toss an error in the case of a node being
a literal.
Diane Trout [Tue, 20 Nov 2012 00:50:49 +0000 (16:50 -0800)]
Uniquely merge BCC and Manager lists for sending notification email.
This uses a set to only send one email address one copy of a notification.
Diane Trout [Fri, 16 Nov 2012 00:01:04 +0000 (16:01 -0800)]
Further attempts to validate RDF models.
I had a bug caused by lane numbers being langauage tagged strings,
and thus not being found by my sparql query.
I found a solution to filter based on just the contents of a string
ignoring the language tag. However I thought not only should I
make it easier to run my RDF model validation code, I should also
double check the literal types.
Previously I just tagged any literal as rdfs:Literal. For ones
that should have a known type, I've changed it to the xmlschema
types.
This patch doesn't actually fix the bug. Just introduces the
diagnostic tool.
Diane Trout [Wed, 14 Nov 2012 19:30:35 +0000 (11:30 -0800)]
Add error message for typoing a result map filename.
As a gift to me when trying to do something while sleepy.
Diane Trout [Wed, 14 Nov 2012 00:38:20 +0000 (16:38 -0800)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Thu, 11 Oct 2012 23:32:27 +0000 (16:32 -0700)]
Catch another lookup error.
Apparently one of gerald_summary[read][lane] was a dictionary
and so threw a KeyError instead of an IndexError. I might as well
catch both.
Diane Trout [Thu, 27 Sep 2012 18:46:56 +0000 (11:46 -0700)]
older librdf wanted to include type information when showing query strings.
So I ran everything through fromTypedNode to convert the nodes
to simple python types.
However fromTypedNode wasn't intended to handle non-literal nodes
so I had to adjust it a bit to return resource nodes safely.
Diane Trout [Thu, 27 Sep 2012 17:37:56 +0000 (10:37 -0700)]
When collecting files for a geo submission, group on library id
instead of the previous grouping on lane.
Diane Trout [Tue, 25 Sep 2012 23:18:42 +0000 (16:18 -0700)]
This might actually generate soft file with raw & supplemental data.
To make working with the development server easier, I changed
the submission class to take a host which it will use to generate
the base library url.
When constructing URLs for files, I'm now using the actual path names
instead of synthesizing something based on the submission name.
This is to limit the amount of knowledge that needs to be passed
between the fastq generation code.
For fastq files it looks at the source file to find the flowcell
information. For supplemental files it looks at the submission
class for that analysis directory and grabs the library id
from there.
Diane Trout [Mon, 24 Sep 2012 23:43:33 +0000 (16:43 -0700)]
Merge changing lane_number to string and sequence finding code changes
I started using actual file paths instead synthetic submission
paths for naming where my sequence files are.
This one still one generate geo submissions correctly as I'm
pretty sure not all of the queries have been updated yet.
Diane Trout [Mon, 24 Sep 2012 23:37:30 +0000 (16:37 -0700)]
Add a log message to for debugging
Diane Trout [Mon, 24 Sep 2012 23:34:44 +0000 (16:34 -0700)]
Define XHTML_RDF_DTD as None when we can't load the DTD
Diane Trout [Mon, 24 Sep 2012 22:28:10 +0000 (15:28 -0700)]
Make the public html pages valid xhtml, and validate more RDFa cases.
Also after I spent time playing with the w3c online validator,
I decided it was best to try and add modest validation to my
unit tests.
So now there's a validate_xhtml function in ethelp.
The one really weird thing is I tried to load the DTD
in the test case, however it looks like librdf clobbered the
XML catalog resolver at some point so the DTD resolver can't
find anything.
Diane Trout [Mon, 24 Sep 2012 22:26:14 +0000 (15:26 -0700)]
remove some dead commented out code.
Diane Trout [Thu, 20 Sep 2012 22:17:53 +0000 (15:17 -0700)]
Fix (some) missing closing tags.
Diane Trout [Thu, 20 Sep 2012 21:35:51 +0000 (14:35 -0700)]
Make a validation error message between different ages of librdf.
result.uri vs result again.
Diane Trout [Thu, 20 Sep 2012 21:26:46 +0000 (14:26 -0700)]
Also make the library index page conform to htsworkflow ontology.
Diane Trout [Thu, 20 Sep 2012 00:08:43 +0000 (17:08 -0700)]
Minor tweaks to deal with the older version of librdf on ubuntu 10.04
things like utf-8 escaping a string, using str(node.uri) instead
of str(node).
Diane Trout [Wed, 19 Sep 2012 23:10:57 +0000 (16:10 -0700)]
Use htsworkflow ontology to validate various RDF using components.
Of course to use the ontology I had to make one first.
Unsurprisingly implementing it touched a bunch of code & templates.
I tried to be more consisten with using mixed-case names for
classes and lower_case names for properties.
There's some inconsistencies. like i use the term notes & comments
in different areas. Also, should I be using my own terms or
do better at reusing more standard ontologies?
Diane Trout [Wed, 19 Sep 2012 23:04:35 +0000 (16:04 -0700)]
Refactor property type validator to support multiple classes for domain/range.
Also test to make sure we can have more than one domain/range statement.
Diane Trout [Wed, 19 Sep 2012 23:03:07 +0000 (16:03 -0700)]
Add stub xhtml vocab ontology, to make model validation quieter.
(the stylesheets got attached as a property of the library or flowcell)
Diane Trout [Tue, 18 Sep 2012 23:34:27 +0000 (16:34 -0700)]
Merge ssh://jumpgate.caltech.edu/var/htsworkflow/htsworkflow
Diane Trout [Tue, 18 Sep 2012 23:33:01 +0000 (16:33 -0700)]
email.bcc should be a list, not a nested list.
NOTIFICATION_BCC was already a list of options.
Diane Trout [Tue, 18 Sep 2012 18:36:16 +0000 (11:36 -0700)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Tue, 18 Sep 2012 18:34:25 +0000 (11:34 -0700)]
Improvements to rdfinfer.
Add rule to infer class and subClassOf memberships,
add testing for the class case.
Add code to run all the validation rules.
Diane Trout [Tue, 18 Sep 2012 18:31:49 +0000 (11:31 -0700)]
Change rdfhelp.dump_model so you can specify a destination stream.
Diane Trout [Tue, 18 Sep 2012 18:20:26 +0000 (11:20 -0700)]
Progress using rdf model to link fastqs with flowcell/lib metadata.
I changed how I was using rdf:type -- the most raw data is now
a 'sequencer_result' and now there's a seperate file_type
attribute to indicate what kind of result file it is.
I renamed find_missing_targets to update_fastq_targets as
in addition to finding what fastqs we need to generate it'll
also download missing flowcell information.
I'm still having trouble fishing out the fastq files so this isn't
ready yet.
Finally minor tweaks to the soft file formatting to try
and get it to render everything without spurious spaces.
Diane Trout [Tue, 18 Sep 2012 17:55:36 +0000 (10:55 -0700)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Sat, 15 Sep 2012 05:49:29 +0000 (22:49 -0700)]
Start implementing infering triples.
This includes utilities to import the common schemas, and
a bit of functionality for validating models, in
addition to the rule to compute inverseOf.
Diane Trout [Wed, 12 Sep 2012 18:35:37 +0000 (11:35 -0700)]
Be more defensive if the database is missing some data instead of
crasing on trying to access an empty list.
Diane Trout [Wed, 12 Sep 2012 18:34:54 +0000 (11:34 -0700)]
fix a wrong variable name
Diane Trout [Wed, 12 Sep 2012 18:32:55 +0000 (11:32 -0700)]
Tweaks to make the sample gather query faster.
Instead of creating a new library term for submissions, I
changed it back to using the stanrard library ontology term.
(making it easier to link things together).
Also apparently you can have nested optional statements.
Diane Trout [Wed, 12 Sep 2012 18:31:19 +0000 (11:31 -0700)]
Add a helpful error message when we need the submission name but it wasn't provided
Diane Trout [Wed, 12 Sep 2012 18:30:26 +0000 (11:30 -0700)]
Another tweak to allow parser detection to work on content-type text/plain urls
Diane Trout [Tue, 11 Sep 2012 23:01:46 +0000 (16:01 -0700)]
Better parser guessing when grabbing plain text from a webserver.
Also fix a typo in an error message
Diane Trout [Mon, 10 Sep 2012 21:16:08 +0000 (14:16 -0700)]
Use the same model for building fastqs as for building soft file
(And let user specify a base filename on command line)
Diane Trout [Mon, 10 Sep 2012 21:15:32 +0000 (14:15 -0700)]
Allow retrying retrieving rdf from website a few times.
Diane Trout [Sat, 1 Sep 2012 02:59:52 +0000 (19:59 -0700)]
Import flowcell information into our model this allows
our super library id & sequence & flowcell info query to work
Diane Trout [Sat, 1 Sep 2012 02:56:55 +0000 (19:56 -0700)]
Have librdf open the urls in load_into_model instead of using python
As a side effect I ended up discovering that my parser type detection
code for file types was wrong, and now it has test code
I was trying to fix a bug where librdf 1.0.10.1-1.1ubuntu1 was
segfaulting when loading into the model but I couldn't figure
out what was doing it.
Diane Trout [Sat, 1 Sep 2012 02:53:03 +0000 (19:53 -0700)]
Automatically add the flowcell subdirectory to the sequence archive path
Diane Trout [Fri, 31 Aug 2012 21:39:45 +0000 (14:39 -0700)]
Older rdf library doesn't like unicode strings as nodes
so manually encode as utf8
Diane Trout [Fri, 31 Aug 2012 19:29:59 +0000 (12:29 -0700)]
Fix matching scanned sequence files to library IDs work for hiseq runs.
The previous version was keying off of flowcell/lane so if you
had multiple libraries from the same flowcell/lane all the sequences
would end up in one of the libraries.
Hopefully this fixes that. Though to do this I ended up changing
the whole structure of condorfastq to be based on updating an RDF model.
This depends on the sequence.py module changes of saving things to
rdf models -- and the new code to infer library ids at that layer.
Diane Trout [Fri, 31 Aug 2012 19:19:42 +0000 (12:19 -0700)]
Inherit from MutableMapping for ResultMap.
This gets me automatic methods.
I did it because my for x in resultmap wasn't working
correctly. It was using a list key (0) instead of a map key
(library id) and tossing a key error.
Also instead of using ResultMap.add_result(key, destination) I can do
ResultMap[key] = destination
Diane Trout [Fri, 31 Aug 2012 19:08:08 +0000 (12:08 -0700)]
Allow option to save/restore a sequence class to a RDF model.
(After doing this I started having dreams of some set of mixins
designed to persist data into different types of storage).
I also renamed the sql save to indicate that its going to a SQL
database.
Also I renamed one of my simplify Uris to stripNamespace
to make it clearer what it was actually doing.
simplify_uri just returns the end of a uri -- regardless of type.
stripNamespace removes a specific namespave from a uri.
Diane Trout [Tue, 28 Aug 2012 23:03:45 +0000 (16:03 -0700)]
Test RDFa encoded information for flowcell & libraries.
This shouls make sure that I have the right class type on flowcells
so I can reliably fish it out of a triple store.
Diane Trout [Tue, 28 Aug 2012 22:57:14 +0000 (15:57 -0700)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Tue, 28 Aug 2012 22:56:22 +0000 (15:56 -0700)]
sanitize_literal had trouble with empty strings.
This fixes that and tests that edge case.
Diane Trout [Tue, 28 Aug 2012 22:48:50 +0000 (15:48 -0700)]
Attempt to come up with regexp to detect hiseq formatted files.
Diane Trout [Thu, 9 Aug 2012 19:53:32 +0000 (12:53 -0700)]
Fix some mismatches between variable names in python code and in templates
Diane Trout [Tue, 7 Aug 2012 02:13:49 +0000 (19:13 -0700)]
Create a lane to file name turtle data file
this required passing the lane ID information back through
the json api.
Diane Trout [Tue, 7 Aug 2012 02:01:51 +0000 (19:01 -0700)]
Simplify code to see if we already have a DataRun created.
Make sure we update the data runs when getting flowcell lane details.
Test to make sure we can't accidentally add more than one data run.
In the test code also change to assert from failUnless.
Diane Trout [Mon, 6 Aug 2012 21:26:26 +0000 (14:26 -0700)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Mon, 6 Aug 2012 21:25:11 +0000 (14:25 -0700)]
Don't link to lane for the raw result files section.
The raw result file code for finding the lane ID was returning
a wrong value (first lane) which was causing trouble on
multi sample hiseq runs.
Diane Trout [Mon, 6 Aug 2012 21:17:01 +0000 (14:17 -0700)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Mon, 6 Aug 2012 21:16:34 +0000 (14:16 -0700)]
Use default printer settings again.
(Removing it broke things)
Diane Trout [Mon, 6 Aug 2012 18:25:24 +0000 (11:25 -0700)]
Test proper setting of sequence project variable
Diane Trout [Thu, 2 Aug 2012 21:52:10 +0000 (14:52 -0700)]
Test more of the sequences class.
(And fix the bugs discovered with better test coverage)
Diane Trout [Thu, 2 Aug 2012 21:51:41 +0000 (14:51 -0700)]
Fix a few more formatting issues with the geo template
Diane Trout [Wed, 1 Aug 2012 23:19:50 +0000 (16:19 -0700)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Wed, 1 Aug 2012 23:18:13 +0000 (16:18 -0700)]
GEO wanted both files for a paired lane to be on a single line.
This required a bit of formatting code to massage the
results of the sparql query into an acceptable form.
Diane Trout [Fri, 27 Jul 2012 22:35:00 +0000 (15:35 -0700)]
Fix issues introduced when switching to the django templating system for fastq generation.
It needs to know where django is, and I didn't include a short form for
identifying flowcells.
Diane Trout [Wed, 25 Jul 2012 18:53:11 +0000 (11:53 -0700)]
Actually we want to capture the unanalyzed lanes if we can...
Diane Trout [Tue, 24 Jul 2012 19:39:34 +0000 (12:39 -0700)]
Get actual list of sequencers used for a library.
I do have pipeline version numbers available, but with the current
version of the query it would end up duplicating the sequencer
model number. (there's a lot more version combinations then
there are sequencers)
Diane Trout [Mon, 23 Jul 2012 18:34:36 +0000 (11:34 -0700)]
Update to our current IPs. I'm tired of syncdb overwriting them.
Diane Trout [Thu, 19 Jul 2012 18:23:40 +0000 (11:23 -0700)]
Adapt building qseq or srf archives to use SampleKey identified lanes
Diane Trout [Thu, 19 Jul 2012 18:14:29 +0000 (11:14 -0700)]
Update save_raw_data to SampleKey api
Diane Trout [Thu, 19 Jul 2012 00:01:03 +0000 (17:01 -0700)]
Add in a few more genomes to detection code
Diane Trout [Tue, 17 Jul 2012 22:56:41 +0000 (15:56 -0700)]
Fix library viewing code to read HiSeq run xmls.
Also still need to fix the samples.view code
Diane Trout [Mon, 16 Jul 2012 22:09:00 +0000 (15:09 -0700)]
Remove default sequence-format so the new auto-detector code is actually called
Diane Trout [Sat, 14 Jul 2012 00:31:41 +0000 (17:31 -0700)]
Attempt to guess raw sequence type for a run.
Also will check for the Unaligned directory first
before looking for the BaseCalls directory, as there's still
a BaseCalls dir.
(It does this by checking for the aligned directory first, and
if it exists using the name in the aligned config file to find
the corresponding base call directory)
Diane Trout [Fri, 13 Jul 2012 23:31:48 +0000 (16:31 -0700)]
Fix a refactoring error
Diane Trout [Thu, 12 Jul 2012 18:16:17 +0000 (11:16 -0700)]
Look for RTAConfig.xml file generated by pipeline 1.12 era.
Diane Trout [Thu, 12 Jul 2012 17:59:19 +0000 (10:59 -0700)]
Use lane 1s matrix when pipeline was set to build matrix from all.
(HiSeq's pipeline doesn't make the lane-less matrix file)
Diane Trout [Thu, 12 Jul 2012 17:58:15 +0000 (10:58 -0700)]
Clean up flowcell ID detection and add support for reading HiSeq IDs
Update the quick and dirty report generator for SampleKey API
Diane Trout [Thu, 12 Jul 2012 17:54:37 +0000 (10:54 -0700)]
fix a variable name typo
Diane Trout [Thu, 12 Jul 2012 17:52:53 +0000 (10:52 -0700)]
Re-enable some suppressed tests and update them for the SampleKey API.
Also try to test getting the HiSeq flowcell ID out of RunInfo.xml
Diane Trout [Thu, 12 Jul 2012 17:49:44 +0000 (10:49 -0700)]
whitespace fix.
Diane Trout [Wed, 11 Jul 2012 00:29:00 +0000 (17:29 -0700)]
Test reading the xml out of a file too.
You write some code and think "oh this is simple,
it should be ok without tests" and then you watch the stack traces.
Now its tested -- at least a bit.
Diane Trout [Tue, 10 Jul 2012 23:55:19 +0000 (16:55 -0700)]
Fix import statement and typos for genomemap
Diane Trout [Tue, 10 Jul 2012 23:38:20 +0000 (16:38 -0700)]
Create a class to convert contig names into genome/contig names
And now being split out I can reasonably test it.
This is needed so when we're reporting where the genome
locations mapped to we can summarize them as a genome
instead of chr1 chr2 chr3 etc.
Diane Trout [Tue, 10 Jul 2012 21:13:46 +0000 (14:13 -0700)]
Guess genome name for building compressed mapping counts from genomesize.xml
The HiSeq pipeline creates a file that has the sizes of the genomes
that it mapped the particular project to, figuring out the genome
version from that is a lot easier than dealing with all the random
possible file names for the config files.
This patch is lacking in that I just hard coded a few genomes
I probably need some mechanism for pulling it from a database.
Diane Trout [Tue, 10 Jul 2012 21:10:45 +0000 (14:10 -0700)]
Give some failover defaults to printer settings.
It's also not certain if we're still using them anywhere.
Diane Trout [Sat, 7 Jul 2012 00:34:27 +0000 (17:34 -0700)]
Use sample keys when looking up lane parameters.
And as a bonus feature I decided to test for a SampleKey incoming
and if its not convert the older lane number to a sample key internally.
One downside is since I'm storing not fully specified SampleKeys
I have to do a brute force lookup of the key.
Diane Trout [Fri, 6 Jul 2012 23:12:25 +0000 (16:12 -0700)]
A few python 2.6 test case incompatibilities.
Diane Trout [Fri, 6 Jul 2012 23:03:51 +0000 (16:03 -0700)]
Switch to regular dictionary instead of ordered dictionary.
python 2.6 doesn't have ordered dictionary. So I switched to using
a regular dictionary and just sorting the returned keys.
Diane Trout [Thu, 5 Jul 2012 17:33:46 +0000 (10:33 -0700)]
Massively rework eland file finding and indexing.
Instead of looking in known locations for specific file names
in priority order, this now scans the gerald tree looking for
any potential eland files.
The eland container class has been changed to be a MutableMapping and
the found files are added. The class internally tracks the priority of
the different file types and will drop any lower files when a higher
priority file shows up.
In addition the key for finding files is now a "SampleKey" class
which supports tracking sample name, lane and read. The SampleKey
also has a fuzzy matching feature that uses "None" as a wildcard.
So you can search for all the samples for a particular end with
something like "SampleKey(read=1)". Needless to say this change
required updating a lot of code that was assuming the nested
list/dictionary structure from before that was tracking read/lane.
Diane Trout [Thu, 5 Jul 2012 17:30:19 +0000 (10:30 -0700)]
Rename chromosome in hiseq test set to match our fake human build
Diane Trout [Sat, 30 Jun 2012 03:23:43 +0000 (20:23 -0700)]
Disable debug messages for each scanned eland file pattern
Given how many places I'm looking for the eland files the debug
messages were getting way too noisy.
Diane Trout [Sat, 30 Jun 2012 03:13:57 +0000 (20:13 -0700)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Sat, 30 Jun 2012 03:06:33 +0000 (20:06 -0700)]
Support multiple having an eland file split into multiple fragments.
The interesting thing was changing the match_codes and mapped_reads
dictionaries into their own classes so I can add them together, and
it'll go through and add matching dictionary entries together.
e.g. MatchCodes({'U0':5})+MatchCodes({'U0':5}) == MatchCodes({'U0': 10})
Unfortunately because there are multiple samples per lane on the
hiseq this still isn't enough to support counting aligned results yet.