htsworkflow.git
7 years agomark the example submission rule files as being raw, so the escapes dont get confused master
Diane Trout [Thu, 23 Oct 2014 23:15:09 +0000 (16:15 -0700)]
mark the example submission rule files as being raw, so the escapes dont get confused

7 years agomost of our test submission rule files use bz2 regexes, so lets make those files
Diane Trout [Thu, 23 Oct 2014 23:14:44 +0000 (16:14 -0700)]
most of our test submission rule files use bz2 regexes, so lets make those files

7 years agoChange unittest2 back into unittest.
Diane Trout [Tue, 6 May 2014 23:45:47 +0000 (16:45 -0700)]
Change unittest2 back into unittest.

Django 1.7 docs pointed out unittest2 was backported into
python 2.7, so they dropped their version.

7 years agoAdd post_json as some DCC objects are created with post requests.
Diane Trout [Tue, 10 Jun 2014 20:13:46 +0000 (13:13 -0700)]
Add post_json as some DCC objects are created with post requests.

considering the code is duplicated, there's probably a way to abstract
those functions.

7 years agoRename headers to json_headers for requests (seemed a bit clearer)
Diane Trout [Tue, 10 Jun 2014 20:09:39 +0000 (13:09 -0700)]
Rename headers to json_headers for requests (seemed a bit clearer)

Also output a more informative http exception

7 years agoUpdates to jsonld context for the DCC schemas.
Diane Trout [Tue, 10 Jun 2014 20:05:41 +0000 (13:05 -0700)]
Updates to jsonld context for the DCC schemas.

Also I added a comment for what their prefix NTR means.

7 years agoUpdate test_encoded test cases due to https now being the default.
Diane Trout [Tue, 10 Jun 2014 22:10:50 +0000 (15:10 -0700)]
Update test_encoded test cases due to https now being the default.

7 years agoDCC switched to https
Diane Trout [Tue, 10 Jun 2014 20:04:54 +0000 (13:04 -0700)]
DCC switched to https

7 years agoUpdate fastqname code & test code to better support having optional compression exten...
Diane Trout [Tue, 10 Jun 2014 22:11:25 +0000 (15:11 -0700)]
Update fastqname code & test code to better support having optional compression extension.

Its a bit weird as thats the first property which is optional, so
it needed some special case logic.

Also I accept either None or '' for the case of no compression extension as
condorfastq passes in compression_extension=''

7 years agoDirectly generate compressed fastq files from HiSeq split fastqs.
Diane Trout [Mon, 12 May 2014 23:16:00 +0000 (16:16 -0700)]
Directly generate compressed fastq files from HiSeq split fastqs.

desplit_fastq is the only fastq builder that has the necessary
compression argument.

7 years agothe filename templates were moved from condorfastq to fastqname a while ago
Diane Trout [Mon, 12 May 2014 23:14:18 +0000 (16:14 -0700)]
the filename templates were moved from condorfastq to fastqname a while ago

7 years agoIndicate when desplit_fastq didn't get any files to process
Diane Trout [Mon, 12 May 2014 23:12:08 +0000 (16:12 -0700)]
Indicate when desplit_fastq didn't get any files to process

7 years agoMake sure we can load a run file that lacks alignment details.
Diane Trout [Tue, 6 May 2014 22:38:48 +0000 (15:38 -0700)]
Make sure we can load a run file that lacks alignment details.

7 years agoFor some reason the previous integrety fixes, added made the failed flowcell be in...
Diane Trout [Thu, 13 Feb 2014 01:28:28 +0000 (17:28 -0800)]
For some reason the previous integrety fixes, added made the failed flowcell be in the second position in the table

7 years agoPostgresql has referential integrety checking.
Diane Trout [Thu, 13 Feb 2014 01:27:30 +0000 (17:27 -0800)]
Postgresql has referential integrety checking.

So some of my dangling foreign keys caused tests to failed.
I resolved them through a few different ways,
  * adding initial_data to include description tables
  * adding some additional data to the test fixture.
  * renumbering affiliation keys to match my small test list.

I had to reorder the app list because I think there was a dependency
in the wrong order.

7 years agoIgnore the path when comparing filenames from the md5 files.
Diane Trout [Tue, 6 May 2014 22:32:45 +0000 (15:32 -0700)]
Ignore the path when comparing filenames from the md5 files.

There were too many other ways of creating them or running
the md5 file creation from too many different places for including
the relative path to be reliable.

7 years agouse autoopen to access fastq files
Diane Trout [Wed, 2 Apr 2014 01:48:40 +0000 (18:48 -0700)]
use autoopen to access fastq files

7 years agoStart implementing tool to compute library fastq quality
Diane Trout [Tue, 1 Apr 2014 21:44:46 +0000 (14:44 -0700)]
Start implementing tool to compute library fastq quality

7 years agoJust use the python 2 url parsers instead of trying to pull it out of requests
Diane Trout [Thu, 13 Feb 2014 01:28:50 +0000 (17:28 -0800)]
Just use the python 2 url parsers instead of trying to pull it out of requests

7 years agoUpdate timestamps in inventory app test data to be timezone aware.
Diane Trout [Thu, 13 Feb 2014 01:21:01 +0000 (17:21 -0800)]
Update timestamps in inventory app test data to be timezone aware.

7 years agopoint default flowcell directory to test data
Diane Trout [Wed, 12 Feb 2014 21:57:39 +0000 (13:57 -0800)]
point default flowcell directory to test data

7 years agoadd default values for some application specific values
Diane Trout [Wed, 12 Feb 2014 21:57:22 +0000 (13:57 -0800)]
add default values for some application specific values

7 years agoConfigure default STATICFILES_DIRS for static files (css, js, imgs)
Diane Trout [Thu, 23 Jan 2014 20:13:51 +0000 (12:13 -0800)]
Configure default STATICFILES_DIRS for static files (css, js, imgs)

And since we're using the newer django static file handling, we
can remove the old debug url configration to find the static files.

7 years agoMathematica had a suggestion for a fastq mime type. I extended for my other sequence...
Diane Trout [Wed, 22 Jan 2014 23:53:41 +0000 (15:53 -0800)]
Mathematica had a suggestion for a fastq mime type. I extended for my other sequence types

7 years agoRegex pattern was wrong.
Diane Trout [Wed, 22 Jan 2014 23:52:53 +0000 (15:52 -0800)]
Regex pattern was wrong.

7 years agoConvert more timestamps to include timezone
Diane Trout [Tue, 21 Jan 2014 23:54:27 +0000 (15:54 -0800)]
Convert more timestamps to include timezone

7 years agoMerge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Tue, 10 Jun 2014 22:48:46 +0000 (15:48 -0700)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow

7 years agoAdd option to compress fastq file
Diane Trout [Tue, 18 Feb 2014 23:34:14 +0000 (15:34 -0800)]
Add option to compress fastq file

7 years agoAllow arbitrarily long regexes in experiment.FileType table
Diane Trout [Wed, 12 Feb 2014 22:40:36 +0000 (14:40 -0800)]
Allow arbitrarily long regexes in experiment.FileType table

8 years agoMerge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Wed, 22 Jan 2014 22:14:26 +0000 (14:14 -0800)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow

8 years agopass filters a string not an undefined variable
Diane Trout [Wed, 22 Jan 2014 22:10:49 +0000 (14:10 -0800)]
pass filters a string not an undefined variable

8 years agoRemove django test environment setup code.
Diane Trout [Fri, 17 Jan 2014 23:04:22 +0000 (15:04 -0800)]
Remove django test environment setup code.

I've found djangos ./manage.py test to work quite well these days,
and attempting to manually setup the django test environment generates
an exception on shutdown.

8 years agoUse SkipIf to skip copier tests if benderjab isn't working.
Diane Trout [Fri, 17 Jan 2014 23:02:08 +0000 (15:02 -0800)]
Use SkipIf to skip copier tests if benderjab isn't working.

This also uses the improved version of unittest in python 2.7+

8 years agoConvert timestamps from the runfolder file to a timezone aware time.
Diane Trout [Fri, 17 Jan 2014 22:30:34 +0000 (14:30 -0800)]
Convert timestamps from the runfolder file to a timezone aware time.

We assume the sequencer was run in the same place as the server software

8 years agodjango provides django.utils.timezone.now to return a timezone aware timestamp if...
Diane Trout [Fri, 17 Jan 2014 22:29:10 +0000 (14:29 -0800)]
django provides django.utils.timezone.now to return a timezone aware timestamp if USE_TZ=True

8 years agoSet default timezone to where our lab is located
Diane Trout [Fri, 17 Jan 2014 22:28:12 +0000 (14:28 -0800)]
Set default timezone to where our lab is located

8 years agoUpdate test data dates to have timezone information
Diane Trout [Fri, 17 Jan 2014 22:27:38 +0000 (14:27 -0800)]
Update test data dates to have timezone information

8 years agoFix test to use newer django database dictionary instead of the old single database...
Diane Trout [Fri, 17 Jan 2014 22:24:38 +0000 (14:24 -0800)]
Fix test to use newer django database dictionary instead of the old single database settings

8 years agoRe-add default email addresses for email notification sender & bcc
Diane Trout [Fri, 17 Jan 2014 22:22:38 +0000 (14:22 -0800)]
Re-add default email addresses for email notification sender & bcc

8 years agoImplement a encoded search to json-ld wrapper.
Diane Trout [Fri, 17 Jan 2014 01:23:31 +0000 (17:23 -0800)]
Implement a encoded search to json-ld wrapper.

An open question is encoded returns terms like "foo.bar" where
bar is an attribute of an object name for foo. However since
foo has no id associated with it, at best I would have a term with
a blank node.

For the moment I'm just deleting those terms. I'm not sure what a
better solution might be.

8 years agomake sure @vocab url has a trailing #. Its needed to construct better triple predicat...
Diane Trout [Fri, 17 Jan 2014 01:22:09 +0000 (17:22 -0800)]
make sure @vocab url has a trailing #. Its needed to construct better triple predicate names.

8 years agoUse new context creation functions to attach context to a json structure.
Diane Trout [Fri, 17 Jan 2014 00:19:31 +0000 (16:19 -0800)]
Use new context creation functions to attach context to a json structure.

8 years agoAdd functions to create a json-ld context and add namespaces to a context
Diane Trout [Fri, 17 Jan 2014 00:06:50 +0000 (16:06 -0800)]
Add functions to create a json-ld context and add namespaces to a context

Having smaller functions like this makes it easier to test.

8 years agoUtility class for accessing ENCODE3's submit / data warehouse server.
Diane Trout [Wed, 15 Jan 2014 01:31:01 +0000 (17:31 -0800)]
Utility class for accessing ENCODE3's submit / data warehouse server.

8 years agoAdapter to use PyLD's json-ld parser to add triples to librdf.
Diane Trout [Wed, 15 Jan 2014 01:29:29 +0000 (17:29 -0800)]
Adapter to use PyLD's json-ld parser to add triples to librdf.

8 years agoremove some trailing whitespace
Diane Trout [Wed, 15 Jan 2014 01:01:45 +0000 (17:01 -0800)]
remove some trailing whitespace

8 years agoremove some trailing whitespace
Diane Trout [Wed, 15 Jan 2014 01:01:18 +0000 (17:01 -0800)]
remove some trailing whitespace

8 years agoAdd option to get_model to disable RDF graph context.
Diane Trout [Wed, 15 Jan 2014 00:58:50 +0000 (16:58 -0800)]
Add option to get_model to disable RDF graph context.

At least with the version of librdf I was using the context
graph really slowed adding tripes down.

8 years agoIgnore kdevelop & kate project files.
Diane Trout [Wed, 8 Jan 2014 18:04:04 +0000 (10:04 -0800)]
Ignore kdevelop & kate project files.

8 years agoRename stripNamespace strip_namespace
Diane Trout [Wed, 8 Jan 2014 18:03:05 +0000 (10:03 -0800)]
Rename stripNamespace strip_namespace

8 years agoMerge branch 'master' of mus.cacr.caltech.edu:htsworkflow
Diane Trout [Mon, 23 Dec 2013 22:30:36 +0000 (14:30 -0800)]
Merge branch 'master' of mus.cacr.caltech.edu:htsworkflow

8 years agoupdate settings for new location of WSGI_APPLICATION, and add a reminder about where...
Diane Trout [Mon, 23 Dec 2013 22:30:16 +0000 (14:30 -0800)]
update settings for new location of WSGI_APPLICATION, and add a reminder about where to set debug

8 years agofix quote typos with the {% static %} template tag
Diane Trout [Mon, 23 Dec 2013 22:29:26 +0000 (14:29 -0800)]
fix quote typos with the {% static %} template tag

8 years agoMove wsgi into the actual project root.
Diane Trout [Mon, 23 Dec 2013 22:28:30 +0000 (14:28 -0800)]
Move wsgi into the actual project root.

8 years agoremove some test code debugging print statements
Diane Trout [Tue, 17 Dec 2013 19:50:12 +0000 (11:50 -0800)]
remove some test code debugging print statements

8 years agoUpdate WSGI_APPLICATION to point to the updated location of the wsgi script
Diane Trout [Tue, 17 Dec 2013 19:44:26 +0000 (11:44 -0800)]
Update WSGI_APPLICATION to point to the updated location of the wsgi script

8 years agoInventory works, even though its was hidden between two non-working modules
Diane Trout [Tue, 17 Dec 2013 19:32:20 +0000 (11:32 -0800)]
Inventory works, even though its was hidden between two non-working modules

8 years agomove an example result directory into git and use it for testing
Diane Trout [Tue, 17 Dec 2013 19:29:25 +0000 (11:29 -0800)]
move an example result directory into git and use it for testing

8 years agoMerge branch 'django1.4'
Diane Trout [Tue, 17 Dec 2013 19:20:03 +0000 (11:20 -0800)]
Merge branch 'django1.4'

there was a conflict with my qualifying the load_pipeline_run_xml function call

Conflicts:
htsworkflow/frontend/samples/views.py

8 years agoUpdate requirements to require diango 1.6
Diane Trout [Tue, 17 Dec 2013 19:19:09 +0000 (11:19 -0800)]
Update requirements to require diango 1.6

8 years agoStart simplyfing settings by using from settings_local import * pattern.
Diane Trout [Tue, 17 Dec 2013 19:15:09 +0000 (11:15 -0800)]
Start simplyfing settings by using from settings_local import * pattern.

This should be a lot less ugly than my previous implementation
using the ini file. However there are still settings that need to
be migrated over.

8 years agoadd new wsgi.py driver
Diane Trout [Tue, 17 Dec 2013 19:14:35 +0000 (11:14 -0800)]
add new wsgi.py driver

8 years agoUpdate templates to consistently use {% static %} instead of hard coding paths
Diane Trout [Tue, 17 Dec 2013 19:13:24 +0000 (11:13 -0800)]
Update templates to consistently use {% static %} instead of hard coding paths

Also import a few media files that were missing.

8 years agoMove loading DTD into a function as the call caused a hang when loaded into wsgi
Diane Trout [Tue, 17 Dec 2013 19:11:04 +0000 (11:11 -0800)]
Move loading DTD into a function as the call caused a hang when loaded into wsgi

8 years agomove django imports around, change one function call to a qualified import
Diane Trout [Tue, 17 Dec 2013 19:10:44 +0000 (11:10 -0800)]
move django imports around, change one function call to a qualified import

8 years agoUpdate exception caught for newer django?
Diane Trout [Tue, 17 Dec 2013 19:09:31 +0000 (11:09 -0800)]
Update exception caught for newer django?

8 years agouse url function call for url definition instead of just a bare tuple
Diane Trout [Tue, 17 Dec 2013 19:08:35 +0000 (11:08 -0800)]
use url function call for url definition instead of just a bare tuple

8 years agoMove use of settings default into the function instead of function definition.
Diane Trout [Tue, 17 Dec 2013 19:05:00 +0000 (11:05 -0800)]
Move use of settings default into the function instead of function definition.

This is because the if its not defined the program wont start up.

8 years agoUpdate manage script to django 1.6
Diane Trout [Tue, 10 Dec 2013 23:17:13 +0000 (15:17 -0800)]
Update manage script to django 1.6

8 years agoBeing able to filter on paired end runs might be useful
Diane Trout [Tue, 10 Dec 2013 23:15:43 +0000 (15:15 -0800)]
Being able to filter on paired end runs might be useful

8 years agoremove deprecated module django.contrib.databrowse
Diane Trout [Tue, 10 Dec 2013 23:15:20 +0000 (15:15 -0800)]
remove deprecated module django.contrib.databrowse

8 years agoWith 1.6 boolean field changed to default Null, so to be backwards compatible set...
Diane Trout [Tue, 10 Dec 2013 23:13:51 +0000 (15:13 -0800)]
With 1.6 boolean field changed to default Null, so to be backwards compatible set default false

8 years agoDjango 1.6 is pickier about enforcing referential integrity
Diane Trout [Tue, 10 Dec 2013 23:12:42 +0000 (15:12 -0800)]
Django 1.6 is pickier about enforcing referential integrity

8 years agoFix urls patterns import to be compatible with django 1.6
Diane Trout [Tue, 10 Dec 2013 23:11:54 +0000 (15:11 -0800)]
Fix urls patterns import to be compatible with django 1.6

8 years agoMerge branch 'django1.4' of mus.cacr.caltech.edu:htsworkflow into django1.4
Diane Trout [Tue, 10 Dec 2013 00:30:05 +0000 (16:30 -0800)]
Merge branch 'django1.4' of mus.cacr.caltech.edu:htsworkflow into django1.4

8 years agoMerge branch 'django1.4' of mus.cacr.caltech.edu:htsworkflow into django1.4
Diane Trout [Tue, 10 Dec 2013 00:13:48 +0000 (16:13 -0800)]
Merge branch 'django1.4' of mus.cacr.caltech.edu:htsworkflow into django1.4

8 years agoAdd new paired library types to our hackish list for library classifications
Diane Trout [Mon, 9 Dec 2013 22:44:10 +0000 (14:44 -0800)]
Add new paired library types to our hackish list for library classifications

8 years agoAttempt to be robust to not having an alignment in our RunXml file
Diane Trout [Fri, 6 Dec 2013 00:28:12 +0000 (16:28 -0800)]
Attempt to be robust to not having an alignment in our RunXml file

8 years agoAttempt to be robust to not having an alignment in our RunXml file
Diane Trout [Fri, 6 Dec 2013 00:28:12 +0000 (16:28 -0800)]
Attempt to be robust to not having an alignment in our RunXml file

8 years agoAfter implementing proper tests get a working implementation of runfolder name from...
Diane Trout [Fri, 6 Dec 2013 00:02:51 +0000 (16:02 -0800)]
After implementing proper tests get a working implementation of runfolder name from IPAR tree.

My first couple tries didn't work hopefully this one will work

8 years agoAfter implementing proper tests get a working implementation of runfolder name from...
Diane Trout [Fri, 6 Dec 2013 00:02:51 +0000 (16:02 -0800)]
After implementing proper tests get a working implementation of runfolder name from IPAR tree.

My first couple tries didn't work hopefully this one will work

8 years agofix typo
Diane Trout [Thu, 5 Dec 2013 23:28:36 +0000 (15:28 -0800)]
fix typo

8 years agofix typo
Diane Trout [Thu, 5 Dec 2013 23:28:36 +0000 (15:28 -0800)]
fix typo

8 years agoMerge branch 'django1.4' of mus.cacr.caltech.edu:htsworkflow into django1.4
Diane Trout [Thu, 5 Dec 2013 23:28:36 +0000 (15:28 -0800)]
Merge branch 'django1.4' of mus.cacr.caltech.edu:htsworkflow into django1.4

8 years agoMerge branch 'django1.4' of mus.cacr.caltech.edu:htsworkflow into django1.4
Diane Trout [Thu, 5 Dec 2013 23:06:14 +0000 (15:06 -0800)]
Merge branch 'django1.4' of mus.cacr.caltech.edu:htsworkflow into django1.4

8 years agoRecent IPAR xml config blocks include the runfolder name
Diane Trout [Thu, 5 Dec 2013 22:45:59 +0000 (14:45 -0800)]
Recent IPAR xml config blocks include the runfolder name

I was previously extracting it in the gerald step, but since we
have a number of runfolders with no aligned directory that wasn't working.

Conflicts:

htsworkflow/pipelines/test/test_runfolder_rta180.py

8 years agoUpdate for UCSC's validate manifest 1.7
Diane Trout [Thu, 5 Dec 2013 22:51:06 +0000 (14:51 -0800)]
Update for UCSC's validate manifest 1.7

8 years agoProvide reporting of what command failed when bigWigInfo isn't available.
Diane Trout [Thu, 5 Dec 2013 22:48:46 +0000 (14:48 -0800)]
Provide reporting of what command failed when bigWigInfo isn't available.

8 years agoRecent IPAR xml config blocks include the runfolder name
Diane Trout [Thu, 5 Dec 2013 22:45:59 +0000 (14:45 -0800)]
Recent IPAR xml config blocks include the runfolder name

I was previously extracting it in the gerald step, but since we
have a number of runfolders with no aligned directory that wasn't working.

8 years agoAdd a way to report what url was failing
Diane Trout [Thu, 5 Dec 2013 22:37:39 +0000 (14:37 -0800)]
Add a way to report what url was failing

8 years agoCatch and report another class of exception
Diane Trout [Mon, 2 Dec 2013 19:24:43 +0000 (11:24 -0800)]
Catch and report another class of exception

8 years agoMerge remote-tracking branch 'brisby/django1.4' into django1.4
Diane Trout [Mon, 21 Oct 2013 21:53:56 +0000 (14:53 -0700)]
Merge remote-tracking branch 'brisby/django1.4' into django1.4

8 years agoPass additional information to the trackhub.
Diane Trout [Mon, 21 Oct 2013 21:51:14 +0000 (14:51 -0700)]
Pass additional information to the trackhub.

This allows adding a term to control visibility of tracks.

There's also a hack to pull out the min/max bigWig values when generating
the trackDb.txt file.

It unfortunately depends on having bigWigInfo from UCSC on your path.

8 years agoMerge branch 'django1.4' of mus.cacr.caltech.edu:htsworkflow into django1.4
Diane Trout [Mon, 21 Oct 2013 20:59:29 +0000 (13:59 -0700)]
Merge branch 'django1.4' of mus.cacr.caltech.edu:htsworkflow into django1.4

8 years agoMerge branch 'master' into django1.4
Diane Trout [Thu, 17 Oct 2013 22:03:43 +0000 (15:03 -0700)]
Merge branch 'master' into django1.4

8 years agoAdd command line feature to srf command
Diane Trout [Thu, 17 Oct 2013 22:02:51 +0000 (15:02 -0700)]
Add command line feature to srf command

this should allow copying out fastq-equivalent files from a run.

8 years agoDynamically respond to the number of lanes.
Diane Trout [Thu, 17 Oct 2013 21:55:14 +0000 (14:55 -0700)]
Dynamically respond to the number of lanes.

Illumina now has single lane flowcells so the assumtion that there
will be 8 lanes is wrong.

8 years agoFix typo.
Diane Trout [Thu, 17 Oct 2013 21:31:29 +0000 (14:31 -0700)]
Fix typo.

8 years agoMerge branch 'django1.4' of mus.cacr.caltech.edu:htsworkflow into django1.4
Diane Trout [Thu, 17 Oct 2013 21:17:52 +0000 (14:17 -0700)]
Merge branch 'django1.4' of mus.cacr.caltech.edu:htsworkflow into django1.4

8 years agoImplement a wrapper around bigWigInfo to report bigWig summary info.
Diane Trout [Thu, 17 Oct 2013 21:12:05 +0000 (14:12 -0700)]
Implement a wrapper around bigWigInfo to report bigWig summary info.

(shocking name there I know)