• 54,564 "successful"
• 339,216 "errors"
• (not all are hostile, and not all are separate events, my log analyzing skills need improvement)

• Its like Swimming in a disease infested swamp.
• Make sure you don't have open wounds.
  • don't run unnecessary services
  • keep software up to date – especially things that use the network
• Analyzed data was from Dec 11 06:28:51 - Jan 13 11:50:22
• Only ssh – and we have throttling turned on.

• Took way too long to order.
• The initial expectations wouldn't work
• Needed several meetings to figure out how to share

• Side effect, we will need to renumber our user IDs to be compatible with CACR

• Henry is using the new server to try move things off of bad disks so we can reformat or replace with larger disks.
• (Can we buy more disks for the genome browser? Currently 1TB disks, would like to replace ~15 disks with 2TB disks.

Though right now still isn't the best time to buy disks. prices are still high due to flooding.

"With operations disrupted at more than a dozen hard disk drive (HDD) factories due to flooding in Thailand, PC manufacturers should prepare for significant supply shortages, market research firm IDC said Thursday." - computer world Nov 10, 2011

• It's actually important

• UCSC is trying to get us to describe our samples using a common vocabulary.

• Current QA Tracks

GM12878 single read RNA-seq Minus Signal Rep 1 from ENCODE/Caltech

shortLabel: GM78 1x75D - 1

Principal Investigator on grant: Myers

Lab producing data: Wold - CalTech

Experiment (Assay) type: RNA-seq

View - Peaks or Signals: Raw signal -

Experiment ID: 125

Cell, tissue or DNA sample: GM12878

Cellular compartment: Whole cell

RNA Extract: Long PolyA+ RNA

Paired/Single reads lengths: 1x75D

Replicate number: 1

Version of data if resubmitted: V3 - A better mapping technique.

Assembly originally mapped to: hg18

ENCODE Data Freeze: post ENCODE Jan 2011 Freeze

UCSC Accession: wgEncodeEH000125

Date submitted to UCSC: 2010-01-04

Date resubmitted to UCSC: 2011-02-14

Date restrictions end: 2010-10-04

Submission ID: 3514

Lab provided identifier: 11011

Mapping algorithm: TH1014

tableName: wgEncodeCaltechRnaSeqGm12878R1x75dMinusSignalRep1V3

fileName: wgEncodeCaltechRnaSeqGm12878R1x75dMinusSignalRep1V3.bigWig

• We call it LHCN-cycling vs LHCN-differentiated
• UCSC/NIH decided there is only the LHCN cell line
• It has two treatments None, and 7 day differentiated
• If you have admin priviledges on jumpgate, use LHCN

• Georgi preps an analysis
• hands files to me
• I figure out what meta-data UCSC wants, and try to shove into databases
• Generate meta-data files
• upload
• discuss what went wrong.

• Additional Meta-data is ENCODE specific
• Labs in ENCODE have enough trouble agreeing on terms.
• Igor pointed out worm and chemical assay groups would have wildly different needs.

• Results of recent meeting with Cricket/Venkat

• 7 samples, no replicates
• Uploaded fastqs only for 11425 11640 11888 11890 11904 11908
• Whose talking to Flo about these samples?

• According to Cricket, Peter Goode said it wasn't core
  • so UCSC wasn't planning on accepting it
  • Perhaps we would be interested in uploading to GEOs UCSC page?
  • (Not much visualization, just a place to download data)
• If this is not acceptable someone needs to get Peter Goode to tell Cricket to accept the data.

March 31

• What we were supposed to be submitting
• Who is the responsible person for generating samples
• Are they on these lists?
• Encode
• Mouse Encode

• Whats next for tracking our data & processing

• extract results for HiSeq runs.
• (well first define what run information to save from HiSeq)
• display HiSeq summary reports.

The biggest problem(1) our lab faces is sharing knowledge about our experiments

(1) (that I have a chance of doing anything about)

• Library 11011 is it good or not?
• 10155?

The jumpgate database is tracking "samples" handled by the gene expression center. Some QA features apply to all experiment types, but some are specific to the lab & experiment.

There's also disagreement about the validity of a QA method.

• Reports generated by pipeline
  • e.g. intensity plots
• read counts
• report average quality scores
• many other things I don't know about?

• Uniformity measurement
• 5'/3' read distribution
• Reads aligned to introns

• Adding to jumpgate would be slow & cluttered
• bolting on random features leads to clunky software
  • see MS Word, iTunes
• it might work if jumpgate was just for us, but other labs will have their own needs, and our analysis may or may not apply.
• Not only does it makes me a bottle-neck, I'm not qualified to pick among the QC methods

• What we want is a central place to find information about an experiment.
  • not a just, is a sample good
  • but, why did we make it?
  • is it related to another sample?
  • what analysis has been done on it.

• Ye olden days had meetings with shouting over mead and pointing at each other with haunches of meat…
  • and secretaries who took notes and mailed them to people
• Has issue in the modern distributed workforce
• Also organizations are prone to forgetting.

• https://woldlab.caltech.edu/wiki/RecentChanges
• https://jumpgate.caltech.edu/wiki/RecentChanges
  • You all can edit the first site, and probably the second site.
  • Its mostly been me and henry who've edited.
• The encode wiki gets more use.
• http://encodewiki.ucsc.edu/EncodeDCC/index.php/Special:RecentChanges

Tried to propose:

http://jumpgate.caltech.edu/wiki/Library/10155

• Would need a few custom macros to "transclude" jumpgate data
• I'm not good at getting buy in.

A directory

• write reports to directory
• Like 70JBCAAXX

• Another idea is to make the library page aware of people linking to it
• e.g. 10155-example.html

• An accessible, usable place to write about an experiment
• A Willingness to write things down.
• It needs to be searchable
• It needs to support showing text, plots, tables, images.
  • Physical objects?
• It should allow sharing and reusing code.

• Jumpgate needs improvements in letting downstream users know there's new data.
• Subscribe to a library?
  • Get an email/im/RSS entry every time new data is added?

After enough abstraction the QC problem turns into

• How do we take notes and share them.

see earlier Wiki

• Tracks process well
• I've seen projects that you could unarchive, type "make paper" and re-run the analysis and get the same paper out the other end.
• Great for tracking processes, less good for why? (Unless you aim for "$ make paper")
• Titus seems to be using it for his lab
• (Also what about relationships between datasets?)

Like: https://mus.cacr.caltech.edu/htsworkflow/

• Its a wiki, but adds ticket tracking
• can be linked to show source code.
• And maybe a gantt chart

(aside)

• Imagine deadlines in a gantt chart.
• Imagine showing estimated duration for sequencing and analysis.

• web based tool for managing experiments
• http://galaxyproject.org
• http://main.g2.bx.psu.edu/u/nickpolato/h/apalmataassembly
• Sysadmins were resistant because we don't think we have a workable database server.

• many people find easy to use
  • information gets locked up on laptops
  • tracking versions is hard. (3-way merge is by hand)
  • or maybe not? google turned up subversion for word files?

• server to host MS Office content
  • "content management"
  • "document management"
• My knowledge is limited
• other, similar, tools might even work

• It's python mathematica in a web page!
• Titus & iPython developers are thinking about developing workflows in ipython notebook.
  • I think this'd be develop a notebook to analyze a dataset
  • The start would have a header defining where the dataset is located
  • Copy notebook, change input?

• http://orgmode.org
• Its everything in an outline in Emacs.

• Instead of organized around a topic like a wiki, blogs are date
• http://ivory.idyll.org/blog/

• I'm bad at picking
  • between lots of functionally similar choices.
  • things that other people will willingly use.

• Blog
• Wiki
• twitter/facebook/google+
• RSS Reader
• Mailing list.

• You are taking notes aren't you?