Imported Upstream version 0.1.9

diff --git a/ChangeLog b/ChangeLog
index 6b0ff6cf4e80c0a9a1e989847e86bd1bba9434fe..12908c968c54e264baff1562e7cea623fb145dcb 100644 (file)
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,1307 @@
+------------------------------------------------------------------------
+r782 | lh3lh3 | 2010-10-27 19:58:54 -0400 (Wed, 27 Oct 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam_plcmd.c
+
+fixed a silly bug in pileup
+
+------------------------------------------------------------------------
+r781 | lh3lh3 | 2010-10-27 14:39:48 -0400 (Wed, 27 Oct 2010) | 5 lines
+Changed paths:
+   M /trunk/samtools/ChangeLog
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bam_sort.c
+   M /trunk/samtools/bamtk.c
+   M /trunk/samtools/samtools.1
+
+ * samtools-0.1.8-22 (r781)
+ * made BAQ the default behavior of mpileup
+ * updated manual
+ * in merge, force to exit given inconsistent header when "-R" is not in use.
+
+------------------------------------------------------------------------
+r780 | lh3lh3 | 2010-10-27 11:01:11 -0400 (Wed, 27 Oct 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam.h
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-21 (r780)
+ * minor speedup to pileup
+
+------------------------------------------------------------------------
+r779 | lh3lh3 | 2010-10-27 09:58:56 -0400 (Wed, 27 Oct 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam_pileup.c
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/examples/toy.sam
+
+improve pileup a little bit
+
+------------------------------------------------------------------------
+r778 | lh3lh3 | 2010-10-27 00:14:43 -0400 (Wed, 27 Oct 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam.h
+   M /trunk/samtools/bam_pileup.c
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bam_tview.c
+   M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-20 (r778)
+ * speed up pileup, although I do not know how much is the improvement
+
+------------------------------------------------------------------------
+r777 | lh3lh3 | 2010-10-26 17:26:04 -0400 (Tue, 26 Oct 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam_maqcns.c
+   M /trunk/samtools/bam_maqcns.h
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bamtk.c
+   M /trunk/samtools/examples/Makefile
+
+ * samtools-0.1.8-19 (r777)
+ * integrate mpileup features to pileup: min_baseQ, capQ, prob_realn, paired-only and biased prior
+
+------------------------------------------------------------------------
+r776 | lh3lh3 | 2010-10-26 15:27:46 -0400 (Tue, 26 Oct 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam_md.c
+
+remove local realignment (probabilistic realignment is still there)
+
+------------------------------------------------------------------------
+r774 | jmarshall | 2010-10-21 06:52:38 -0400 (Thu, 21 Oct 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/sam_view.c
+
+Add the relevant filename or region to error messages, and cause a failure
+exit status where appropriate.  Based on a patch provided by Marcel Martin.
+
+------------------------------------------------------------------------
+r773 | lh3lh3 | 2010-10-19 19:44:31 -0400 (Tue, 19 Oct 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/examples/toy.sam
+   M /trunk/samtools/kaln.c
+
+ * Minor code changes. No real effect.
+ * change quality to 30 in toy.sam
+
+------------------------------------------------------------------------
+r772 | lh3lh3 | 2010-10-18 23:40:13 -0400 (Mon, 18 Oct 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/examples/toy.fa
+   M /trunk/samtools/examples/toy.sam
+
+added another toy example
+
+------------------------------------------------------------------------
+r771 | lh3lh3 | 2010-10-13 23:32:12 -0400 (Wed, 13 Oct 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/call1.c
+   M /trunk/samtools/bcftools/ld.c
+   M /trunk/samtools/bcftools/vcfutils.pl
+
+improve the LD statistics
+
+------------------------------------------------------------------------
+r770 | lh3lh3 | 2010-10-12 23:49:26 -0400 (Tue, 12 Oct 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bcftools/call1.c
+   M /trunk/samtools/bcftools/vcfutils.pl
+
+ * a minor fix to the -L option
+ * add ldstats to vcfutils.pl
+
+------------------------------------------------------------------------
+r769 | lh3lh3 | 2010-10-12 15:51:57 -0400 (Tue, 12 Oct 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/bcf.c
+
+a minor change
+
+------------------------------------------------------------------------
+r768 | lh3lh3 | 2010-10-12 15:49:06 -0400 (Tue, 12 Oct 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/Makefile
+   A /trunk/samtools/bcftools/ld.c
+
+forget to add the key file
+
+------------------------------------------------------------------------
+r767 | lh3lh3 | 2010-10-12 15:48:46 -0400 (Tue, 12 Oct 2010) | 4 lines
+Changed paths:
+   M /trunk/samtools/bcftools/Makefile
+   M /trunk/samtools/bcftools/bcf.c
+   M /trunk/samtools/bcftools/bcf.h
+   M /trunk/samtools/bcftools/call1.c
+   M /trunk/samtools/bcftools/prob1.c
+   M /trunk/samtools/bcftools/vcfutils.pl
+
+ * vcfutils.pl: fixed a typo in help message
+ * added APIs: bcf_append_info() and bcf_cpy()
+ * calculate adjacent LD
+
+------------------------------------------------------------------------
+r766 | lh3lh3 | 2010-10-11 11:06:40 -0400 (Mon, 11 Oct 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/vcfutils.pl
+
+added filter for samtools/bcftools genetated VCFs
+
+------------------------------------------------------------------------
+r765 | lh3lh3 | 2010-10-05 14:05:18 -0400 (Tue, 05 Oct 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bcftools/vcfutils.pl
+   M /trunk/samtools/kaln.c
+
+ * removed a comment line in kaln.c
+ * vcfutils.pl fillac works when GT is not the first field
+
+------------------------------------------------------------------------
+r764 | petulda | 2010-10-05 08:59:36 -0400 (Tue, 05 Oct 2010) | 1 line
+Changed paths:
+   A /trunk/samtools/bcftools/bcf-fix.pl
+
+Convert VCF output of "bcftools view -bgcv" to a valid VCF file
+------------------------------------------------------------------------
+r763 | lh3lh3 | 2010-10-02 22:51:03 -0400 (Sat, 02 Oct 2010) | 4 lines
+Changed paths:
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bamtk.c
+   A /trunk/samtools/bcftools/bcftools.1
+   M /trunk/samtools/bcftools/call1.c
+   M /trunk/samtools/samtools.1
+
+ * samtools-0.1.8-18 (r763)
+ * added bcftools manual page
+ * minor fix to mpileup and view command lines
+
+------------------------------------------------------------------------
+r762 | lh3lh3 | 2010-10-02 21:46:25 -0400 (Sat, 02 Oct 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bcftools/bcf.c
+   M /trunk/samtools/bcftools/call1.c
+   M /trunk/samtools/bcftools/vcfutils.pl
+
+ * vcfutils.pl qstats: calculate marginal ts/tv
+ * allow to call genotypes at variant sites
+
+------------------------------------------------------------------------
+r761 | lh3lh3 | 2010-10-01 00:29:55 -0400 (Fri, 01 Oct 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/kaln.c
+   M /trunk/samtools/misc/HmmGlocal.java
+
+I am changing the gap open probability back to 0.001. It seems that
+being conservative here is a good thing...
+
+------------------------------------------------------------------------
+r760 | lh3lh3 | 2010-10-01 00:11:27 -0400 (Fri, 01 Oct 2010) | 5 lines
+Changed paths:
+   M /trunk/samtools/bamtk.c
+   M /trunk/samtools/kaln.c
+   A /trunk/samtools/misc/HmmGlocal.java
+
+ * samtools-0.1.8-17 (r760)
+ * the default gap open penalty is too small (a typo)
+ * added comments on hmm_realn
+ * Java implementation
+
+------------------------------------------------------------------------
+r759 | lh3lh3 | 2010-09-30 10:12:54 -0400 (Thu, 30 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bamtk.c
+
+mark samtools-0.1.8-16 (r759)
+
+------------------------------------------------------------------------
+r758 | lh3lh3 | 2010-09-30 10:12:02 -0400 (Thu, 30 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/kaln.c
+
+round to the nearest integer
+
+------------------------------------------------------------------------
+r757 | lh3lh3 | 2010-09-28 17:16:43 -0400 (Tue, 28 Sep 2010) | 4 lines
+Changed paths:
+   M /trunk/samtools/kaln.c
+
+I was trying to accelerate ka_prob_glocal() as this will be the
+bottleneck. After an hour, the only gain is to change division to
+multiplication. OK. I will stop.
+
+------------------------------------------------------------------------
+r756 | lh3lh3 | 2010-09-28 16:57:49 -0400 (Tue, 28 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/kaln.c
+
+this is interesting. multiplication is much faster than division, at least on my Mac
+
+------------------------------------------------------------------------
+r755 | lh3lh3 | 2010-09-28 16:19:13 -0400 (Tue, 28 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/kaln.c
+
+minor changes
+
+------------------------------------------------------------------------
+r754 | lh3lh3 | 2010-09-28 15:44:16 -0400 (Tue, 28 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam_md.c
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/kaln.c
+
+prob_realn() seems working!
+
+------------------------------------------------------------------------
+r753 | lh3lh3 | 2010-09-28 12:48:23 -0400 (Tue, 28 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/kaln.c
+
+minor
+
+------------------------------------------------------------------------
+r752 | lh3lh3 | 2010-09-28 12:47:41 -0400 (Tue, 28 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/kaln.c
+   M /trunk/samtools/kaln.h
+
+Convert phredQ to probabilities
+
+------------------------------------------------------------------------
+r751 | lh3lh3 | 2010-09-28 12:32:08 -0400 (Tue, 28 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/kaln.c
+   M /trunk/samtools/kaln.h
+
+Implement the glocal HMM; discard the extention HMM
+
+------------------------------------------------------------------------
+r750 | lh3lh3 | 2010-09-28 00:06:11 -0400 (Tue, 28 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/kaln.c
+
+improve numerical stability
+
+------------------------------------------------------------------------
+r749 | lh3lh3 | 2010-09-27 23:27:54 -0400 (Mon, 27 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/kaln.c
+
+more comments
+
+------------------------------------------------------------------------
+r748 | lh3lh3 | 2010-09-27 23:17:16 -0400 (Mon, 27 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/kaln.c
+
+fixed a bug in banded DP
+
+------------------------------------------------------------------------
+r747 | lh3lh3 | 2010-09-27 23:05:12 -0400 (Mon, 27 Sep 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/kaln.c
+
+ * fixed that weird issue.
+ * the banded version is NOT working
+
+------------------------------------------------------------------------
+r746 | lh3lh3 | 2010-09-27 22:57:05 -0400 (Mon, 27 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/kaln.c
+
+More comments. This version seems working, but something is a little weird...
+
+------------------------------------------------------------------------
+r745 | lh3lh3 | 2010-09-27 17:21:40 -0400 (Mon, 27 Sep 2010) | 6 lines
+Changed paths:
+   M /trunk/samtools/kaln.c
+
+A little code cleanup. Now the forward and backback algorithms give
+nearly identical P(x), which means both are close to the correct
+forms. However, I have only tested on toy examples. Minor errors in
+the implementation may not be obvious.
+
+
+------------------------------------------------------------------------
+r744 | lh3lh3 | 2010-09-27 16:55:15 -0400 (Mon, 27 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bam_sort.c
+   M /trunk/samtools/kaln.c
+   M /trunk/samtools/kaln.h
+
+...
+
+------------------------------------------------------------------------
+r743 | jmarshall | 2010-09-27 08:19:06 -0400 (Mon, 27 Sep 2010) | 6 lines
+Changed paths:
+   M /trunk/samtools/bam_sort.c
+
+Abort if merge -h's INH.SAM cannot be opened, just as we abort
+if any of the IN#.BAM input files cannot be opened.
+
+Also propagate any error indication returned by bam_merge_core()
+to samtools merge's exit status.
+
+------------------------------------------------------------------------
+r741 | jmarshall | 2010-09-24 11:08:24 -0400 (Fri, 24 Sep 2010) | 5 lines
+Changed paths:
+   M /trunk/samtools/bam_index.c
+
+Use bam_validate1() to detect garbage records in the event of a corrupt
+BAI index file that causes a bam_seek() to an invalid position.  At most
+one record (namely, the bam_iter_read terminator) is tested per bam_fetch()
+call, so the cost is insignificant in the normal case.
+
+------------------------------------------------------------------------
+r740 | jmarshall | 2010-09-24 11:00:19 -0400 (Fri, 24 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam.c
+   M /trunk/samtools/bam.h
+
+Add bam_validate1().
+
+------------------------------------------------------------------------
+r739 | lh3lh3 | 2010-09-22 12:07:50 -0400 (Wed, 22 Sep 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam_md.c
+   M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-15 (r379)
+ * allow to change capQ parameter in calmd
+
+------------------------------------------------------------------------
+r738 | jmarshall | 2010-09-22 11:15:33 -0400 (Wed, 22 Sep 2010) | 13 lines
+Changed paths:
+   M /trunk/samtools/bam_index.c
+   M /trunk/samtools/sam_view.c
+
+When bam_read1() returns an error (return value <= -2), propagate that error
+to bam_iter_read()'s own return value.  Similarly, also propagate it up to
+bam_fetch()'s return value.  Previously bam_fetch() always returned 0, and
+callers ignored its return value anyway.  With this change, 0 continues to
+indicate success, while <= -2 (which can be written as < 0, as -1 is never
+returned) indicates corrupted input.
+
+bam_iter_read() ought also to propagate errors returned by bam_seek().
+
+main_samview() can now print an error message and fail when bam_fetch()
+detects that a .bai index file is corrupted or otherwise does not correspond
+to the .bam file it is being used with.
+
+------------------------------------------------------------------------
+r737 | jmarshall | 2010-09-22 10:47:42 -0400 (Wed, 22 Sep 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam_index.c
+
+0 is a successful return value from bam_read1().  (In practice, it never
+returns 0 anyway; but all the other callers treat 0 as successful.)
+
+------------------------------------------------------------------------
+r736 | lh3lh3 | 2010-09-20 17:43:08 -0400 (Mon, 20 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam.h
+   M /trunk/samtools/bam_index.c
+   M /trunk/samtools/bam_sort.c
+
+ * merge files region-by-region. work on small examples but more tests are needed.
+
+------------------------------------------------------------------------
+r735 | lh3lh3 | 2010-09-20 16:56:24 -0400 (Mon, 20 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/vcfutils.pl
+
+improve qstats by checking the alleles as well
+
+------------------------------------------------------------------------
+r734 | lh3lh3 | 2010-09-17 18:12:13 -0400 (Fri, 17 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/vcfutils.pl
+
+convert UCSC SNP SQL dump to VCF
+
+------------------------------------------------------------------------
+r733 | lh3lh3 | 2010-09-17 13:02:11 -0400 (Fri, 17 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/vcfutils.pl
+
+hapmap2vcf convertor
+
+------------------------------------------------------------------------
+r732 | lh3lh3 | 2010-09-17 10:11:37 -0400 (Fri, 17 Sep 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bcftools/Makefile
+   M /trunk/samtools/bcftools/bcf.c
+   M /trunk/samtools/bcftools/bcf.h
+   M /trunk/samtools/bcftools/vcf.c
+
+ * added comments
+ * VCF->BCF is not possible without knowing the sequence dictionary before hand...
+
+------------------------------------------------------------------------
+r731 | lh3lh3 | 2010-09-17 09:15:53 -0400 (Fri, 17 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam2bcf.c
+   M /trunk/samtools/bcftools/bcf.c
+   M /trunk/samtools/bcftools/bcf.h
+   M /trunk/samtools/bcftools/bcfutils.c
+   M /trunk/samtools/bcftools/call1.c
+   M /trunk/samtools/bcftools/vcf.c
+
+ * put n_smpl to "bcf1_t" to simplify API a little
+
+------------------------------------------------------------------------
+r730 | lh3lh3 | 2010-09-16 21:36:01 -0400 (Thu, 16 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/bcf.h
+   M /trunk/samtools/bcftools/call1.c
+   M /trunk/samtools/bcftools/index.c
+
+fixed a bug in indexing
+
+------------------------------------------------------------------------
+r729 | lh3lh3 | 2010-09-16 16:54:48 -0400 (Thu, 16 Sep 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam.c
+   M /trunk/samtools/bam_md.c
+   M /trunk/samtools/bam_pileup.c
+
+ * fixed a bug in capQ
+ * valgrind identifies a use of uninitialised value, but I have not fixed it.
+
+------------------------------------------------------------------------
+r728 | lh3lh3 | 2010-09-16 15:03:59 -0400 (Thu, 16 Sep 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bgzip.c
+   M /trunk/samtools/razip.c
+
+ * fixed a bug in razip: -c will delete the input file
+ * copy tabix/bgzip to here
+
+------------------------------------------------------------------------
+r727 | lh3lh3 | 2010-09-16 13:45:49 -0400 (Thu, 16 Sep 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam_md.c
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-14 (r727)
+ * allow to change the capQ parameter at the command line
+
+------------------------------------------------------------------------
+r726 | lh3lh3 | 2010-09-16 13:38:43 -0400 (Thu, 16 Sep 2010) | 4 lines
+Changed paths:
+   M /trunk/samtools/bam_md.c
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bcftools/vcfutils.pl
+   M /trunk/samtools/misc/samtools.pl
+
+ * added varFilter to vcfutils.pl
+ * reimplement realn(). now it performs a local alignment
+ * added cap_mapQ() to cap mapping quality when there are many substitutions
+
+------------------------------------------------------------------------
+r724 | lh3lh3 | 2010-09-15 00:18:31 -0400 (Wed, 15 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/Makefile
+   A /trunk/samtools/bcftools/bcf2qcall.c
+   M /trunk/samtools/bcftools/call1.c
+
+ * convert BCF to QCALL input
+
+------------------------------------------------------------------------
+r723 | lh3lh3 | 2010-09-14 22:41:50 -0400 (Tue, 14 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam_md.c
+
+dynamic band width in realignment
+
+------------------------------------------------------------------------
+r722 | lh3lh3 | 2010-09-14 22:05:32 -0400 (Tue, 14 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam_md.c
+   M /trunk/samtools/bam_plcmd.c
+
+fixed a bug in realignment
+
+------------------------------------------------------------------------
+r721 | lh3lh3 | 2010-09-14 20:54:09 -0400 (Tue, 14 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/prob1.c
+
+fixed a minor issue
+
+------------------------------------------------------------------------
+r720 | lh3lh3 | 2010-09-14 19:25:10 -0400 (Tue, 14 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/Makefile
+   M /trunk/samtools/bam_maqcns.c
+   M /trunk/samtools/bam_md.c
+
+fixed a bug in realignment
+
+------------------------------------------------------------------------
+r719 | lh3lh3 | 2010-09-14 19:18:24 -0400 (Tue, 14 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam_plcmd.c
+
+minor changes. It is BUGGY now!
+
+------------------------------------------------------------------------
+r718 | lh3lh3 | 2010-09-14 16:32:33 -0400 (Tue, 14 Sep 2010) | 4 lines
+Changed paths:
+   M /trunk/samtools/bam_md.c
+   M /trunk/samtools/bam_pileup.c
+   M /trunk/samtools/kaln.c
+   M /trunk/samtools/kaln.h
+
+ * aggressive gapped aligner is implemented in calmd.
+ * distinguish gap_open and gap_end_open in banded alignment
+ * make tview accepts alignment with heading and tailing D
+
+------------------------------------------------------------------------
+r717 | jmarshall | 2010-09-14 09:04:28 -0400 (Tue, 14 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools
+
+Add svn:ignore properties for generated files that don't appear in "make all".
+
+------------------------------------------------------------------------
+r716 | jmarshall | 2010-09-13 08:37:53 -0400 (Mon, 13 Sep 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools
+   M /trunk/samtools/bcftools
+   M /trunk/samtools/misc
+
+Add svn:ignore properties listing the generated files.
+(Except for *.o, which we'll assume is in global-ignores.)
+
+------------------------------------------------------------------------
+r715 | lh3lh3 | 2010-09-08 12:53:55 -0400 (Wed, 08 Sep 2010) | 5 lines
+Changed paths:
+   M /trunk/samtools/bamtk.c
+   M /trunk/samtools/bcftools/call1.c
+   M /trunk/samtools/bcftools/prob1.c
+   M /trunk/samtools/sample.c
+   M /trunk/samtools/sample.h
+
+ * samtools-0.1.8-13 (r715)
+ * fixed a bug in identifying SM across files
+ * bcftools: estimate heterozygosity
+ * bcftools: allow to skip sites without reference bases
+
+------------------------------------------------------------------------
+r713 | lh3lh3 | 2010-09-03 17:19:12 -0400 (Fri, 03 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/Makefile
+   M /trunk/samtools/bcftools/call1.c
+   M /trunk/samtools/bcftools/prob1.c
+   M /trunk/samtools/bcftools/prob1.h
+
+quite a lot changes to the contrast caller, but I still feel something is missing...
+
+------------------------------------------------------------------------
+r711 | lh3lh3 | 2010-09-03 00:30:48 -0400 (Fri, 03 Sep 2010) | 4 lines
+Changed paths:
+   M /trunk/samtools/bcftools/Makefile
+   M /trunk/samtools/bcftools/call1.c
+   M /trunk/samtools/bcftools/prob1.c
+   M /trunk/samtools/bcftools/vcfutils.pl
+
+ * changed 3.434 to 4.343 (typo!)
+ * fixed a bug in the contrast caller
+ * calculate heterozygosity
+
+------------------------------------------------------------------------
+r710 | lh3lh3 | 2010-09-01 23:24:47 -0400 (Wed, 01 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/bcf.h
+   M /trunk/samtools/bcftools/bcfutils.c
+   M /trunk/samtools/bcftools/call1.c
+
+SNP calling from the GL field
+
+------------------------------------------------------------------------
+r709 | lh3lh3 | 2010-09-01 18:52:30 -0400 (Wed, 01 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/vcf.c
+
+fixed another problem
+
+------------------------------------------------------------------------
+r708 | lh3lh3 | 2010-09-01 18:31:17 -0400 (Wed, 01 Sep 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bcftools/bcf.c
+   M /trunk/samtools/bcftools/vcf.c
+
+ * fixed bugs in parsing VCF
+ * parser now works with GT/GQ/DP/PL/GL
+
+------------------------------------------------------------------------
+r707 | lh3lh3 | 2010-09-01 15:28:29 -0400 (Wed, 01 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/Makefile
+   M /trunk/samtools/bcftools/prob1.c
+
+Do not compile _BCF_QUAD by default
+
+------------------------------------------------------------------------
+r706 | lh3lh3 | 2010-09-01 15:21:41 -0400 (Wed, 01 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/bcf.c
+   M /trunk/samtools/bcftools/bcf.h
+   M /trunk/samtools/bcftools/bcfutils.c
+   M /trunk/samtools/bcftools/call1.c
+
+Write the correct ALT and PL in the SNP calling mode.
+
+------------------------------------------------------------------------
+r705 | lh3lh3 | 2010-09-01 12:50:33 -0400 (Wed, 01 Sep 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/vcfutils.pl
+
+more commands for my own uses
+
+------------------------------------------------------------------------
+r704 | lh3lh3 | 2010-09-01 09:26:10 -0400 (Wed, 01 Sep 2010) | 2 lines
+Changed paths:
+   A /trunk/samtools/bcftools/vcfutils.pl
+
+Utilities for processing VCF
+
+------------------------------------------------------------------------
+r703 | lh3lh3 | 2010-08-31 16:44:57 -0400 (Tue, 31 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/Makefile
+   M /trunk/samtools/bcftools/call1.c
+   M /trunk/samtools/bcftools/prob1.c
+   M /trunk/samtools/bcftools/prob1.h
+
+preliminary contrast variant caller
+
+------------------------------------------------------------------------
+r702 | lh3lh3 | 2010-08-31 12:28:39 -0400 (Tue, 31 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/call1.c
+   M /trunk/samtools/bcftools/prob1.c
+   M /trunk/samtools/bcftools/prob1.h
+
+z' and z'' can be calculated
+
+------------------------------------------------------------------------
+r701 | lh3lh3 | 2010-08-31 10:20:57 -0400 (Tue, 31 Aug 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bcftools/Makefile
+   A /trunk/samtools/bcftools/call1.c (from /trunk/samtools/bcftools/vcfout.c:699)
+   M /trunk/samtools/bcftools/prob1.c
+   D /trunk/samtools/bcftools/vcfout.c
+
+ * rename vcfout.c as call1.c
+ * prepare to add two-sample comparison
+
+------------------------------------------------------------------------
+r699 | lh3lh3 | 2010-08-24 15:28:16 -0400 (Tue, 24 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/vcfout.c
+
+fixed a bug in calculating the t statistics
+
+------------------------------------------------------------------------
+r698 | lh3lh3 | 2010-08-24 14:05:50 -0400 (Tue, 24 Aug 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam2bcf.c
+   M /trunk/samtools/bam2bcf.h
+   M /trunk/samtools/bamtk.c
+   M /trunk/samtools/bcftools/kfunc.c
+   M /trunk/samtools/bcftools/vcfout.c
+
+ * samtools-0.1.8-13 (r698)
+ * perform one-tailed t-test for baseQ, mapQ and endDist
+
+------------------------------------------------------------------------
+r697 | lh3lh3 | 2010-08-24 12:30:13 -0400 (Tue, 24 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/kfunc.c
+
+added regularized incomplete beta function
+
+------------------------------------------------------------------------
+r695 | lh3lh3 | 2010-08-23 17:36:17 -0400 (Mon, 23 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam_maqcns.c
+   M /trunk/samtools/bam_plcmd.c
+
+change the default correlation coefficient
+
+------------------------------------------------------------------------
+r694 | lh3lh3 | 2010-08-23 14:46:52 -0400 (Mon, 23 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/bcf.c
+   M /trunk/samtools/bcftools/vcfout.c
+
+print QUAL as floating numbers
+
+------------------------------------------------------------------------
+r693 | lh3lh3 | 2010-08-23 14:06:07 -0400 (Mon, 23 Aug 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/Makefile
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bamtk.c
+   M /trunk/samtools/examples/Makefile
+   A /trunk/samtools/sample.c
+   A /trunk/samtools/sample.h
+
+ * samtools-0.1.8-12 (r692)
+ * group data by samples in "mpileup -g"
+
+------------------------------------------------------------------------
+r692 | lh3lh3 | 2010-08-23 10:58:53 -0400 (Mon, 23 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/Makefile
+   D /trunk/samtools/bam_mcns.c
+   D /trunk/samtools/bam_mcns.h
+   M /trunk/samtools/bam_plcmd.c
+
+remove VCF output in mpileup
+
+------------------------------------------------------------------------
+r691 | lh3lh3 | 2010-08-23 10:48:20 -0400 (Mon, 23 Aug 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam2bcf.c
+   M /trunk/samtools/bam2bcf.h
+
+ * use the revised MAQ error model for mpileup
+ * prepare to remove the independent model from mpileup
+
+------------------------------------------------------------------------
+r690 | lh3lh3 | 2010-08-20 15:46:40 -0400 (Fri, 20 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/Makefile
+   M /trunk/samtools/bam_maqcns.c
+   M /trunk/samtools/bam_maqcns.h
+   M /trunk/samtools/bam_plcmd.c
+   A /trunk/samtools/errmod.c
+   A /trunk/samtools/errmod.h
+   M /trunk/samtools/ksort.h
+
+added revised MAQ error model
+
+------------------------------------------------------------------------
+r689 | lh3lh3 | 2010-08-18 09:55:20 -0400 (Wed, 18 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/prob1.c
+   M /trunk/samtools/bcftools/prob1.h
+   M /trunk/samtools/bcftools/vcfout.c
+
+allow to read the prior from the error output. EM iteration is working.
+
+------------------------------------------------------------------------
+r688 | lh3lh3 | 2010-08-17 12:12:20 -0400 (Tue, 17 Aug 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bcftools/main.c
+   M /trunk/samtools/bcftools/vcf.c
+
+ * write a little more VCF header
+ * concatenate BCFs
+
+------------------------------------------------------------------------
+r687 | lh3lh3 | 2010-08-16 20:53:16 -0400 (Mon, 16 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/bcf.c
+   M /trunk/samtools/bcftools/bcf.h
+   M /trunk/samtools/bcftools/bcf.tex
+
+use float for QUAL
+
+------------------------------------------------------------------------
+r686 | lh3lh3 | 2010-08-14 00:11:13 -0400 (Sat, 14 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/bcf.c
+   M /trunk/samtools/bcftools/prob1.c
+
+faster for large sample size (in principle)
+
+------------------------------------------------------------------------
+r685 | lh3lh3 | 2010-08-13 23:28:31 -0400 (Fri, 13 Aug 2010) | 4 lines
+Changed paths:
+   M /trunk/samtools/bcftools/prob1.c
+
+ * a numerically stable method to calculate z_{jk}
+ * currently slower than the old method but will be important for large sample size
+ * in principle, we can speed up for large n, but have not tried
+
+------------------------------------------------------------------------
+r684 | lh3lh3 | 2010-08-11 21:58:31 -0400 (Wed, 11 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/vcfout.c
+
+fixed an issue in parsing integer
+
+------------------------------------------------------------------------
+r683 | lh3lh3 | 2010-08-09 13:05:07 -0400 (Mon, 09 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/bcf.c
+
+do not print refname if file is converted from VCF
+
+------------------------------------------------------------------------
+r682 | lh3lh3 | 2010-08-09 12:59:47 -0400 (Mon, 09 Aug 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bcftools/vcf.c
+
+ * parse PL
+ * fixed a bug in parsing VCF
+
+------------------------------------------------------------------------
+r681 | lh3lh3 | 2010-08-09 12:49:23 -0400 (Mon, 09 Aug 2010) | 4 lines
+Changed paths:
+   M /trunk/samtools/bcftools/bcf.c
+   M /trunk/samtools/bcftools/bcf.h
+   M /trunk/samtools/bcftools/bcfutils.c
+   M /trunk/samtools/bcftools/main.c
+   M /trunk/samtools/bcftools/vcf.c
+   M /trunk/samtools/bcftools/vcfout.c
+   M /trunk/samtools/bgzf.c
+   M /trunk/samtools/kstring.c
+
+ * fixed a bug in kstrtok@kstring.c
+ * preliminary VCF parser (not parse everything for now)
+ * improved view interface
+
+------------------------------------------------------------------------
+r680 | lh3lh3 | 2010-08-09 10:43:13 -0400 (Mon, 09 Aug 2010) | 4 lines
+Changed paths:
+   M /trunk/samtools/bcftools/bcf.c
+   M /trunk/samtools/bcftools/bcf.h
+   M /trunk/samtools/bcftools/vcfout.c
+   M /trunk/samtools/kstring.c
+   M /trunk/samtools/kstring.h
+
+ * improved kstring (added kstrtok)
+ * removed the limit on the format string length in bcftools
+ * use kstrtok to parse format which fixed a bug in the old code
+
+------------------------------------------------------------------------
+r679 | lh3lh3 | 2010-08-09 01:12:05 -0400 (Mon, 09 Aug 2010) | 2 lines
+Changed paths:
+   A /trunk/samtools/bcftools/README
+   M /trunk/samtools/bcftools/vcfout.c
+
+help messages
+
+------------------------------------------------------------------------
+r678 | lh3lh3 | 2010-08-09 00:01:52 -0400 (Mon, 09 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/vcfout.c
+
+perform single-tail test for ED4
+
+------------------------------------------------------------------------
+r677 | lh3lh3 | 2010-08-08 23:48:35 -0400 (Sun, 08 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/Makefile
+   M /trunk/samtools/bcftools/kfunc.c
+   M /trunk/samtools/bcftools/vcfout.c
+
+ * test depth, end distance and HWE
+
+------------------------------------------------------------------------
+r676 | lh3lh3 | 2010-08-08 02:04:15 -0400 (Sun, 08 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/kfunc.c
+
+reimplement incomplete gamma functions. no copy-paste
+
+------------------------------------------------------------------------
+r675 | lh3lh3 | 2010-08-06 22:42:54 -0400 (Fri, 06 Aug 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam2bcf.c
+   M /trunk/samtools/bam2bcf.h
+   M /trunk/samtools/bcftools/fet.c
+   M /trunk/samtools/bcftools/prob1.c
+   M /trunk/samtools/bcftools/prob1.h
+   M /trunk/samtools/bcftools/vcfout.c
+
+ * bcftools: add HWE (no testing for now)
+ * record end dist in a 2x2 table, not avg, std any more
+
+------------------------------------------------------------------------
+r674 | lh3lh3 | 2010-08-06 17:30:16 -0400 (Fri, 06 Aug 2010) | 3 lines
+Changed paths:
+   A /trunk/samtools/bcftools/kfunc.c
+
+ * Special functions: log(gamma()), erfc(), P(a,x) (incomplete gamma)
+ * Not using Numerical Recipe due to licensing issues
+
+------------------------------------------------------------------------
+r673 | lh3lh3 | 2010-08-05 23:46:53 -0400 (Thu, 05 Aug 2010) | 2 lines
+Changed paths:
+   A /trunk/samtools/bcftools/fet.c
+
+Fisher's exact test
+
+------------------------------------------------------------------------
+r672 | lh3lh3 | 2010-08-05 21:48:33 -0400 (Thu, 05 Aug 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam2bcf.c
+   M /trunk/samtools/bam2bcf.h
+   M /trunk/samtools/bamtk.c
+   M /trunk/samtools/examples/Makefile
+
+ * samtools-0.1.8-11 (r672)
+ * collect more stats for allele balance test in bcftools (not yet)
+
+------------------------------------------------------------------------
+r671 | lh3lh3 | 2010-08-05 16:17:58 -0400 (Thu, 05 Aug 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bcftools/bcf.c
+   M /trunk/samtools/bcftools/main.c
+
+ * the code base is stablized again.
+ * I will delay the vcf parser, which is quite complicated but with little value for now
+
+------------------------------------------------------------------------
+r670 | lh3lh3 | 2010-08-05 16:03:23 -0400 (Thu, 05 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/examples/Makefile
+
+minor
+
+------------------------------------------------------------------------
+r669 | lh3lh3 | 2010-08-05 16:03:08 -0400 (Thu, 05 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcftools/vcf.c
+
+unfinished vcf parser
+
+------------------------------------------------------------------------
+r668 | lh3lh3 | 2010-08-05 15:46:40 -0400 (Thu, 05 Aug 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bcftools/Makefile
+   M /trunk/samtools/bcftools/bcf.c
+   M /trunk/samtools/bcftools/bcf.h
+   M /trunk/samtools/bcftools/bcfutils.c
+   M /trunk/samtools/bcftools/index.c
+   M /trunk/samtools/bcftools/main.c
+   A /trunk/samtools/bcftools/vcf.c
+   M /trunk/samtools/bcftools/vcfout.c
+
+ * added prelimiary VCF parser (not finished)
+ * change struct a bit
+
+------------------------------------------------------------------------
+r667 | lh3lh3 | 2010-08-03 22:35:27 -0400 (Tue, 03 Aug 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam2bcf.c
+   M /trunk/samtools/bam2bcf.h
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bcftools/bcf.c
+
+ * allow to set min base q
+ * fixed a bug in mpileup -u
+
+------------------------------------------------------------------------
+r666 | lh3lh3 | 2010-08-03 22:08:44 -0400 (Tue, 03 Aug 2010) | 2 lines
+Changed paths:
+   A /trunk/samtools/bcftools/bcf.tex
+
+spec
+
+------------------------------------------------------------------------
+r665 | lh3lh3 | 2010-08-03 21:18:57 -0400 (Tue, 03 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/examples/Makefile
+
+added more examples
+
+------------------------------------------------------------------------
+r664 | lh3lh3 | 2010-08-03 21:13:00 -0400 (Tue, 03 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/Makefile
+   M /trunk/samtools/bam2bcf.c
+   M /trunk/samtools/bam2bcf.h
+   M /trunk/samtools/bcftools/Makefile
+
+fixed compilation error
+
+------------------------------------------------------------------------
+r662 | lh3lh3 | 2010-08-03 21:04:00 -0400 (Tue, 03 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/Makefile
+   D /trunk/samtools/bcf.c
+   D /trunk/samtools/bcf.h
+   A /trunk/samtools/bcftools
+   A /trunk/samtools/bcftools/Makefile
+   A /trunk/samtools/bcftools/bcf.c
+   A /trunk/samtools/bcftools/bcf.h
+   A /trunk/samtools/bcftools/bcfutils.c
+   A /trunk/samtools/bcftools/index.c
+   A /trunk/samtools/bcftools/main.c
+   A /trunk/samtools/bcftools/prob1.c
+   A /trunk/samtools/bcftools/prob1.h
+   A /trunk/samtools/bcftools/vcfout.c
+
+move bcftools to samtools
+
+------------------------------------------------------------------------
+r660 | lh3lh3 | 2010-08-03 15:58:32 -0400 (Tue, 03 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam2bcf.c
+
+fixed another minor bug
+
+------------------------------------------------------------------------
+r658 | lh3lh3 | 2010-08-03 15:06:45 -0400 (Tue, 03 Aug 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bamtk.c
+   M /trunk/samtools/bcf.c
+
+ * samtools-0.1.8-10 (r658)
+ * fixed a bug in bam2bcf when the reference is N
+
+------------------------------------------------------------------------
+r657 | lh3lh3 | 2010-08-03 14:50:23 -0400 (Tue, 03 Aug 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam2bcf.c
+   M /trunk/samtools/bam2bcf.h
+
+ * fixed a bug
+ * treat ambiguous ref base as the fifth base
+
+------------------------------------------------------------------------
+r654 | lh3lh3 | 2010-08-02 17:38:27 -0400 (Mon, 02 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/bcftools/bcf.c
+   M /trunk/samtools/bcf.c
+
+missing a column in VCF output...
+
+------------------------------------------------------------------------
+r653 | lh3lh3 | 2010-08-02 17:31:33 -0400 (Mon, 02 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcf.c
+
+fixed a memory leak
+
+------------------------------------------------------------------------
+r651 | lh3lh3 | 2010-08-02 17:27:31 -0400 (Mon, 02 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bcf.c
+
+fixed a bug in bcf reader
+
+------------------------------------------------------------------------
+r650 | lh3lh3 | 2010-08-02 17:00:41 -0400 (Mon, 02 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam2bcf.c
+
+fixed a bug
+
+------------------------------------------------------------------------
+r649 | lh3lh3 | 2010-08-02 16:49:35 -0400 (Mon, 02 Aug 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/Makefile
+   M /trunk/samtools/bam2bcf.c
+   M /trunk/samtools/bam2bcf.h
+   M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-9 (r649)
+ * lossless representation of PL in BCF output
+
+------------------------------------------------------------------------
+r648 | lh3lh3 | 2010-08-02 16:07:25 -0400 (Mon, 02 Aug 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/Makefile
+   A /trunk/samtools/bam2bcf.c
+   A /trunk/samtools/bam2bcf.h
+   M /trunk/samtools/bam_plcmd.c
+   A /trunk/samtools/bcf.c
+   A /trunk/samtools/bcf.h
+
+Generate binary VCF
+
+------------------------------------------------------------------------
+r644 | lh3lh3 | 2010-07-28 11:59:19 -0400 (Wed, 28 Jul 2010) | 5 lines
+Changed paths:
+   M /trunk/samtools/bam_mcns.c
+   M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-8 (r644)
+ * mpileup becomes a little stable again
+ * the method is slightly different, but is more theoretically correct
+ * snp calling is O(n^2) instead of O(n^3)
+
+------------------------------------------------------------------------
+r643 | lh3lh3 | 2010-07-28 11:54:15 -0400 (Wed, 28 Jul 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam_mcns.c
+
+ * fixed a STUPID bug, which cost me a lot of time.
+ * I am going to clean up mcns a little bit
+
+------------------------------------------------------------------------
+r642 | lh3lh3 | 2010-07-27 23:23:07 -0400 (Tue, 27 Jul 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam_mcns.c
+   M /trunk/samtools/bam_mcns.h
+   M /trunk/samtools/bam_plcmd.c
+
+supposedly this is THE correct implementation, but more testing is needed
+
+------------------------------------------------------------------------
+r641 | lh3lh3 | 2010-07-27 22:43:39 -0400 (Tue, 27 Jul 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam_mcns.c
+
+NOT ready yet. Going to make further changes...
+
+------------------------------------------------------------------------
+r639 | lh3lh3 | 2010-07-25 22:18:38 -0400 (Sun, 25 Jul 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam_mcns.c
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-7 (r639)
+ * fixed the reference allele assignment
+
+------------------------------------------------------------------------
+r638 | lh3lh3 | 2010-07-25 12:01:26 -0400 (Sun, 25 Jul 2010) | 5 lines
+Changed paths:
+   M /trunk/samtools/bam_mcns.c
+   M /trunk/samtools/bam_mcns.h
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-6 (r638)
+ * skip isnan/isinf in case of float underflow
+ * added the flat prior
+ * fixed an issue where there are no reads supporting the reference
+
+------------------------------------------------------------------------
+r637 | lh3lh3 | 2010-07-24 14:16:27 -0400 (Sat, 24 Jul 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam_plcmd.c
+
+minor changes
+
+------------------------------------------------------------------------
+r636 | lh3lh3 | 2010-07-24 14:07:27 -0400 (Sat, 24 Jul 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam_mcns.c
+   M /trunk/samtools/bam_mcns.h
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bamtk.c
+
+minor tweaks
+
+------------------------------------------------------------------------
+r635 | lh3lh3 | 2010-07-24 01:49:49 -0400 (Sat, 24 Jul 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam_mcns.c
+   M /trunk/samtools/bam_mcns.h
+   M /trunk/samtools/bam_plcmd.c
+
+posterior expectation FINALLY working. I am so tired...
+
+------------------------------------------------------------------------
+r633 | lh3lh3 | 2010-07-23 13:50:48 -0400 (Fri, 23 Jul 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam_plcmd.c
+
+another minor fix to mpileup
+
+------------------------------------------------------------------------
+r632 | lh3lh3 | 2010-07-23 13:43:31 -0400 (Fri, 23 Jul 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam_plcmd.c
+
+added the format column
+
+------------------------------------------------------------------------
+r631 | lh3lh3 | 2010-07-23 13:25:44 -0400 (Fri, 23 Jul 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam_mcns.c
+   M /trunk/samtools/bam_mcns.h
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bamtk.c
+
+added an alternative prior
+
+------------------------------------------------------------------------
+r628 | lh3lh3 | 2010-07-23 11:48:51 -0400 (Fri, 23 Jul 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/bam_mcns.c
+   M /trunk/samtools/bam_mcns.h
+   M /trunk/samtools/bam_plcmd.c
+
+calculate posterior allele frequency
+
+------------------------------------------------------------------------
+r627 | lh3lh3 | 2010-07-22 21:39:13 -0400 (Thu, 22 Jul 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam_mcns.c
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-3 (r627)
+ * multi-sample snp calling appears to work. More tests needed.
+
+------------------------------------------------------------------------
+r626 | lh3lh3 | 2010-07-22 16:37:56 -0400 (Thu, 22 Jul 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam_mcns.c
+   M /trunk/samtools/bam_mcns.h
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bam_tview.c
+
+ * preliminary multisample SNP caller.
+ * something looks not so right, but it largely works
+
+------------------------------------------------------------------------
+r617 | lh3lh3 | 2010-07-14 16:26:27 -0400 (Wed, 14 Jul 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam_mcns.c
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-2 (r617)
+ * allele frequency calculation apparently works...
+
+------------------------------------------------------------------------
+r616 | lh3lh3 | 2010-07-14 13:33:51 -0400 (Wed, 14 Jul 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/Makefile
+   A /trunk/samtools/bam_mcns.c
+   A /trunk/samtools/bam_mcns.h
+   M /trunk/samtools/bam_plcmd.c
+
+ * added mutli-sample framework. It is not working, yet.
+ * improved the mpileup interface
+
+------------------------------------------------------------------------
+r615 | lh3lh3 | 2010-07-13 14:50:12 -0400 (Tue, 13 Jul 2010) | 3 lines
+Changed paths:
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bamtk.c
+   M /trunk/samtools/misc/Makefile
+
+ * samtools-0.1.8-1 (r615)
+ * allow to get mpileup at required sites
+
+------------------------------------------------------------------------
+r613 | lh3lh3 | 2010-07-11 22:40:56 -0400 (Sun, 11 Jul 2010) | 2 lines
+Changed paths:
+   M /trunk/samtools/ChangeLog
+   M /trunk/samtools/NEWS
+   M /trunk/samtools/bam_plcmd.c
+   M /trunk/samtools/bamtk.c
+   M /trunk/samtools/samtools.1
+
+Release samtools-0.1.8
+

 ------------------------------------------------------------------------
 r612 | lh3lh3 | 2010-07-11 21:08:56 -0400 (Sun, 11 Jul 2010) | 2 lines
 Changed paths:
 ------------------------------------------------------------------------
 r612 | lh3lh3 | 2010-07-11 21:08:56 -0400 (Sun, 11 Jul 2010) | 2 lines
 Changed paths:

diff --git a/Makefile b/Makefile
index 35d578ffe50ded7ff4eb81caa084dbb245b17e5a..77cda865c4b38df76fdafe338feae873dabde0e6 100644 (file)
--- a/Makefile
+++ b/Makefile
@@ -7,10 +7,10 @@ LOBJS=                bgzf.o kstring.o bam_aux.o bam.o bam_import.o sam.o bam_index.o \
                        $(KNETFILE_O) bam_sort.o sam_header.o bam_reheader.o
 AOBJS=         bam_tview.o bam_maqcns.o bam_plcmd.o sam_view.o \
                        bam_rmdup.o bam_rmdupse.o bam_mate.o bam_stat.o bam_color.o     \
                        $(KNETFILE_O) bam_sort.o sam_header.o bam_reheader.o
 AOBJS=         bam_tview.o bam_maqcns.o bam_plcmd.o sam_view.o \
                        bam_rmdup.o bam_rmdupse.o bam_mate.o bam_stat.o bam_color.o     \

-                       bamtk.o kaln.o
+                       bamtk.o kaln.o bam2bcf.o errmod.o sample.o

 PROG=          samtools
 PROG=          samtools

-INCLUDES=
-SUBDIRS=       . misc
+INCLUDES=      -I.
+SUBDIRS=       . bcftools misc

 LIBPATH=
 LIBCURSES=     -lcurses # -lXCurses
 
 LIBPATH=
 LIBCURSES=     -lcurses # -lXCurses
 


@@ -39,8 +39,8 @@ lib:libbam.a
 libbam.a:$(LOBJS)
                $(AR) -cru $@ $(LOBJS)
 
 libbam.a:$(LOBJS)
                $(AR) -cru $@ $(LOBJS)
 

-samtools:$(AOBJS) libbam.a
-               $(CC) $(CFLAGS) -o $@ $(AOBJS) libbam.a -lm $(LIBPATH) $(LIBCURSES) -lz
+samtools:lib-recur $(AOBJS)
+               $(CC) $(CFLAGS) -o $@ $(AOBJS) libbam.a -lm $(LIBPATH) $(LIBCURSES) -lz -Lbcftools -lbcf

 
 razip:razip.o razf.o $(KNETFILE_O)
                $(CC) $(CFLAGS) -o $@ razf.o razip.o $(KNETFILE_O) -lz
 
 razip:razip.o razf.o $(KNETFILE_O)
                $(CC) $(CFLAGS) -o $@ razf.o razip.o $(KNETFILE_O) -lz


@@ -53,15 +53,18 @@ bam.o:bam.h razf.h bam_endian.h kstring.h sam_header.h
 sam.o:sam.h bam.h
 bam_import.o:bam.h kseq.h khash.h razf.h
 bam_pileup.o:bam.h razf.h ksort.h
 sam.o:sam.h bam.h
 bam_import.o:bam.h kseq.h khash.h razf.h
 bam_pileup.o:bam.h razf.h ksort.h

-bam_plcmd.o:bam.h faidx.h bam_maqcns.h glf.h
+bam_plcmd.o:bam.h faidx.h bam_maqcns.h glf.h bcftools/bcf.h bam2bcf.h

 bam_index.o:bam.h khash.h ksort.h razf.h bam_endian.h
 bam_lpileup.o:bam.h ksort.h
 bam_tview.o:bam.h faidx.h bam_maqcns.h
 bam_index.o:bam.h khash.h ksort.h razf.h bam_endian.h
 bam_lpileup.o:bam.h ksort.h
 bam_tview.o:bam.h faidx.h bam_maqcns.h

-bam_maqcns.o:bam.h ksort.h bam_maqcns.h
+bam_maqcns.o:bam.h ksort.h bam_maqcns.h kaln.h

 bam_sort.o:bam.h ksort.h razf.h
 bam_md.o:bam.h faidx.h
 glf.o:glf.h
 sam_header.o:sam_header.h khash.h
 bam_sort.o:bam.h ksort.h razf.h
 bam_md.o:bam.h faidx.h
 glf.o:glf.h
 sam_header.o:sam_header.h khash.h

+bcf.o:bcftools/bcf.h
+bam2bcf.o:bam2bcf.h errmod.h bcftools/bcf.h
+errmod.o:errmod.h

 
 faidx.o:faidx.h razf.h khash.h
 faidx_main.o:faidx.h razf.h
 
 faidx.o:faidx.h razf.h khash.h
 faidx_main.o:faidx.h razf.h

diff --git a/NEWS b/NEWS
index 28d6aaa81269b72f0b8cb26a971c5bd55e33c050..82646ba812fa66f90c1ca4cf2473645f87809b31 100644 (file)
--- a/NEWS
+++ b/NEWS
@@ -1,3 +1,52 @@
+Beta Release 0.1.9 (27 October, 2010)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This release is featured as the first major improvement to the samtools'
+SNP caller.  It comes with a revised MAQ error model, the support of
+multi-sample SNP calling and the computation of base alignment quality
+(BAQ).
+
+The revised MAQ error model is based on the original model. It solves an
+issue of miscalling SNPs in repetitive regions. Althought such SNPs can
+usually be filtered at a later step, they mess up unfiltered calls. This
+is a theoretical flaw in the original model. The revised MAQ model
+deprecates the orginal MAQ model and the simplified SOAPsnp model.
+
+Multi-sample SNP calling is separated in two steps. The first is done by
+samtools mpileup and the second by a new program, bcftools, which is
+included in the samtools source code tree. Multi-sample SNP calling also
+works for single sample and has the advantage of enabling more powerful
+filtration. It is likely to deprecate pileup in future once a proper
+indel calling method is implemented.
+
+BAQ is the Phred-scaled probability of a read base being wrongly
+aligned. Capping base quality by BAQ has been shown to be very effective
+in suppressing false SNPs caused by misalignments around indels or in
+low-complexity regions with acceptable compromise on computation
+time. This strategy is highly recommended and can be used with other SNP
+callers as well.
+
+In addition to the three major improvements, other notable changes are:
+
+ * Changes to the pileup format. A reference skip (the N CIGAR operator)
+   is shown as '<' or '>' depending on the strand. Tview is also changed
+   accordingly.
+
+ * Accelerated pileup. The plain pileup is about 50% faster.
+
+ * Regional merge. The merge command now accepts a new option to merge
+   files in a specified region.
+
+ * Fixed a bug in bgzip and razip which causes source files to be
+   deleted even if option -c is applied.
+
+ * In APIs, propogate errors to downstream callers and make samtools
+   return non-zero values once errors occur.
+
+(0.1.9: 27 October 2010, r783)
+
+
+

 Beta Release 0.1.8 (11 July, 2010)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Beta Release 0.1.8 (11 July, 2010)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

diff --git a/bam.c b/bam.c
index 94b0aa8bb85bef4ae4a787af3337f1765ed8899e..521c1dda7897aa2b4489b103e1ee1a84129d6c63 100644 (file)
--- a/bam.c
+++ b/bam.c
@@ -245,7 +245,11 @@ char *bam_format1_core(const bam_header_t *header, const bam1_t *b, int of)
                kputc('\t', &str);
        }
        if (c->tid < 0) kputsn("*\t", 2, &str);
                kputc('\t', &str);
        }
        if (c->tid < 0) kputsn("*\t", 2, &str);

-       else { kputs(header->target_name[c->tid], &str); kputc('\t', &str); }
+       else {
+               if (header) kputs(header->target_name[c->tid] , &str);
+               else kputw(c->tid, &str);
+               kputc('\t', &str);
+       }

        kputw(c->pos + 1, &str); kputc('\t', &str); kputw(c->qual, &str); kputc('\t', &str);
        if (c->n_cigar == 0) kputc('*', &str);
        else {
        kputw(c->pos + 1, &str); kputc('\t', &str); kputw(c->qual, &str); kputc('\t', &str);
        if (c->n_cigar == 0) kputc('*', &str);
        else {


@@ -257,7 +261,11 @@ char *bam_format1_core(const bam_header_t *header, const bam1_t *b, int of)
        kputc('\t', &str);
        if (c->mtid < 0) kputsn("*\t", 2, &str);
        else if (c->mtid == c->tid) kputsn("=\t", 2, &str);
        kputc('\t', &str);
        if (c->mtid < 0) kputsn("*\t", 2, &str);
        else if (c->mtid == c->tid) kputsn("=\t", 2, &str);

-       else { kputs(header->target_name[c->mtid], &str); kputc('\t', &str); }
+       else {
+               if (header) kputs(header->target_name[c->mtid], &str);
+               else kputw(c->mtid, &str);
+               kputc('\t', &str);
+       }

        kputw(c->mpos + 1, &str); kputc('\t', &str); kputw(c->isize, &str); kputc('\t', &str);
        if (c->l_qseq) {
                for (i = 0; i < c->l_qseq; ++i) kputc(bam_nt16_rev_table[bam1_seqi(s, i)], &str);
        kputw(c->mpos + 1, &str); kputc('\t', &str); kputw(c->isize, &str); kputc('\t', &str);
        if (c->l_qseq) {
                for (i = 0; i < c->l_qseq; ++i) kputc(bam_nt16_rev_table[bam1_seqi(s, i)], &str);


@@ -297,6 +305,22 @@ void bam_view1(const bam_header_t *header, const bam1_t *b)
        free(s);
 }
 
        free(s);
 }
 

+int bam_validate1(const bam_header_t *header, const bam1_t *b)
+{
+       char *s;
+
+       if (b->core.tid < -1 || b->core.mtid < -1) return 0;
+       if (header && (b->core.tid >= header->n_targets || b->core.mtid >= header->n_targets)) return 0;
+
+       if (b->data_len < b->core.l_qname) return 0;
+       s = memchr(bam1_qname(b), '\0', b->core.l_qname);
+       if (s != &bam1_qname(b)[b->core.l_qname-1]) return 0;
+
+       // FIXME: Other fields could also be checked, especially the auxiliary data
+
+       return 1;
+}
+

 // FIXME: we should also check the LB tag associated with each alignment
 const char *bam_get_library(bam_header_t *h, const bam1_t *b)
 {
 // FIXME: we should also check the LB tag associated with each alignment
 const char *bam_get_library(bam_header_t *h, const bam1_t *b)
 {

diff --git a/bam.h b/bam.h
index 8e26ea69d3b7c6b008aba6112052310a4072b08a..4c8536fd33366883028ff9b77bb95049682c5f9f 100644 (file)
--- a/bam.h
+++ b/bam.h
@@ -1,6 +1,6 @@
 /* The MIT License
 
 /* The MIT License
 

-   Copyright (c) 2008 Genome Research Ltd (GRL).
+   Copyright (c) 2008-2010 Genome Research Ltd (GRL).

 
    Permission is hereby granted, free of charge, to any person obtaining
    a copy of this software and associated documentation files (the
 
    Permission is hereby granted, free of charge, to any person obtaining
    a copy of this software and associated documentation files (the


@@ -230,7 +230,7 @@ typedef struct __bam_iter_t *bam_iter_t;
   @param  b  pointer to an alignment
   @return    pointer to quality string
  */
   @param  b  pointer to an alignment
   @return    pointer to quality string
  */

-#define bam1_qual(b) ((b)->data + (b)->core.n_cigar*4 + (b)->core.l_qname + ((b)->core.l_qseq + 1)/2)
+#define bam1_qual(b) ((b)->data + (b)->core.n_cigar*4 + (b)->core.l_qname + (((b)->core.l_qseq + 1)>>1))

 
 /*! @function
   @abstract  Get a base on read
 
 /*! @function
   @abstract  Get a base on read


@@ -455,6 +455,21 @@ extern "C" {
 
        char *bam_format1_core(const bam_header_t *header, const bam1_t *b, int of);
 
 
        char *bam_format1_core(const bam_header_t *header, const bam1_t *b, int of);
 

+       /*!
+         @abstract       Check whether a BAM record is plausibly valid
+         @param  header  associated header structure, or NULL if unavailable
+         @param  b       alignment to validate
+         @return         0 if the alignment is invalid; non-zero otherwise
+
+         @discussion  Simple consistency check of some of the fields of the
+         alignment record.  If the header is provided, several additional checks
+         are made.  Not all fields are checked, so a non-zero result is not a
+         guarantee that the record is valid.  However it is usually good enough
+         to detect when bam_seek() has been called with a virtual file offset
+         that is not the offset of an alignment record.
+        */
+       int bam_validate1(const bam_header_t *header, const bam1_t *b);
+

        const char *bam_get_library(bam_header_t *header, const bam1_t *b);
 
 
        const char *bam_get_library(bam_header_t *header, const bam1_t *b);
 
 


@@ -480,7 +495,7 @@ extern "C" {
                bam1_t *b;
                int32_t qpos;
                int indel, level;
                bam1_t *b;
                int32_t qpos;
                int indel, level;

-               uint32_t is_del:1, is_head:1, is_tail:1;
+               uint32_t is_del:1, is_head:1, is_tail:1, is_refskip:1;

        } bam_pileup1_t;
 
        typedef int (*bam_plp_auto_f)(void *data, bam1_t *b);
        } bam_pileup1_t;
 
        typedef int (*bam_plp_auto_f)(void *data, bam1_t *b);

diff --git a/bam2bcf.c b/bam2bcf.c
new file mode 100644 (file)
index 0000000..e55212c
--- /dev/null
+++ b/bam2bcf.c
@@ -0,0 +1,186 @@
+#include <math.h>
+#include <stdint.h>
+#include "bam.h"
+#include "kstring.h"
+#include "bam2bcf.h"
+#include "errmod.h"
+#include "bcftools/bcf.h"
+
+extern void ks_introsort_uint32_t(size_t n, uint32_t a[]);
+
+#define CALL_ETA 0.03f
+#define CALL_MAX 256
+#define CALL_DEFTHETA 0.83f
+
+#define CAP_DIST 25
+
+struct __bcf_callaux_t {
+       int max_bases, capQ, min_baseQ;
+       uint16_t *bases;
+       errmod_t *e;
+};
+
+bcf_callaux_t *bcf_call_init(double theta, int min_baseQ)
+{
+       bcf_callaux_t *bca;
+       if (theta <= 0.) theta = CALL_DEFTHETA;
+       bca = calloc(1, sizeof(bcf_callaux_t));
+       bca->capQ = 60;
+       bca->min_baseQ = min_baseQ;
+       bca->e = errmod_init(1. - theta);
+       return bca;
+}
+
+void bcf_call_destroy(bcf_callaux_t *bca)
+{
+       if (bca == 0) return;
+       errmod_destroy(bca->e);
+       free(bca->bases); free(bca);
+}
+
+int bcf_call_glfgen(int _n, const bam_pileup1_t *pl, int ref_base /*4-bit*/, bcf_callaux_t *bca, bcf_callret1_t *r)
+{
+       int i, n, ref4;
+       memset(r, 0, sizeof(bcf_callret1_t));
+       ref4 = bam_nt16_nt4_table[ref_base];
+       if (_n == 0) return -1;
+
+       // enlarge the bases array if necessary
+       if (bca->max_bases < _n) {
+               bca->max_bases = _n;
+               kroundup32(bca->max_bases);
+               bca->bases = (uint16_t*)realloc(bca->bases, 2 * bca->max_bases);
+       }
+       // fill the bases array
+       memset(r, 0, sizeof(bcf_callret1_t));
+       for (i = n = 0; i < _n; ++i) {
+               const bam_pileup1_t *p = pl + i;
+               int q, b, mapQ, baseQ, is_diff, min_dist;
+               // set base
+               if (p->is_del || (p->b->core.flag&BAM_FUNMAP)) continue; // skip unmapped reads and deleted bases
+               baseQ = q = (int)bam1_qual(p->b)[p->qpos]; // base quality
+               if (q < bca->min_baseQ) continue;
+               mapQ = p->b->core.qual < bca->capQ? p->b->core.qual : bca->capQ;
+               if (q > mapQ) q = mapQ;
+               if (q > 63) q = 63;
+               if (q < 4) q = 4;
+               b = bam1_seqi(bam1_seq(p->b), p->qpos); // base
+               b = bam_nt16_nt4_table[b? b : ref_base]; // b is the 2-bit base
+               bca->bases[n++] = q<<5 | (int)bam1_strand(p->b)<<4 | b;
+               // collect annotations
+               r->qsum[b] += q;
+               is_diff = (ref4 < 4 && b == ref4)? 0 : 1;
+               ++r->anno[0<<2|is_diff<<1|bam1_strand(p->b)];
+               min_dist = p->b->core.l_qseq - 1 - p->qpos;
+               if (min_dist > p->qpos) min_dist = p->qpos;
+               if (min_dist > CAP_DIST) min_dist = CAP_DIST;
+               r->anno[1<<2|is_diff<<1|0] += baseQ;
+               r->anno[1<<2|is_diff<<1|1] += baseQ * baseQ;
+               r->anno[2<<2|is_diff<<1|0] += mapQ;
+               r->anno[2<<2|is_diff<<1|1] += mapQ * mapQ;
+               r->anno[3<<2|is_diff<<1|0] += min_dist;
+               r->anno[3<<2|is_diff<<1|1] += min_dist * min_dist;
+       }
+       r->depth = n;
+       // glfgen
+       errmod_cal(bca->e, n, 5, bca->bases, r->p);
+       return r->depth;
+}
+
+int bcf_call_combine(int n, const bcf_callret1_t *calls, int ref_base /*4-bit*/, bcf_call_t *call)
+{
+       int ref4, i, j, qsum[4];
+       int64_t tmp;
+       call->ori_ref = ref4 = bam_nt16_nt4_table[ref_base];
+       if (ref4 > 4) ref4 = 4;
+       // calculate qsum
+       memset(qsum, 0, 4 * sizeof(int));
+       for (i = 0; i < n; ++i)
+               for (j = 0; j < 4; ++j)
+                       qsum[j] += calls[i].qsum[j];
+       for (j = 0; j < 4; ++j) qsum[j] = qsum[j] << 2 | j;
+       // find the top 2 alleles
+       for (i = 1; i < 4; ++i) // insertion sort
+               for (j = i; j > 0 && qsum[j] < qsum[j-1]; --j)
+                       tmp = qsum[j], qsum[j] = qsum[j-1], qsum[j-1] = tmp;
+       // set the reference allele and alternative allele(s)
+       for (i = 0; i < 5; ++i) call->a[i] = -1;
+       call->unseen = -1;
+       call->a[0] = ref4;
+       for (i = 3, j = 1; i >= 0; --i) {
+               if ((qsum[i]&3) != ref4) {
+                       if (qsum[i]>>2 != 0) call->a[j++] = qsum[i]&3;
+                       else break;
+               }
+       }
+       if (((ref4 < 4 && j < 4) || (ref4 == 4 && j < 5)) && i >= 0)
+               call->unseen = j, call->a[j++] = qsum[i]&3;
+       call->n_alleles = j;
+       // set the PL array
+       if (call->n < n) {
+               call->n = n;
+               call->PL = realloc(call->PL, 15 * n);
+       }
+       {
+               int x, g[15], z;
+               double sum_min = 0.;
+               x = call->n_alleles * (call->n_alleles + 1) / 2;
+               // get the possible genotypes
+               for (i = z = 0; i < call->n_alleles; ++i)
+                       for (j = i; j < call->n_alleles; ++j)
+                               g[z++] = call->a[i] * 5 + call->a[j];
+               for (i = 0; i < n; ++i) {
+                       uint8_t *PL = call->PL + x * i;
+                       const bcf_callret1_t *r = calls + i;
+                       float min = 1e37;
+                       for (j = 0; j < x; ++j)
+                               if (min > r->p[g[j]]) min = r->p[g[j]];
+                       sum_min += min;
+                       for (j = 0; j < x; ++j) {
+                               int y;
+                               y = (int)(r->p[g[j]] - min + .499);
+                               if (y > 255) y = 255;
+                               PL[j] = y;
+                       }
+               }
+               call->shift = (int)(sum_min + .499);
+       }
+       // combine annotations
+       memset(call->anno, 0, 16 * sizeof(int));
+       for (i = call->depth = 0, tmp = 0; i < n; ++i) {
+               call->depth += calls[i].depth;
+               for (j = 0; j < 16; ++j) call->anno[j] += calls[i].anno[j];
+       }
+       return 0;
+}
+
+int bcf_call2bcf(int tid, int pos, bcf_call_t *bc, bcf1_t *b)
+{
+       kstring_t s;
+       int i;
+       b->n_smpl = bc->n;
+       b->tid = tid; b->pos = pos; b->qual = 0;
+       s.s = b->str; s.m = b->m_str; s.l = 0;
+       kputc('\0', &s);
+       kputc("ACGTN"[bc->ori_ref], &s); kputc('\0', &s);
+       for (i = 1; i < 5; ++i) {
+               if (bc->a[i] < 0) break;
+               if (i > 1) kputc(',', &s);
+               kputc(bc->unseen == i? 'X' : "ACGT"[bc->a[i]], &s);
+       }
+       kputc('\0', &s);
+       kputc('\0', &s);
+       // INFO
+       kputs("I16=", &s);
+       for (i = 0; i < 16; ++i) {
+               if (i) kputc(',', &s);
+               kputw(bc->anno[i], &s);
+       }
+       kputc('\0', &s);
+       // FMT
+       kputs("PL", &s); kputc('\0', &s);
+       b->m_str = s.m; b->str = s.s; b->l_str = s.l;
+       bcf_sync(b);
+       memcpy(b->gi[0].data, bc->PL, b->gi[0].len * bc->n);
+       return 0;
+}

diff --git a/bam2bcf.h b/bam2bcf.h
new file mode 100644 (file)
index 0000000..0dddb92
--- /dev/null
+++ b/bam2bcf.h
@@ -0,0 +1,37 @@
+#ifndef BAM2BCF_H
+#define BAM2BCF_H
+
+#include <stdint.h>
+#include "bcftools/bcf.h"
+
+struct __bcf_callaux_t;
+typedef struct __bcf_callaux_t bcf_callaux_t;
+
+typedef struct {
+       int depth, qsum[4];
+       int anno[16];
+       float p[25];
+} bcf_callret1_t;
+
+typedef struct {
+       int a[5]; // alleles: ref, alt, alt2, alt3
+       int n, n_alleles, shift, ori_ref, unseen;
+       int anno[16], depth;
+       uint8_t *PL;
+} bcf_call_t;
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+       bcf_callaux_t *bcf_call_init(double theta, int min_baseQ);
+       void bcf_call_destroy(bcf_callaux_t *bca);
+       int bcf_call_glfgen(int _n, const bam_pileup1_t *pl, int ref_base /*4-bit*/, bcf_callaux_t *bca, bcf_callret1_t *r);
+       int bcf_call_combine(int n, const bcf_callret1_t *calls, int ref_base /*4-bit*/, bcf_call_t *call);
+       int bcf_call2bcf(int tid, int pos, bcf_call_t *bc, bcf1_t *b);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif

diff --git a/bam_index.c b/bam_index.c
index 4152f2022f3178ef4fd0ee6b6001effae8f2ce3c..1ad2e93e5051972b084513d5d7391c45c6cf7bf2 100644 (file)
--- a/bam_index.c
+++ b/bam_index.c
@@ -656,17 +656,17 @@ void bam_iter_destroy(bam_iter_t iter)
 
 int bam_iter_read(bamFile fp, bam_iter_t iter, bam1_t *b)
 {
 
 int bam_iter_read(bamFile fp, bam_iter_t iter, bam1_t *b)
 {

-       if (iter->finished) return -1;
-       if (iter->from_first) {
-               int ret = bam_read1(fp, b);
-               if (ret < 0) iter->finished = 1;
+       int ret;
+       if (iter && iter->finished) return -1;
+       if (iter == 0 || iter->from_first) {
+               ret = bam_read1(fp, b);
+               if (ret < 0 && iter) iter->finished = 1;

                return ret;
        }
        if (iter->off == 0) return -1;
        for (;;) {
                return ret;
        }
        if (iter->off == 0) return -1;
        for (;;) {

-               int ret;

                if (iter->curr_off == 0 || iter->curr_off >= iter->off[iter->i].v) { // then jump to the next chunk
                if (iter->curr_off == 0 || iter->curr_off >= iter->off[iter->i].v) { // then jump to the next chunk

-                       if (iter->i == iter->n_off - 1) break; // no more chunks
+                       if (iter->i == iter->n_off - 1) { ret = -1; break; } // no more chunks

                        if (iter->i >= 0) assert(iter->curr_off == iter->off[iter->i].v); // otherwise bug
                        if (iter->i < 0 || iter->off[iter->i].v != iter->off[iter->i+1].u) { // not adjacent chunks; then seek
                                bam_seek(fp, iter->off[iter->i+1].u, SEEK_SET);
                        if (iter->i >= 0) assert(iter->curr_off == iter->off[iter->i].v); // otherwise bug
                        if (iter->i < 0 || iter->off[iter->i].v != iter->off[iter->i+1].u) { // not adjacent chunks; then seek
                                bam_seek(fp, iter->off[iter->i+1].u, SEEK_SET);


@@ -674,23 +674,27 @@ int bam_iter_read(bamFile fp, bam_iter_t iter, bam1_t *b)
                        }
                        ++iter->i;
                }
                        }
                        ++iter->i;
                }

-               if ((ret = bam_read1(fp, b)) > 0) {
+               if ((ret = bam_read1(fp, b)) >= 0) {

                        iter->curr_off = bam_tell(fp);
                        iter->curr_off = bam_tell(fp);

-                       if (b->core.tid != iter->tid || b->core.pos >= iter->end) break; // no need to proceed
+                       if (b->core.tid != iter->tid || b->core.pos >= iter->end) { // no need to proceed
+                               ret = bam_validate1(NULL, b)? -1 : -5; // determine whether end of region or error
+                               break;
+                       }

                        else if (is_overlap(iter->beg, iter->end, b)) return ret;
                        else if (is_overlap(iter->beg, iter->end, b)) return ret;

-               } else break; // end of file
+               } else break; // end of file or error

        }
        iter->finished = 1;
        }
        iter->finished = 1;

-       return -1;
+       return ret;

 }
 
 int bam_fetch(bamFile fp, const bam_index_t *idx, int tid, int beg, int end, void *data, bam_fetch_f func)
 {
 }
 
 int bam_fetch(bamFile fp, const bam_index_t *idx, int tid, int beg, int end, void *data, bam_fetch_f func)
 {

+       int ret;

        bam_iter_t iter;
        bam1_t *b;
        b = bam_init1();
        iter = bam_iter_query(idx, tid, beg, end);
        bam_iter_t iter;
        bam1_t *b;
        b = bam_init1();
        iter = bam_iter_query(idx, tid, beg, end);

-       while (bam_iter_read(fp, iter, b) >= 0) func(b, data);
+       while ((ret = bam_iter_read(fp, iter, b)) >= 0) func(b, data);

        bam_destroy1(b);
        bam_destroy1(b);

-       return 0;
+       return (ret == -1)? 0 : ret;

 }
 }

diff --git a/bam_maqcns.c b/bam_maqcns.c
index cad63d772268db289467c6a4fd625e50b7ac3889..2f3fc082be79aa0fa3feae9c1d9a03ad44592877 100644 (file)
--- a/bam_maqcns.c
+++ b/bam_maqcns.c
@@ -3,6 +3,7 @@
 #include "bam.h"
 #include "bam_maqcns.h"
 #include "ksort.h"
 #include "bam.h"
 #include "bam_maqcns.h"
 #include "ksort.h"

+#include "errmod.h"

 #include "kaln.h"
 KSORT_INIT_GENERIC(uint32_t)
 
 #include "kaln.h"
 KSORT_INIT_GENERIC(uint32_t)
 


@@ -12,12 +13,13 @@ KSORT_INIT_GENERIC(uint32_t)
 typedef struct __bmc_aux_t {
        int max;
        uint32_t *info;
 typedef struct __bmc_aux_t {
        int max;
        uint32_t *info;

+       uint16_t *info16;
+       errmod_t *em;

 } bmc_aux_t;
 
 typedef struct {
        float esum[4], fsum[4];
        uint32_t c[4];
 } bmc_aux_t;
 
 typedef struct {
        float esum[4], fsum[4];
        uint32_t c[4];

-       uint32_t rms_mapQ;

 } glf_call_aux_t;
 
 char bam_nt16_nt4_table[] = { 4, 0, 1, 4, 2, 4, 4, 4, 3, 4, 4, 4, 4, 4, 4, 4 };
 } glf_call_aux_t;
 
 char bam_nt16_nt4_table[] = { 4, 0, 1, 4, 2, 4, 4, 4, 3, 4, 4, 4, 4, 4, 4, 4 };


@@ -41,7 +43,7 @@ static void cal_het(bam_maqcns_t *aa)
        for (n1 = 0; n1 < 256; ++n1) {
                for (n2 = 0; n2 < 256; ++n2) {
                        long double sum = 0.0;
        for (n1 = 0; n1 < 256; ++n1) {
                for (n2 = 0; n2 < 256; ++n2) {
                        long double sum = 0.0;

-                       double lC = aa->is_soap? 0 : lgamma(n1+n2+1) - lgamma(n1+1) - lgamma(n2+1); // \binom{n1+n2}{n1}
+                       double lC = aa->errmod == BAM_ERRMOD_SOAP? 0 : lgamma(n1+n2+1) - lgamma(n1+1) - lgamma(n2+1);

                        for (k = 1; k <= aa->n_hap - 1; ++k) {
                                double pk = 1.0 / k / sum_harmo;
                                double log1 = log((double)k/aa->n_hap);
                        for (k = 1; k <= aa->n_hap - 1; ++k) {
                                double pk = 1.0 / k / sum_harmo;
                                double log1 = log((double)k/aa->n_hap);


@@ -62,6 +64,7 @@ static void cal_coef(bam_maqcns_t *aa)
        long double sum_a[257], b[256], q_c[256], tmp[256], fk2[256];
        double *lC;
 
        long double sum_a[257], b[256], q_c[256], tmp[256], fk2[256];
        double *lC;
 

+       if (aa->errmod == BAM_ERRMOD_MAQ2) return; // no need to do the following

        // aa->lhet will be allocated and initialized 
        free(aa->fk); free(aa->coef);
        aa->coef = 0;
        // aa->lhet will be allocated and initialized 
        free(aa->fk); free(aa->coef);
        aa->coef = 0;


@@ -71,7 +74,7 @@ static void cal_coef(bam_maqcns_t *aa)
                aa->fk[n] = pow(aa->theta, n) * (1.0 - aa->eta) + aa->eta;
                fk2[n] = aa->fk[n>>1]; // this is an approximation, assuming reads equally likely come from both strands
        }
                aa->fk[n] = pow(aa->theta, n) * (1.0 - aa->eta) + aa->eta;
                fk2[n] = aa->fk[n>>1]; // this is an approximation, assuming reads equally likely come from both strands
        }

-       if (aa->is_soap) return;
+       if (aa->errmod == BAM_ERRMOD_SOAP) return;

        aa->coef = (double*)calloc(256*256*64, sizeof(double));
        lC = (double*)calloc(256 * 256, sizeof(double));
        for (n = 1; n != 256; ++n)
        aa->coef = (double*)calloc(256*256*64, sizeof(double));
        lC = (double*)calloc(256 * 256, sizeof(double));
        for (n = 1; n != 256; ++n)


@@ -107,28 +110,31 @@ bam_maqcns_t *bam_maqcns_init()
        bm = (bam_maqcns_t*)calloc(1, sizeof(bam_maqcns_t));
        bm->aux = (bmc_aux_t*)calloc(1, sizeof(bmc_aux_t));
        bm->het_rate = 0.001;
        bm = (bam_maqcns_t*)calloc(1, sizeof(bam_maqcns_t));
        bm->aux = (bmc_aux_t*)calloc(1, sizeof(bmc_aux_t));
        bm->het_rate = 0.001;

-       bm->theta = 0.85;
+       bm->theta = 0.83f;

        bm->n_hap = 2;
        bm->eta = 0.03;
        bm->cap_mapQ = 60;
        bm->n_hap = 2;
        bm->eta = 0.03;
        bm->cap_mapQ = 60;

+       bm->min_baseQ = 13;

        return bm;
 }
 
 void bam_maqcns_prepare(bam_maqcns_t *bm)
 {
        return bm;
 }
 
 void bam_maqcns_prepare(bam_maqcns_t *bm)
 {

+       if (bm->errmod == BAM_ERRMOD_MAQ2) bm->aux->em = errmod_init(1. - bm->theta);

        cal_coef(bm); cal_het(bm);
 }
 
 void bam_maqcns_destroy(bam_maqcns_t *bm)
 {
        if (bm == 0) return;
        cal_coef(bm); cal_het(bm);
 }
 
 void bam_maqcns_destroy(bam_maqcns_t *bm)
 {
        if (bm == 0) return;

-       free(bm->lhet); free(bm->fk); free(bm->coef); free(bm->aux->info);
+       free(bm->lhet); free(bm->fk); free(bm->coef); free(bm->aux->info); free(bm->aux->info16);
+       if (bm->aux->em) errmod_destroy(bm->aux->em);

        free(bm->aux); free(bm);
 }
 
 glf1_t *bam_maqcns_glfgen(int _n, const bam_pileup1_t *pl, uint8_t ref_base, bam_maqcns_t *bm)
 {
        free(bm->aux); free(bm);
 }
 
 glf1_t *bam_maqcns_glfgen(int _n, const bam_pileup1_t *pl, uint8_t ref_base, bam_maqcns_t *bm)
 {

-       glf_call_aux_t *b;
+       glf_call_aux_t *b = 0;

        int i, j, k, w[8], c, n;
        glf1_t *g = (glf1_t*)calloc(1, sizeof(glf1_t));
        float p[16], min_p = 1e30;
        int i, j, k, w[8], c, n;
        glf1_t *g = (glf1_t*)calloc(1, sizeof(glf1_t));
        float p[16], min_p = 1e30;


@@ -142,28 +148,39 @@ glf1_t *bam_maqcns_glfgen(int _n, const bam_pileup1_t *pl, uint8_t ref_base, bam
                bm->aux->max = _n;
                kroundup32(bm->aux->max);
                bm->aux->info = (uint32_t*)realloc(bm->aux->info, 4 * bm->aux->max);
                bm->aux->max = _n;
                kroundup32(bm->aux->max);
                bm->aux->info = (uint32_t*)realloc(bm->aux->info, 4 * bm->aux->max);

+               bm->aux->info16 = (uint16_t*)realloc(bm->aux->info16, 2 * bm->aux->max);

        }
        }

-       for (i = n = 0; i < _n; ++i) {
+       for (i = n = 0, rms = 0; i < _n; ++i) {

                const bam_pileup1_t *p = pl + i;
                uint32_t q, x = 0, qq;
                const bam_pileup1_t *p = pl + i;
                uint32_t q, x = 0, qq;

+               uint16_t y = 0;

                if (p->is_del || (p->b->core.flag&BAM_FUNMAP)) continue;
                q = (uint32_t)bam1_qual(p->b)[p->qpos];
                if (p->is_del || (p->b->core.flag&BAM_FUNMAP)) continue;
                q = (uint32_t)bam1_qual(p->b)[p->qpos];

+               if (q < bm->min_baseQ) continue;

                x |= (uint32_t)bam1_strand(p->b) << 18 | q << 8 | p->b->core.qual;
                x |= (uint32_t)bam1_strand(p->b) << 18 | q << 8 | p->b->core.qual;

+               y |= bam1_strand(p->b)<<4;

                if (p->b->core.qual < q) q = p->b->core.qual;
                if (p->b->core.qual < q) q = p->b->core.qual;

+               c = p->b->core.qual < bm->cap_mapQ? p->b->core.qual : bm->cap_mapQ;
+               rms += c * c;

                x |= q << 24;
                x |= q << 24;

+               y |= q << 5;

                qq = bam1_seqi(bam1_seq(p->b), p->qpos);
                q = bam_nt16_nt4_table[qq? qq : ref_base];
                qq = bam1_seqi(bam1_seq(p->b), p->qpos);
                q = bam_nt16_nt4_table[qq? qq : ref_base];

-               if (!p->is_del && q < 4) x |= 1 << 21 | q << 16;
+               if (!p->is_del && q < 4) x |= 1 << 21 | q << 16, y |= q;
+               bm->aux->info16[n] = y;

                bm->aux->info[n++] = x;
        }
                bm->aux->info[n++] = x;
        }

+       rms = (uint8_t)(sqrt((double)rms / n) + .499);
+       if (bm->errmod == BAM_ERRMOD_MAQ2) {
+               errmod_cal(bm->aux->em, n, 4, bm->aux->info16, p);
+               goto goto_glf;
+       }

        ks_introsort(uint32_t, n, bm->aux->info);
        // generate esum and fsum
        b = (glf_call_aux_t*)calloc(1, sizeof(glf_call_aux_t));
        for (k = 0; k != 8; ++k) w[k] = 0;
        ks_introsort(uint32_t, n, bm->aux->info);
        // generate esum and fsum
        b = (glf_call_aux_t*)calloc(1, sizeof(glf_call_aux_t));
        for (k = 0; k != 8; ++k) w[k] = 0;

-       rms = 0;

        for (j = n - 1; j >= 0; --j) { // calculate esum and fsum
                uint32_t info = bm->aux->info[j];
        for (j = n - 1; j >= 0; --j) { // calculate esum and fsum
                uint32_t info = bm->aux->info[j];

-               int tmp;

                if (info>>24 < 4 && (info>>8&0x3f) != 0) info = 4<<24 | (info&0xffffff);
                k = info>>16&7;
                if (info>>24 > 0) {
                if (info>>24 < 4 && (info>>8&0x3f) != 0) info = 4<<24 | (info&0xffffff);
                k = info>>16&7;
                if (info>>24 > 0) {


@@ -172,17 +189,14 @@ glf1_t *bam_maqcns_glfgen(int _n, const bam_pileup1_t *pl, uint8_t ref_base, bam
                        if (w[k] < 0xff) ++w[k];
                        ++b->c[k&3];
                }
                        if (w[k] < 0xff) ++w[k];
                        ++b->c[k&3];
                }

-               tmp = (int)(info&0xff) < bm->cap_mapQ? (int)(info&0xff) : bm->cap_mapQ;
-               rms += tmp * tmp;

        }
        }

-       b->rms_mapQ = (uint8_t)(sqrt((double)rms / n) + .499);

        // rescale ->c[]
        for (j = c = 0; j != 4; ++j) c += b->c[j];
        if (c > 255) {
                for (j = 0; j != 4; ++j) b->c[j] = (int)(254.0 * b->c[j] / c + 0.5);
                for (j = c = 0; j != 4; ++j) c += b->c[j];
        }
        // rescale ->c[]
        for (j = c = 0; j != 4; ++j) c += b->c[j];
        if (c > 255) {
                for (j = 0; j != 4; ++j) b->c[j] = (int)(254.0 * b->c[j] / c + 0.5);
                for (j = c = 0; j != 4; ++j) c += b->c[j];
        }

-       if (!bm->is_soap) {
+       if (bm->errmod == BAM_ERRMOD_MAQ) {

                // generate likelihood
                for (j = 0; j != 4; ++j) {
                        // homozygous
                // generate likelihood
                for (j = 0; j != 4; ++j) {
                        // homozygous


@@ -234,7 +248,7 @@ glf1_t *bam_maqcns_glfgen(int _n, const bam_pileup1_t *pl, uint8_t ref_base, bam
                        if (max1 > max2 && (min_k != max_k || min1 + 1.0 > min2))
                                p[max_k<<2|max_k] = min1 > 1.0? min1 - 1.0 : 0.0;
                }
                        if (max1 > max2 && (min_k != max_k || min1 + 1.0 > min2))
                                p[max_k<<2|max_k] = min1 > 1.0? min1 - 1.0 : 0.0;
                }

-       } else { // apply the SOAP model
+       } else if (bm->errmod == BAM_ERRMOD_SOAP) { // apply the SOAP model

                // generate likelihood
                for (j = 0; j != 4; ++j) {
                        float tmp;
                // generate likelihood
                for (j = 0; j != 4; ++j) {
                        float tmp;


@@ -251,8 +265,9 @@ glf1_t *bam_maqcns_glfgen(int _n, const bam_pileup1_t *pl, uint8_t ref_base, bam
                }
        }
 
                }
        }
 

+goto_glf:

        // convert necessary information to glf1_t
        // convert necessary information to glf1_t

-       g->ref_base = ref_base; g->max_mapQ = b->rms_mapQ;
+       g->ref_base = ref_base; g->max_mapQ = rms;

        g->depth = n > 16777215? 16777215 : n;
        for (j = 0; j != 4; ++j)
                for (k = j; k < 4; ++k)
        g->depth = n > 16777215? 16777215 : n;
        for (j = 0; j != 4; ++j)
                for (k = j; k < 4; ++k)


@@ -268,26 +283,25 @@ glf1_t *bam_maqcns_glfgen(int _n, const bam_pileup1_t *pl, uint8_t ref_base, bam
 
 uint32_t glf2cns(const glf1_t *g, int q_r)
 {
 
 uint32_t glf2cns(const glf1_t *g, int q_r)
 {

-       int i, j, k, tmp[16], min = 10000, min2 = 10000, min3 = 10000, min_g = -1, min_g2 = -1;
+       int i, j, k, p[10], ref4;

        uint32_t x = 0;
        uint32_t x = 0;

+       ref4 = bam_nt16_nt4_table[g->ref_base];

        for (i = k = 0; i < 4; ++i)
                for (j = i; j < 4; ++j) {
        for (i = k = 0; i < 4; ++i)
                for (j = i; j < 4; ++j) {

-                       tmp[j<<2|i] = -1;
-                       tmp[i<<2|j] = g->lk[k++] + (i == j? 0 : q_r);
+                       int prior = (i == ref4 && j == ref4? 0 : i == ref4 || j == ref4? q_r : q_r + 3);
+                       p[k] = (g->lk[k] + prior)<<4 | i<<2 | j;
+                       ++k;

                }
                }

-       for (i = 0; i < 16; ++i) {
-               if (tmp[i] < 0) continue;
-               if (tmp[i] < min) {
-                       min3 = min2; min2 = min; min = tmp[i]; min_g2 = min_g; min_g = i;
-               } else if (tmp[i] < min2) {
-                       min3 = min2; min2 = tmp[i]; min_g2 = i;
-               } else if (tmp[i] < min3) min3 = tmp[i];
-       }
-       x = min_g >= 0? (1U<<(min_g>>2&3) | 1U<<(min_g&3)) << 28 : 0xf << 28;
-       x |= min_g2 >= 0? (1U<<(min_g2>>2&3) | 1U<<(min_g2&3)) << 24 : 0xf << 24;
-       x |= (uint32_t)g->max_mapQ << 16;
-       x |= min2 < 10000? (min2 - min < 256? min2 - min : 255) << 8 : 0xff << 8;
-       x |= min2 < 10000 && min3 < 10000? (min3 - min2 < 256? min3 - min2 : 255) : 0xff;
+       for (i = 1; i < 10; ++i) // insertion sort
+               for (j = i; j > 0 && p[j] < p[j-1]; --j)
+                       k = p[j], p[j] = p[j-1], p[j-1] = k;
+       x = (1u<<(p[0]&3) | 1u<<(p[0]>>2&3)) << 28; // the best genotype
+       x |= (uint32_t)g->max_mapQ << 16; // rms mapQ
+       x |= ((p[1]>>4) - (p[0]>>4) < 256? (p[1]>>4) - (p[0]>>4) : 255) << 8; // consensus Q
+       for (k = 0; k < 10; ++k)
+               if ((p[k]&0xf) == (ref4<<2|ref4)) break;
+       if (k == 10) k = 9;
+       x |= (p[k]>>4) - (p[0]>>4) < 256? (p[k]>>4) - (p[0]>>4) : 255; // snp Q

        return x;
 }
 
        return x;
 }
 


@@ -297,7 +311,7 @@ uint32_t bam_maqcns_call(int n, const bam_pileup1_t *pl, bam_maqcns_t *bm)
        uint32_t x;
        if (n) {
                g = bam_maqcns_glfgen(n, pl, 0xf, bm);
        uint32_t x;
        if (n) {
                g = bam_maqcns_glfgen(n, pl, 0xf, bm);

-               x = glf2cns(g, (int)(bm->q_r + 0.5));
+               x = g->depth == 0? (0xfU<<28 | 0xfU<<24) : glf2cns(g, (int)(bm->q_r + 0.5));

                free(g);
        } else x = 0xfU<<28 | 0xfU<<24;
        return x;
                free(g);
        } else x = 0xfU<<28 | 0xfU<<24;
        return x;


@@ -448,7 +462,7 @@ bam_maqindel_ret_t *bam_maqindel(int n, int pos, const bam_maqindel_opt_t *mi, c
                for (i = 0; i < n_types; ++i) {
                        ka_param_t ap = ka_param_blast;
                        ap.band_width = 2 * types[n_types - 1] + 2;
                for (i = 0; i < n_types; ++i) {
                        ka_param_t ap = ka_param_blast;
                        ap.band_width = 2 * types[n_types - 1] + 2;

-                       ap.gap_end = 0;
+                       ap.gap_end_ext = 0;

                        // write ref2
                        for (k = 0, j = left; j <= pos; ++j)
                                ref2[k++] = bam_nt16_nt4_table[bam_nt16_table[(int)ref[j]]];
                        // write ref2
                        for (k = 0, j = left; j <= pos; ++j)
                                ref2[k++] = bam_nt16_nt4_table[bam_nt16_table[(int)ref[j]]];

diff --git a/bam_maqcns.h b/bam_maqcns.h
index 6cc5355fc2db2b1b19e6a3d2d330306a240fa15a..291ae533c3c38cd152b3c9c8fa8e820386a3718d 100644 (file)
--- a/bam_maqcns.h
+++ b/bam_maqcns.h
@@ -3,11 +3,15 @@
 
 #include "glf.h"
 
 
 #include "glf.h"
 

+#define BAM_ERRMOD_MAQ2 0
+#define BAM_ERRMOD_MAQ  1
+#define BAM_ERRMOD_SOAP 2
+

 struct __bmc_aux_t;
 
 typedef struct {
        float het_rate, theta;
 struct __bmc_aux_t;
 
 typedef struct {
        float het_rate, theta;

-       int n_hap, cap_mapQ, is_soap;
+       int n_hap, cap_mapQ, errmod, min_baseQ;

 
        float eta, q_r;
        double *fk, *coef;
 
        float eta, q_r;
        double *fk, *coef;

diff --git a/bam_md.c b/bam_md.c
index 17b0a4a4fa6943313be1b6d78b18c5255fe44982..f65b845b5d594980d4b4dba36122672c71b19cc9 100644 (file)
--- a/bam_md.c
+++ b/bam_md.c
@@ -2,9 +2,11 @@
 #include <assert.h>
 #include <string.h>
 #include <ctype.h>
 #include <assert.h>
 #include <string.h>
 #include <ctype.h>

+#include <math.h>

 #include "faidx.h"
 #include "sam.h"
 #include "kstring.h"
 #include "faidx.h"
 #include "sam.h"
 #include "kstring.h"

+#include "kaln.h"

 
 void bam_fillmd1_core(bam1_t *b, char *ref, int is_equal, int max_nm)
 {
 
 void bam_fillmd1_core(bam1_t *b, char *ref, int is_equal, int max_nm)
 {


@@ -108,24 +110,134 @@ void bam_fillmd1(bam1_t *b, char *ref, int is_equal)
        bam_fillmd1_core(b, ref, is_equal, 0);
 }
 
        bam_fillmd1_core(b, ref, is_equal, 0);
 }
 

+int bam_cap_mapQ(bam1_t *b, char *ref, int thres)
+{
+       uint8_t *seq = bam1_seq(b), *qual = bam1_qual(b);
+       uint32_t *cigar = bam1_cigar(b);
+       bam1_core_t *c = &b->core;
+       int i, x, y, mm, q, len, clip_l, clip_q;
+       double t;
+       if (thres < 0) thres = 40; // set the default
+       mm = q = len = clip_l = clip_q = 0;
+       for (i = y = 0, x = c->pos; i < c->n_cigar; ++i) {
+               int j, l = cigar[i]>>4, op = cigar[i]&0xf;
+               if (op == BAM_CMATCH) {
+                       for (j = 0; j < l; ++j) {
+                               int z = y + j;
+                               int c1 = bam1_seqi(seq, z), c2 = bam_nt16_table[(int)ref[x+j]];
+                               if (ref[x+j] == 0) break; // out of boundary
+                               if (c2 != 15 && c1 != 15 && qual[z] >= 13) { // not ambiguous
+                                       ++len;
+                                       if (c1 && c1 != c2 && qual[z] >= 13) { // mismatch
+                                               ++mm;
+                                               q += qual[z] > 33? 33 : qual[z];
+                                       }
+                               }
+                       }
+                       if (j < l) break;
+                       x += l; y += l; len += l;
+               } else if (op == BAM_CDEL) {
+                       for (j = 0; j < l; ++j)
+                               if (ref[x+j] == 0) break;
+                       if (j < l) break;
+                       x += l;
+               } else if (op == BAM_CSOFT_CLIP) {
+                       for (j = 0; j < l; ++j) clip_q += qual[y+j];
+                       clip_l += l;
+                       y += l;
+               } else if (op == BAM_CHARD_CLIP) {
+                       clip_q += 13 * l;
+                       clip_l += l;
+               } else if (op == BAM_CINS) y += l;
+               else if (op == BAM_CREF_SKIP) x += l;
+       }
+       for (i = 0, t = 1; i < mm; ++i)
+               t *= (double)len / (i+1);
+       t = q - 4.343 * log(t) + clip_q / 5.;
+       if (t > thres) return -1;
+       if (t < 0) t = 0;
+       t = sqrt((thres - t) / thres) * thres;
+//     fprintf(stderr, "%s %lf %d\n", bam1_qname(b), t, q);
+       return (int)(t + .499);
+}
+
+int bam_prob_realn(bam1_t *b, const char *ref)
+{
+       int k, i, bw, x, y, yb, ye, xb, xe;
+       uint32_t *cigar = bam1_cigar(b);
+       bam1_core_t *c = &b->core;
+       ka_probpar_t conf = ka_probpar_def;
+       // find the start and end of the alignment
+       if (c->flag & BAM_FUNMAP) return -1;
+       x = c->pos, y = 0, yb = ye = xb = xe = -1;
+       for (k = 0; k < c->n_cigar; ++k) {
+               int op, l;
+               op = cigar[k]&0xf; l = cigar[k]>>4;
+               if (op == BAM_CMATCH) {
+                       if (yb < 0) yb = y;
+                       if (xb < 0) xb = x;
+                       ye = y + l; xe = x + l;
+                       x += l; y += l;
+               } else if (op == BAM_CSOFT_CLIP || op == BAM_CINS) y += l;
+               else if (op == BAM_CDEL) x += l;
+               else if (op == BAM_CREF_SKIP) return -1;
+       }
+       // set bandwidth and the start and the end
+       bw = 7;
+       if (abs((xe - xb) - (ye - yb)) > bw)
+               bw = abs((xe - xb) - (ye - yb)) + 3;
+       conf.bw = bw;
+       xb -= yb + bw/2; if (xb < 0) xb = 0;
+       xe += c->l_qseq - ye + bw/2;
+       if (xe - xb - c->l_qseq > bw)
+               xb += (xe - xb - c->l_qseq - bw) / 2, xe -= (xe - xb - c->l_qseq - bw) / 2;
+       { // glocal
+               uint8_t *s, *r, *q, *seq = bam1_seq(b), *qual = bam1_qual(b);
+               int *state;
+               s = calloc(c->l_qseq, 1);
+               for (i = 0; i < c->l_qseq; ++i) s[i] = bam_nt16_nt4_table[bam1_seqi(seq, i)];
+               r = calloc(xe - xb, 1);
+               for (i = xb; i < xe; ++i)
+                       r[i-xb] = bam_nt16_nt4_table[bam_nt16_table[(int)ref[i]]];
+               state = calloc(c->l_qseq, sizeof(int));
+               q = calloc(c->l_qseq, 1);
+               ka_prob_glocal(r, xe-xb, s, c->l_qseq, qual, &conf, state, q);
+               for (k = 0, x = c->pos, y = 0; k < c->n_cigar; ++k) {
+                       int op = cigar[k]&0xf, l = cigar[k]>>4;
+                       if (op == BAM_CMATCH) {
+                               for (i = y; i < y + l; ++i) {
+                                       if ((state[i]&3) != 0 || state[i]>>2 != x - xb + (i - y)) qual[i] = 0;
+                                       else qual[i] = qual[i] < q[i]? qual[i] : q[i];
+                               }
+                               x += l; y += l;
+                       } else if (op == BAM_CSOFT_CLIP || op == BAM_CINS) y += l;
+                       else if (op == BAM_CDEL) x += l;
+               }
+               free(s); free(r); free(q); free(state);
+       }
+       return 0;
+}
+

 int bam_fillmd(int argc, char *argv[])
 {
 int bam_fillmd(int argc, char *argv[])
 {

-       int c, is_equal = 0, tid = -2, ret, len, is_bam_out, is_sam_in, is_uncompressed, max_nm = 0;
+       int c, is_equal = 0, tid = -2, ret, len, is_bam_out, is_sam_in, is_uncompressed, max_nm = 0, is_realn, capQ = 0;

        samfile_t *fp, *fpout = 0;
        faidx_t *fai;
        char *ref = 0, mode_w[8], mode_r[8];
        bam1_t *b;
 
        samfile_t *fp, *fpout = 0;
        faidx_t *fai;
        char *ref = 0, mode_w[8], mode_r[8];
        bam1_t *b;
 

-       is_bam_out = is_sam_in = is_uncompressed = 0;
+       is_bam_out = is_sam_in = is_uncompressed = is_realn = 0;

        mode_w[0] = mode_r[0] = 0;
        strcpy(mode_r, "r"); strcpy(mode_w, "w");
        mode_w[0] = mode_r[0] = 0;
        strcpy(mode_r, "r"); strcpy(mode_w, "w");

-       while ((c = getopt(argc, argv, "eubSn:")) >= 0) {
+       while ((c = getopt(argc, argv, "reubSC:n:")) >= 0) {

                switch (c) {
                switch (c) {

+               case 'r': is_realn = 1; break;

                case 'e': is_equal = 1; break;
                case 'b': is_bam_out = 1; break;
                case 'u': is_uncompressed = is_bam_out = 1; break;
                case 'S': is_sam_in = 1; break;
                case 'n': max_nm = atoi(optarg); break;
                case 'e': is_equal = 1; break;
                case 'b': is_bam_out = 1; break;
                case 'u': is_uncompressed = is_bam_out = 1; break;
                case 'S': is_sam_in = 1; break;
                case 'n': max_nm = atoi(optarg); break;

+               case 'C': capQ = atoi(optarg); break;

                default: fprintf(stderr, "[bam_fillmd] unrecognized option '-%c'\n", c); return 1;
                }
        }
                default: fprintf(stderr, "[bam_fillmd] unrecognized option '-%c'\n", c); return 1;
                }
        }


@@ -135,11 +247,12 @@ int bam_fillmd(int argc, char *argv[])
        if (is_uncompressed) strcat(mode_w, "u");
        if (optind + 1 >= argc) {
                fprintf(stderr, "\n");
        if (is_uncompressed) strcat(mode_w, "u");
        if (optind + 1 >= argc) {
                fprintf(stderr, "\n");

-               fprintf(stderr, "Usage:   samtools fillmd [-eubS] <aln.bam> <ref.fasta>\n\n");
+               fprintf(stderr, "Usage:   samtools fillmd [-eubrS] <aln.bam> <ref.fasta>\n\n");

                fprintf(stderr, "Options: -e       change identical bases to '='\n");
                fprintf(stderr, "         -u       uncompressed BAM output (for piping)\n");
                fprintf(stderr, "         -b       compressed BAM output\n");
                fprintf(stderr, "Options: -e       change identical bases to '='\n");
                fprintf(stderr, "         -u       uncompressed BAM output (for piping)\n");
                fprintf(stderr, "         -b       compressed BAM output\n");

-               fprintf(stderr, "         -S       the input is SAM with header\n\n");
+               fprintf(stderr, "         -S       the input is SAM with header\n");
+               fprintf(stderr, "         -r       read-independent local realignment\n\n");

                return 1;
        }
        fp = samopen(argv[optind], mode_r, 0);
                return 1;
        }
        fp = samopen(argv[optind], mode_r, 0);


@@ -162,6 +275,12 @@ int bam_fillmd(int argc, char *argv[])
                                        fprintf(stderr, "[bam_fillmd] fail to find sequence '%s' in the reference.\n",
                                                        fp->header->target_name[tid]);
                        }
                                        fprintf(stderr, "[bam_fillmd] fail to find sequence '%s' in the reference.\n",
                                                        fp->header->target_name[tid]);
                        }

+//                     if (is_realn) bam_realn(b, ref);
+                       if (is_realn) bam_prob_realn(b, ref);
+                       if (capQ > 10) {
+                               int q = bam_cap_mapQ(b, ref, capQ);
+                               if (b->core.qual > q) b->core.qual = q;
+                       }

                        if (ref) bam_fillmd1_core(b, ref, is_equal, max_nm);
                }
                samwrite(fpout, b);
                        if (ref) bam_fillmd1_core(b, ref, is_equal, max_nm);
                }
                samwrite(fpout, b);

diff --git a/bam_pileup.c b/bam_pileup.c
index 3c41a169127fcc5d3977df3e8febd944b7ced77e..a49f795d73d4f0ed1ad3a9592108f3be2a917579 100644 (file)
--- a/bam_pileup.c
+++ b/bam_pileup.c
@@ -4,9 +4,16 @@
 #include <assert.h>
 #include "sam.h"
 
 #include <assert.h>
 #include "sam.h"
 

+typedef struct {
+       int k, x, y, end;
+} cstate_t;
+
+static cstate_t g_cstate_null = { -1, 0, 0, 0 };
+

 typedef struct __linkbuf_t {
        bam1_t b;
        uint32_t beg, end;
 typedef struct __linkbuf_t {
        bam1_t b;
        uint32_t beg, end;

+       cstate_t s;

        struct __linkbuf_t *next;
 } lbnode_t;
 
        struct __linkbuf_t *next;
 } lbnode_t;
 


@@ -53,68 +60,86 @@ static inline void mp_free(mempool_t *mp, lbnode_t *p)
 
 /* --- BEGIN: Auxiliary functions */
 
 
 /* --- BEGIN: Auxiliary functions */
 

-static inline int resolve_cigar(bam_pileup1_t *p, uint32_t pos)
+/* s->k: the index of the CIGAR operator that has just been processed.
+   s->x: the reference coordinate of the start of s->k
+   s->y: the query coordiante of the start of s->k
+ */
+static inline int resolve_cigar2(bam_pileup1_t *p, uint32_t pos, cstate_t *s)

 {
 {

-       unsigned k;
+#define _cop(c) ((c)&BAM_CIGAR_MASK)
+#define _cln(c) ((c)>>BAM_CIGAR_SHIFT)
+

        bam1_t *b = p->b;
        bam1_core_t *c = &b->core;
        bam1_t *b = p->b;
        bam1_core_t *c = &b->core;

-       uint32_t x = c->pos, y = 0;
-       int ret = 1, is_restart = 1;
-
-       if (c->flag&BAM_FUNMAP) return 0; // unmapped read
-       assert(x <= pos); // otherwise a bug
-       p->qpos = -1; p->indel = 0; p->is_del = p->is_head = p->is_tail = 0;
-       for (k = 0; k < c->n_cigar; ++k) {
-               int op = bam1_cigar(b)[k] & BAM_CIGAR_MASK; // operation
-               int l = bam1_cigar(b)[k] >> BAM_CIGAR_SHIFT; // length
-               if (op == BAM_CMATCH) { // NOTE: this assumes the first and the last operation MUST BE a match or a clip
-                       if (x + l > pos) { // overlap with pos
-                               p->indel = p->is_del = 0;
-                               p->qpos = y + (pos - x);
-                               if (x == pos && is_restart) p->is_head = 1;
-                               if (x + l - 1 == pos) { // come to the end of a match
-                                       int has_next_match = 0;
-                                       unsigned i;
-                                       for (i = k + 1; i < c->n_cigar; ++i) {
-                                               uint32_t cigar = bam1_cigar(b)[i];
-                                               int opi = cigar&BAM_CIGAR_MASK;
-                                               if (opi == BAM_CMATCH) {
-                                                       has_next_match = 1;
-                                                       break;
-                                               } else if (opi == BAM_CSOFT_CLIP || opi == BAM_CREF_SKIP || opi == BAM_CHARD_CLIP) break;
-                                       }
-                                       if (!has_next_match) p->is_tail = 1;
-                                       if (k < c->n_cigar - 1 && has_next_match) { // there are additional operation(s)
-                                               uint32_t cigar = bam1_cigar(b)[k+1]; // next CIGAR
-                                               int op_next = cigar&BAM_CIGAR_MASK; // next CIGAR operation
-                                               if (op_next == BAM_CDEL) p->indel = -(int32_t)(cigar>>BAM_CIGAR_SHIFT); // del
-                                               else if (op_next == BAM_CINS) p->indel = cigar>>BAM_CIGAR_SHIFT; // ins
-                                               else if (op_next == BAM_CPAD && k + 2 < c->n_cigar) { // no working for adjacent padding
-                                                       cigar = bam1_cigar(b)[k+2]; op_next = cigar&BAM_CIGAR_MASK;
-                                                       if (op_next == BAM_CDEL) p->indel = -(int32_t)(cigar>>BAM_CIGAR_SHIFT); // del
-                                                       else if (op_next == BAM_CINS) p->indel = cigar>>BAM_CIGAR_SHIFT; // ins
-                                               }
-                                       }
+       uint32_t *cigar = bam1_cigar(b);
+       int k, is_head = 0;
+       // determine the current CIGAR operation
+//     fprintf(stderr, "%s\tpos=%d\tend=%d\t(%d,%d,%d)\n", bam1_qname(b), pos, s->end, s->k, s->x, s->y);
+       if (s->k == -1) { // never processed
+               is_head = 1;
+               if (c->n_cigar == 1) { // just one operation, save a loop
+                       if (_cop(cigar[0]) == BAM_CMATCH) s->k = 0, s->x = c->pos, s->y = 0;
+               } else { // find the first match
+                       for (k = 0, s->x = c->pos, s->y = 0; k < c->n_cigar; ++k) {
+                               int op = _cop(cigar[k]);
+                               int l = _cln(cigar[k]);
+                               if (op == BAM_CMATCH) break;
+                               else if (op == BAM_CDEL || op == BAM_CREF_SKIP) s->x += l;
+                               else if (op == BAM_CINS || op == BAM_CSOFT_CLIP) s->y += l;
+                       }
+                       assert(k < c->n_cigar);
+                       s->k = k;
+               }
+       } else { // the read has been processed before
+               int op, l = _cln(cigar[s->k]);
+               if (pos - s->x >= l) { // jump to the next operation
+                       assert(s->k < c->n_cigar); // otherwise a bug: this function should not be called in this case
+                       op = _cop(cigar[s->k+1]);
+                       if (op == BAM_CMATCH || op == BAM_CDEL || op == BAM_CREF_SKIP) { // jump to the next without a loop
+                               if (_cop(cigar[s->k]) == BAM_CMATCH) s->y += l;
+                               s->x += l;
+                               ++s->k;
+                       } else { // find the next M/D/N
+                               if (_cop(cigar[s->k]) == BAM_CMATCH) s->y += l;
+                               s->x += l;
+                               for (k = s->k + 1; k < c->n_cigar; ++k) {
+                                       op = _cop(cigar[k]), l = _cln(cigar[k]);
+                                       if (op == BAM_CMATCH || op == BAM_CDEL || op == BAM_CREF_SKIP) break;
+                                       else if (op == BAM_CINS || op == BAM_CSOFT_CLIP) s->y += l;

                                }
                                }

+                               s->k = k;

                        }
                        }

-                       x += l; y += l;
-               } else if (op == BAM_CDEL) { // then set ->is_del
-                       if (x + l > pos) {
-                               p->indel = 0; p->is_del = 1;
-                               p->qpos = y + (pos - x);
+                       assert(s->k < c->n_cigar); // otherwise a bug
+               } // else, do nothing
+       }
+       { // collect pileup information
+               int op, l;
+               op = _cop(cigar[s->k]); l = _cln(cigar[s->k]);
+               p->is_del = p->indel = p->is_refskip = 0;
+               if (s->x + l - 1 == pos && s->k + 1 < c->n_cigar) { // peek the next operation
+                       int op2 = _cop(cigar[s->k+1]);
+                       int l2 = _cln(cigar[s->k+1]);
+                       if (op2 == BAM_CDEL) p->indel = -(int)l2;
+                       else if (op2 == BAM_CINS) p->indel = l2;
+                       else if (op2 == BAM_CPAD && s->k + 2 < c->n_cigar) { // no working for adjacent padding
+                               int l3 = 0;
+                               for (k = s->k + 2; k < c->n_cigar; ++k) {
+                                       op2 = _cop(cigar[k]); l2 = _cln(cigar[k]);
+                                       if (op2 == BAM_CINS) l3 += l2;
+                                       else if (op2 == BAM_CDEL || op2 == BAM_CMATCH || op2 == BAM_CREF_SKIP) break;
+                               }
+                               if (l3 > 0) p->indel = l3;

                        }
                        }

-                       x += l;
-               } else if (op == BAM_CREF_SKIP) x += l;
-               else if (op == BAM_CINS || op == BAM_CSOFT_CLIP) y += l;
-               if (is_restart) is_restart ^= (op == BAM_CMATCH);
-               else is_restart ^= (op == BAM_CREF_SKIP || op == BAM_CSOFT_CLIP || op == BAM_CHARD_CLIP);
-               if (x > pos) {
-                       if (op == BAM_CREF_SKIP) ret = 0; // then do not put it into pileup at all
-                       break;

                }
                }

+               if (op == BAM_CMATCH) {
+                       p->qpos = s->y + (pos - s->x);
+               } else if (op == BAM_CDEL || op == BAM_CREF_SKIP) {
+                       p->is_del = 1; p->qpos = s->y; // FIXME: distinguish D and N!!!!!
+                       p->is_refskip = (op == BAM_CREF_SKIP);
+               } // cannot be other operations; otherwise a bug
+               p->is_head = (pos == c->pos); p->is_tail = (pos == s->end);

        }
        }

-       assert(x > pos); // otherwise a bug
-       return ret;
+       return 1;

 }
 
 /* --- END: Auxiliary functions */
 }
 
 /* --- END: Auxiliary functions */


@@ -183,7 +208,7 @@ const bam_pileup1_t *bam_plp_next(bam_plp_t iter, int *_tid, int *_pos, int *_n_
                                        iter->plp = (bam_pileup1_t*)realloc(iter->plp, sizeof(bam_pileup1_t) * iter->max_plp);
                                }
                                iter->plp[n_plp].b = &p->b;
                                        iter->plp = (bam_pileup1_t*)realloc(iter->plp, sizeof(bam_pileup1_t) * iter->max_plp);
                                }
                                iter->plp[n_plp].b = &p->b;

-                               if (resolve_cigar(iter->plp + n_plp, iter->pos)) ++n_plp; // skip the read if we are looking at ref-skip
+                               if (resolve_cigar2(iter->plp + n_plp, iter->pos, &p->s)) ++n_plp; // actually always true...

                        }
                }
                iter->head = iter->dummy->next; // dummy->next may be changed
                        }
                }
                iter->head = iter->dummy->next; // dummy->next may be changed


@@ -217,6 +242,7 @@ int bam_plp_push(bam_plp_t iter, const bam1_t *b)
                if (b->core.flag & iter->flag_mask) return 0;
                bam_copy1(&iter->tail->b, b);
                iter->tail->beg = b->core.pos; iter->tail->end = bam_calend(&b->core, bam1_cigar(b));
                if (b->core.flag & iter->flag_mask) return 0;
                bam_copy1(&iter->tail->b, b);
                iter->tail->beg = b->core.pos; iter->tail->end = bam_calend(&b->core, bam1_cigar(b));

+               iter->tail->s = g_cstate_null; iter->tail->s.end = iter->tail->end - 1; // initialize cstate_t

                if (b->core.tid < iter->max_tid) {
                        fprintf(stderr, "[bam_pileup_core] the input is not sorted (chromosomes out of order)\n");
                        iter->error = 1;
                if (b->core.tid < iter->max_tid) {
                        fprintf(stderr, "[bam_pileup_core] the input is not sorted (chromosomes out of order)\n");
                        iter->error = 1;


@@ -387,7 +413,7 @@ int bam_mplp_auto(bam_mplp_t iter, int *_tid, int *_pos, int *n_plp, const bam_p
        if (new_min == (uint64_t)-1) return 0;
        *_tid = new_min>>32; *_pos = (uint32_t)new_min;
        for (i = 0; i < iter->n; ++i) {
        if (new_min == (uint64_t)-1) return 0;
        *_tid = new_min>>32; *_pos = (uint32_t)new_min;
        for (i = 0; i < iter->n; ++i) {

-               if (iter->pos[i] == iter->min) {
+               if (iter->pos[i] == iter->min) { // FIXME: valgrind reports "uninitialised value(s) at this line"

                        n_plp[i] = iter->n_plp[i], plp[i] = iter->plp[i];
                        ++ret;
                } else n_plp[i] = 0, plp[i] = 0;
                        n_plp[i] = iter->n_plp[i], plp[i] = iter->plp[i];
                        ++ret;
                } else n_plp[i] = 0, plp[i] = 0;

diff --git a/bam_plcmd.c b/bam_plcmd.c
index 6804795bd107405982cada8544051cc073fdf55e..dca85181f99bf1c7343650846039153a019e7ea9 100644 (file)
--- a/bam_plcmd.c
+++ b/bam_plcmd.c
@@ -22,6 +22,7 @@ KHASH_MAP_INIT_INT64(64, indel_list_t)
 #define BAM_PLF_1STBASE    0x80
 #define BAM_PLF_ALLBASE    0x100
 #define BAM_PLF_READPOS    0x200
 #define BAM_PLF_1STBASE    0x80
 #define BAM_PLF_ALLBASE    0x100
 #define BAM_PLF_READPOS    0x200

+#define BAM_PLF_NOBAQ      0x400

 
 typedef struct {
        bam_header_t *h;
 
 typedef struct {
        bam_header_t *h;


@@ -32,6 +33,7 @@ typedef struct {
        uint32_t format;
        int tid, len, last_pos;
        int mask;
        uint32_t format;
        int tid, len, last_pos;
        int mask;

+       int capQ_thres, min_baseQ;

     int max_depth;  // for indel calling, ignore reads with the depth too high. 0 for unlimited
        char *ref;
        glfFile fp_glf; // for glf output only
     int max_depth;  // for indel calling, ignore reads with the depth too high. 0 for unlimited
        char *ref;
        glfFile fp_glf; // for glf output only


@@ -89,6 +91,21 @@ static khash_t(64) *load_pos(const char *fn, bam_header_t *h)
        return hash;
 }
 
        return hash;
 }
 

+static inline int printw(int c, FILE *fp)
+{
+       char buf[16];
+       int l, x;
+       if (c == 0) return fputc('0', fp);
+       for (l = 0, x = c < 0? -c : c; x > 0; x /= 10) buf[l++] = x%10 + '0';
+       if (c < 0) buf[l++] = '-';
+       buf[l] = 0;
+       for (x = 0; x < l/2; ++x) {
+               int y = buf[x]; buf[x] = buf[l-1-x]; buf[l-1-x] = y;
+       }
+       fputs(buf, fp);
+       return 0;
+}
+

 // an analogy to pileup_func() below
 static int glt3_func(uint32_t tid, uint32_t pos, int n, const bam_pileup1_t *pu, void *data)
 {
 // an analogy to pileup_func() below
 static int glt3_func(uint32_t tid, uint32_t pos, int n, const bam_pileup1_t *pu, void *data)
 {


@@ -158,29 +175,38 @@ static int glt3_func(uint32_t tid, uint32_t pos, int n, const bam_pileup1_t *pu,
        return 0;
 }
 
        return 0;
 }
 

-static void pileup_seq(const bam_pileup1_t *p, int pos, int ref_len, const char *ref)
+static inline void pileup_seq(const bam_pileup1_t *p, int pos, int ref_len, const char *ref)

 {
 {

-       if (p->is_head) printf("^%c", p->b->core.qual > 93? 126 : p->b->core.qual + 33);
+       int j;
+       if (p->is_head) {
+               putchar('^');
+               putchar(p->b->core.qual > 93? 126 : p->b->core.qual + 33);
+       }

        if (!p->is_del) {
        if (!p->is_del) {

-               int j, rb, c = bam_nt16_rev_table[bam1_seqi(bam1_seq(p->b), p->qpos)];
-               rb = (ref && pos < ref_len)? ref[pos] : 'N';
-               if (c == '=' || toupper(c) == toupper(rb)) c = bam1_strand(p->b)? ',' : '.';
-               else c = bam1_strand(p->b)? tolower(c) : toupper(c);
+               int c = bam_nt16_rev_table[bam1_seqi(bam1_seq(p->b), p->qpos)];
+               if (ref) {
+                       int rb = pos < ref_len? ref[pos] : 'N';
+                       if (c == '=' || bam_nt16_table[c] == bam_nt16_table[rb]) c = bam1_strand(p->b)? ',' : '.';
+                       else c = bam1_strand(p->b)? tolower(c) : toupper(c);
+               } else {
+                       if (c == '=') c = bam1_strand(p->b)? ',' : '.';
+                       else c = bam1_strand(p->b)? tolower(c) : toupper(c);
+               }

                putchar(c);
                putchar(c);

-               if (p->indel > 0) {
-                       printf("+%d", p->indel);
-                       for (j = 1; j <= p->indel; ++j) {
-                               c = bam_nt16_rev_table[bam1_seqi(bam1_seq(p->b), p->qpos + j)];
-                               putchar(bam1_strand(p->b)? tolower(c) : toupper(c));
-                       }
-               } else if (p->indel < 0) {
-                       printf("%d", p->indel);
-                       for (j = 1; j <= -p->indel; ++j) {
-                               c = (ref && (int)pos+j < ref_len)? ref[pos+j] : 'N';
-                               putchar(bam1_strand(p->b)? tolower(c) : toupper(c));
-                       }
+       } else putchar(p->is_refskip? (bam1_strand(p->b)? '<' : '>') : '*');
+       if (p->indel > 0) {
+               putchar('+'); printw(p->indel, stdout);
+               for (j = 1; j <= p->indel; ++j) {
+                       int c = bam_nt16_rev_table[bam1_seqi(bam1_seq(p->b), p->qpos + j)];
+                       putchar(bam1_strand(p->b)? tolower(c) : toupper(c));
+               }
+       } else if (p->indel < 0) {
+               printw(p->indel, stdout);
+               for (j = 1; j <= -p->indel; ++j) {
+                       int c = (ref && (int)pos+j < ref_len)? ref[pos+j] : 'N';
+                       putchar(bam1_strand(p->b)? tolower(c) : toupper(c));

                }
                }

-       } else putchar('*');
+       }

        if (p->is_tail) putchar('$');
 }
 
        if (p->is_tail) putchar('$');
 }
 


@@ -232,7 +258,11 @@ static int pileup_func(uint32_t tid, uint32_t pos, int n, const bam_pileup1_t *p
                        }
                        rmsQ = (uint64_t)(sqrt((double)rmsQ / n) + .499);
                        cns = b<<28 | 0xf<<24 | rmsQ<<16 | 60<<8;
                        }
                        rmsQ = (uint64_t)(sqrt((double)rmsQ / n) + .499);
                        cns = b<<28 | 0xf<<24 | rmsQ<<16 | 60<<8;

-               } else cns = bam_maqcns_call(n, pu, d->c);
+               } else {
+                       glf1_t *g = bam_maqcns_glfgen(n, pu, bam_nt16_table[rb], d->c);
+                       cns = g->depth == 0? (0xfu<<28 | 0xf<<24) : glf2cns(g, (int)(d->c->q_r + .499));
+                       free(g);
+               }

        }
     if ((d->format & (BAM_PLF_CNS|BAM_PLF_INDEL_ONLY)) && d->ref && pos < d->len) { // call indels
         int m = (!d->max_depth || d->max_depth>n) ? n : d->max_depth;
        }
     if ((d->format & (BAM_PLF_CNS|BAM_PLF_INDEL_ONLY)) && d->ref && pos < d->len) { // call indels
         int m = (!d->max_depth || d->max_depth>n) ? n : d->max_depth;


@@ -242,7 +272,7 @@ static int pileup_func(uint32_t tid, uint32_t pos, int n, const bam_pileup1_t *p
        }
        // when only variant sites are asked for, test if the site is a variant
        if ((d->format & BAM_PLF_CNS) && (d->format & BAM_PLF_VAR_ONLY)) {
        }
        // when only variant sites are asked for, test if the site is a variant
        if ((d->format & BAM_PLF_CNS) && (d->format & BAM_PLF_VAR_ONLY)) {

-               if (!(bam_nt16_table[rb] != 15 && cns>>28 != bam_nt16_table[rb])) { // not a SNP
+               if (!(bam_nt16_table[rb] != 15 && cns>>28 != 15 && cns>>28 != bam_nt16_table[rb])) { // not a SNP

                        if (!(r && (r->gt == 2 || strcmp(r->s[r->gt], "*")))) { // not an indel
                                if (r) bam_maqindel_ret_destroy(r);
                                return 0;
                        if (!(r && (r->gt == 2 || strcmp(r->s[r->gt], "*")))) { // not an indel
                                if (r) bam_maqindel_ret_destroy(r);
                                return 0;


@@ -250,30 +280,29 @@ static int pileup_func(uint32_t tid, uint32_t pos, int n, const bam_pileup1_t *p
                }
        }
        // print the first 3 columns
                }
        }
        // print the first 3 columns

-       printf("%s\t%d\t%c\t", d->h->target_name[tid], pos + 1, rb);
+       fputs(d->h->target_name[tid], stdout); putchar('\t');
+       printw(pos+1, stdout); putchar('\t'); putchar(rb); putchar('\t');

        // print consensus information if required
        if (d->format & BAM_PLF_CNS) {
        // print consensus information if required
        if (d->format & BAM_PLF_CNS) {

-               int ref_q, rb4 = bam_nt16_table[rb];
-               ref_q = 0;
-               if (rb4 != 15 && cns>>28 != 15 && cns>>28 != rb4) { // a SNP
-                       ref_q = ((cns>>24&0xf) == rb4)? cns>>8&0xff : (cns>>8&0xff) + (cns&0xff);
-                       if (ref_q > 255) ref_q = 255;
-               }
-               rms_mapq = cns>>16&0xff;
-               printf("%c\t%d\t%d\t%d\t", bam_nt16_rev_table[cns>>28], cns>>8&0xff, ref_q, rms_mapq);
+               putchar(bam_nt16_rev_table[cns>>28]); putchar('\t');
+               printw(cns>>8&0xff, stdout); putchar('\t');
+               printw(cns&0xff, stdout); putchar('\t');
+               printw(cns>>16&0xff, stdout); putchar('\t');

        }
        // print pileup sequences
        }
        // print pileup sequences

-       printf("%d\t", n);
-       rms_aux = 0; // we need to recalculate rms_mapq when -c is not flagged on the command line
-       for (i = 0; i < n; ++i) {
-               const bam_pileup1_t *p = pu + i;
-               int tmp = p->b->core.qual < d->c->cap_mapQ? p->b->core.qual : d->c->cap_mapQ;
-               rms_aux += tmp * tmp;
-               pileup_seq(p, pos, d->len, d->ref);
-       }
+       printw(n, stdout); putchar('\t');
+       for (i = 0; i < n; ++i)
+               pileup_seq(pu + i, pos, d->len, d->ref);

        // finalize rms_mapq
        // finalize rms_mapq

-       rms_aux = (uint64_t)(sqrt((double)rms_aux / n) + .499);
-       if (rms_mapq < 0) rms_mapq = rms_aux;
+       if (d->format & BAM_PLF_CNS) {
+               for (i = rms_aux = 0; i < n; ++i) {
+                       const bam_pileup1_t *p = pu + i;
+                       int tmp = p->b->core.qual < d->c->cap_mapQ? p->b->core.qual : d->c->cap_mapQ;
+                       rms_aux += tmp * tmp;
+               }
+               rms_aux = (uint64_t)(sqrt((double)rms_aux / n) + .499);
+               if (rms_mapq < 0) rms_mapq = rms_aux;
+       }

        putchar('\t');
        // print quality
        for (i = 0; i < n; ++i) {
        putchar('\t');
        // print quality
        for (i = 0; i < n; ++i) {


@@ -312,7 +341,7 @@ static int pileup_func(uint32_t tid, uint32_t pos, int n, const bam_pileup1_t *p
                for (i = 0; i < n; ++i) {
                        int x = pu[i].qpos;
                        int l = pu[i].b->core.l_qseq;
                for (i = 0; i < n; ++i) {
                        int x = pu[i].qpos;
                        int l = pu[i].b->core.l_qseq;

-                       printf("%d,", x < l/2? x+1 : -((l-1)-x+1));
+                       printw(x < l/2? x+1 : -((l-1)-x+1), stdout); putchar(',');

                }
        }
        putchar('\n');
                }
        }
        putchar('\n');


@@ -338,15 +367,17 @@ int bam_pileup(int argc, char *argv[])
        int c, is_SAM = 0;
        char *fn_list = 0, *fn_fa = 0, *fn_pos = 0;
        pu_data_t *d = (pu_data_t*)calloc(1, sizeof(pu_data_t));
        int c, is_SAM = 0;
        char *fn_list = 0, *fn_fa = 0, *fn_pos = 0;
        pu_data_t *d = (pu_data_t*)calloc(1, sizeof(pu_data_t));

-    d->max_depth = 0;
-       d->tid = -1; d->mask = BAM_DEF_MASK;
+    d->max_depth = 1024; d->tid = -1; d->mask = BAM_DEF_MASK; d->min_baseQ = 13;

        d->c = bam_maqcns_init();
        d->c = bam_maqcns_init();

-       d->c->is_soap = 1; // change the default model
+       d->c->errmod = BAM_ERRMOD_MAQ2; // change the default model

        d->ido = bam_maqindel_opt_init();
        d->ido = bam_maqindel_opt_init();

-       while ((c = getopt(argc, argv, "st:f:cT:N:r:l:d:im:gI:G:vM:S2aR:PA")) >= 0) {
+       while ((c = getopt(argc, argv, "st:f:cT:N:r:l:d:im:gI:G:vM:S2aR:PAQ:C:B")) >= 0) {

                switch (c) {
                switch (c) {

-               case 'a': d->c->is_soap = 1; break;
-               case 'A': d->c->is_soap = 0; break;
+               case 'Q': d->c->min_baseQ = atoi(optarg); break;
+               case 'C': d->capQ_thres = atoi(optarg); break;
+               case 'B': d->format |= BAM_PLF_NOBAQ; break;
+               case 'a': d->c->errmod = BAM_ERRMOD_SOAP; break;
+               case 'A': d->c->errmod = BAM_ERRMOD_MAQ; break;

                case 's': d->format |= BAM_PLF_SIMPLE; break;
                case 't': fn_list = strdup(optarg); break;
                case 'l': fn_pos = strdup(optarg); break;
                case 's': d->format |= BAM_PLF_SIMPLE; break;
                case 't': fn_list = strdup(optarg); break;
                case 'l': fn_pos = strdup(optarg); break;


@@ -375,29 +406,34 @@ int bam_pileup(int argc, char *argv[])
                default: fprintf(stderr, "Unrecognizd option '-%c'.\n", c); return 1;
                }
        }
                default: fprintf(stderr, "Unrecognizd option '-%c'.\n", c); return 1;
                }
        }

+       if (d->c->errmod != BAM_ERRMOD_MAQ2) d->c->theta += 0.02;
+       if (d->c->theta > 1.0) d->c->theta = 1.0;

        if (fn_list) is_SAM = 1;
        if (optind == argc) {
                fprintf(stderr, "\n");
                fprintf(stderr, "Usage:  samtools pileup [options] <in.bam>|<in.sam>\n\n");
                fprintf(stderr, "Option: -s        simple (yet incomplete) pileup format\n");
                fprintf(stderr, "        -S        the input is in SAM\n");
        if (fn_list) is_SAM = 1;
        if (optind == argc) {
                fprintf(stderr, "\n");
                fprintf(stderr, "Usage:  samtools pileup [options] <in.bam>|<in.sam>\n\n");
                fprintf(stderr, "Option: -s        simple (yet incomplete) pileup format\n");
                fprintf(stderr, "        -S        the input is in SAM\n");

-               fprintf(stderr, "        -A        use the MAQ model for SNP calling\n");
+               fprintf(stderr, "        -B        disable BAQ computation\n");
+               fprintf(stderr, "        -A        use the original MAQ model for SNP calling (DEPRECATED)\n");

                fprintf(stderr, "        -2        output the 2nd best call and quality\n");
                fprintf(stderr, "        -i        only show lines/consensus with indels\n");
                fprintf(stderr, "        -2        output the 2nd best call and quality\n");
                fprintf(stderr, "        -i        only show lines/consensus with indels\n");

-               fprintf(stderr, "        -m INT    filtering reads with bits in INT [%d]\n", d->mask);
+               fprintf(stderr, "        -Q INT    min base quality (possibly capped by BAQ) [%d]\n", d->c->min_baseQ);
+               fprintf(stderr, "        -C INT    coefficient for adjusting mapQ of poor mappings [%d]\n", d->capQ_thres);
+               fprintf(stderr, "        -m INT    filtering reads with bits in INT [0x%x]\n", d->mask);

                fprintf(stderr, "        -M INT    cap mapping quality at INT [%d]\n", d->c->cap_mapQ);
                fprintf(stderr, "        -M INT    cap mapping quality at INT [%d]\n", d->c->cap_mapQ);

-        fprintf(stderr, "        -d INT    limit maximum depth for indels [unlimited]\n");
+        fprintf(stderr, "        -d INT    limit maximum depth for indels [%d]\n", d->max_depth);

                fprintf(stderr, "        -t FILE   list of reference sequences (force -S)\n");
                fprintf(stderr, "        -l FILE   list of sites at which pileup is output\n");
                fprintf(stderr, "        -f FILE   reference sequence in the FASTA format\n\n");
                fprintf(stderr, "        -t FILE   list of reference sequences (force -S)\n");
                fprintf(stderr, "        -l FILE   list of sites at which pileup is output\n");
                fprintf(stderr, "        -f FILE   reference sequence in the FASTA format\n\n");

-               fprintf(stderr, "        -c        output the SOAPsnp consensus sequence\n");
+               fprintf(stderr, "        -c        compute the consensus sequence\n");

                fprintf(stderr, "        -v        print variants only (for -c)\n");
                fprintf(stderr, "        -v        print variants only (for -c)\n");

-               fprintf(stderr, "        -g        output in the GLFv3 format (suppressing -c/-i/-s)\n");
-               fprintf(stderr, "        -T FLOAT  theta in maq consensus calling model (for -c/-g) [%f]\n", d->c->theta);
-               fprintf(stderr, "        -N INT    number of haplotypes in the sample (for -c/-g) [%d]\n", d->c->n_hap);
-               fprintf(stderr, "        -r FLOAT  prior of a difference between two haplotypes (for -c/-g) [%f]\n", d->c->het_rate);
-               fprintf(stderr, "        -G FLOAT  prior of an indel between two haplotypes (for -c/-g) [%f]\n", d->ido->r_indel);
-               fprintf(stderr, "        -I INT    phred prob. of an indel in sequencing/prep. (for -c/-g) [%d]\n", d->ido->q_indel);
+               fprintf(stderr, "        -g        output in the GLFv3 format (DEPRECATED)\n");
+               fprintf(stderr, "        -T FLOAT  theta in maq consensus calling model (for -c) [%.4g]\n", d->c->theta);
+               fprintf(stderr, "        -N INT    number of haplotypes in the sample (for -c) [%d]\n", d->c->n_hap);
+               fprintf(stderr, "        -r FLOAT  prior of a difference between two haplotypes (for -c) [%.4g]\n", d->c->het_rate);
+               fprintf(stderr, "        -G FLOAT  prior of an indel between two haplotypes (for -c) [%.4g]\n", d->ido->r_indel);
+               fprintf(stderr, "        -I INT    phred prob. of an indel in sequencing/prep. (for -c) [%d]\n", d->ido->q_indel);

                fprintf(stderr, "\n");
                free(fn_list); free(fn_fa); free(d);
                return 1;
                fprintf(stderr, "\n");
                free(fn_list); free(fn_fa); free(d);
                return 1;


@@ -425,7 +461,43 @@ int bam_pileup(int argc, char *argv[])
                }
                d->h = fp->header;
                if (fn_pos) d->hash = load_pos(fn_pos, d->h);
                }
                d->h = fp->header;
                if (fn_pos) d->hash = load_pos(fn_pos, d->h);

-               sampileup(fp, d->mask, pileup_func, d);
+               { // run pileup
+                       extern int bam_prob_realn(bam1_t *b, const char *ref);
+                       extern int bam_cap_mapQ(bam1_t *b, char *ref, int thres);
+                       bam1_t *b;
+                       int ret, tid, pos, n_plp;
+                       bam_plp_t iter;
+                       const bam_pileup1_t *plp;
+                       b = bam_init1();
+                       iter = bam_plp_init(0, 0);
+                       bam_plp_set_mask(iter, d->mask);
+                       while ((ret = samread(fp, b)) >= 0) {
+                               int skip = 0;
+                               if ((int)b->core.tid < 0) break;
+                               // update d->ref if necessary
+                               if (d->fai && (int)b->core.tid != d->tid) {
+                                       free(d->ref);
+                                       d->ref = faidx_fetch_seq(d->fai, d->h->target_name[b->core.tid], 0, 0x7fffffff, &d->len);
+                                       d->tid = b->core.tid;
+                               }
+                               if (d->ref && (d->format&BAM_PLF_CNS) && !(d->format&BAM_PLF_NOBAQ)) bam_prob_realn(b, d->ref);
+                               if (d->ref && (d->format&BAM_PLF_CNS) && d->capQ_thres > 10) {
+                                       int q = bam_cap_mapQ(b, d->ref, d->capQ_thres);
+                                       if (q < 0) skip = 1;
+                                       else if (b->core.qual > q) b->core.qual = q;
+                               } else if (b->core.flag&BAM_FUNMAP) skip = 1;
+                               else if ((d->format&BAM_PLF_CNS) && (b->core.flag&1) && !(b->core.flag&2)) skip = 1;
+                               if (skip) continue;
+                               bam_plp_push(iter, b);
+                               while ((plp = bam_plp_next(iter, &tid, &pos, &n_plp)) != 0)
+                                       pileup_func(tid, pos, n_plp, plp, d);
+                       }
+                       bam_plp_push(iter, 0);
+                       while ((plp = bam_plp_next(iter, &tid, &pos, &n_plp)) != 0)
+                               pileup_func(tid, pos, n_plp, plp, d);
+                       bam_plp_destroy(iter);
+                       bam_destroy1(b);
+               }

                samclose(fp); // d->h will be destroyed here
        }
 
                samclose(fp); // d->h will be destroyed here
        }
 


@@ -448,21 +520,80 @@ int bam_pileup(int argc, char *argv[])
  * mpileup *
  ***********/
 
  * mpileup *
  ***********/
 

+#include <assert.h>
+#include "bam2bcf.h"
+#include "sample.h"
+
+#define MPLP_GLF   0x10
+#define MPLP_NO_COMP 0x20
+#define MPLP_NO_ORPHAN 0x40
+#define MPLP_REALN   0x80
+

 typedef struct {
 typedef struct {

-       char *reg;
+       int max_mq, min_mq, flag, min_baseQ, capQ_thres;
+       char *reg, *fn_pos;

        faidx_t *fai;
        faidx_t *fai;

+       kh_64_t *hash;

 } mplp_conf_t;
 
 typedef struct {
        bamFile fp;
        bam_iter_t iter;
 } mplp_conf_t;
 
 typedef struct {
        bamFile fp;
        bam_iter_t iter;

+       int min_mq, flag, ref_id, capQ_thres;
+       char *ref;

 } mplp_aux_t;
 
 } mplp_aux_t;
 

+typedef struct {
+       int n;
+       int *n_plp, *m_plp;
+       bam_pileup1_t **plp;
+} mplp_pileup_t;
+

 static int mplp_func(void *data, bam1_t *b)
 {
 static int mplp_func(void *data, bam1_t *b)
 {

+       extern int bam_realn(bam1_t *b, const char *ref);
+       extern int bam_prob_realn(bam1_t *b, const char *ref);
+       extern int bam_cap_mapQ(bam1_t *b, char *ref, int thres);

        mplp_aux_t *ma = (mplp_aux_t*)data;
        mplp_aux_t *ma = (mplp_aux_t*)data;

-       if (ma->iter) return bam_iter_read(ma->fp, ma->iter, b);
-       return bam_read1(ma->fp, b);
+       int ret, skip = 0;
+       do {
+               int has_ref = (ma->ref && ma->ref_id == b->core.tid)? 1 : 0;
+               ret = ma->iter? bam_iter_read(ma->fp, ma->iter, b) : bam_read1(ma->fp, b);
+               if (ret < 0) break;
+               skip = 0;
+               if (has_ref && (ma->flag&MPLP_REALN)) bam_prob_realn(b, ma->ref);
+               if (has_ref && ma->capQ_thres > 10) {
+                       int q = bam_cap_mapQ(b, ma->ref, ma->capQ_thres);
+                       if (q < 0) skip = 1;
+                       else if (b->core.qual > q) b->core.qual = q;
+               } else if (b->core.flag&BAM_FUNMAP) skip = 1;
+               else if (b->core.qual < ma->min_mq) skip = 1; 
+               else if ((ma->flag&MPLP_NO_ORPHAN) && (b->core.flag&1) && !(b->core.flag&2)) skip = 1;
+       } while (skip);
+       return ret;
+}
+
+static void group_smpl(mplp_pileup_t *m, bam_sample_t *sm, kstring_t *buf,
+                                          int n, char *const*fn, int *n_plp, const bam_pileup1_t **plp)
+{
+       int i, j;
+       memset(m->n_plp, 0, m->n * sizeof(int));
+       for (i = 0; i < n; ++i) {
+               for (j = 0; j < n_plp[i]; ++j) {
+                       const bam_pileup1_t *p = plp[i] + j;
+                       uint8_t *q;
+                       int id = -1;
+                       q = bam_aux_get(p->b, "RG");
+                       if (q) id = bam_smpl_rg2smid(sm, fn[i], (char*)q+1, buf);
+                       if (id < 0) id = bam_smpl_rg2smid(sm, fn[i], 0, buf);
+                       assert(id >= 0 && id < m->n);
+                       if (m->n_plp[id] == m->m_plp[id]) {
+                               m->m_plp[id] = m->m_plp[id]? m->m_plp[id]<<1 : 8;
+                               m->plp[id] = realloc(m->plp[id], sizeof(bam_pileup1_t) * m->m_plp[id]);
+                       }
+                       m->plp[id][m->n_plp[id]++] = *p;
+               }
+       }

 }
 
 static int mpileup(mplp_conf_t *conf, int n, char **fn)
 }
 
 static int mpileup(mplp_conf_t *conf, int n, char **fn)


@@ -473,16 +604,36 @@ static int mpileup(mplp_conf_t *conf, int n, char **fn)
        bam_mplp_t iter;
        bam_header_t *h = 0;
        char *ref;
        bam_mplp_t iter;
        bam_header_t *h = 0;
        char *ref;

-       // allocate
+       khash_t(64) *hash = 0;
+
+       bcf_callaux_t *bca = 0;
+       bcf_callret1_t *bcr = 0;
+       bcf_call_t bc;
+       bcf_t *bp = 0;
+       bcf_hdr_t *bh = 0;
+
+       bam_sample_t *sm = 0;
+       kstring_t buf;
+       mplp_pileup_t gplp;
+
+       memset(&gplp, 0, sizeof(mplp_pileup_t));
+       memset(&buf, 0, sizeof(kstring_t));
+       memset(&bc, 0, sizeof(bcf_call_t));

        data = calloc(n, sizeof(void*));
        plp = calloc(n, sizeof(void*));
        n_plp = calloc(n, sizeof(int*));
        data = calloc(n, sizeof(void*));
        plp = calloc(n, sizeof(void*));
        n_plp = calloc(n, sizeof(int*));

+       sm = bam_smpl_init();
+

        // read the header and initialize data
        for (i = 0; i < n; ++i) {
                bam_header_t *h_tmp;
                data[i] = calloc(1, sizeof(mplp_aux_t));
        // read the header and initialize data
        for (i = 0; i < n; ++i) {
                bam_header_t *h_tmp;
                data[i] = calloc(1, sizeof(mplp_aux_t));

-               data[i]->fp = bam_open(fn[i], "r");
+               data[i]->min_mq = conf->min_mq;
+               data[i]->flag = conf->flag;
+               data[i]->capQ_thres = conf->capQ_thres;
+               data[i]->fp = strcmp(fn[i], "-") == 0? bam_dopen(fileno(stdin), "r") : bam_open(fn[i], "r");

                h_tmp = bam_header_read(data[i]->fp);
                h_tmp = bam_header_read(data[i]->fp);

+               bam_smpl_add(sm, fn[i], h_tmp->text);

                if (conf->reg) {
                        int beg, end;
                        bam_index_t *idx;
                if (conf->reg) {
                        int beg, end;
                        bam_index_t *idx;


@@ -505,35 +656,100 @@ static int mpileup(mplp_conf_t *conf, int n, char **fn)
                        bam_header_destroy(h_tmp);
                }
        }
                        bam_header_destroy(h_tmp);
                }
        }

-       // mpileup
+       gplp.n = sm->n;
+       gplp.n_plp = calloc(sm->n, sizeof(int));
+       gplp.m_plp = calloc(sm->n, sizeof(int));
+       gplp.plp = calloc(sm->n, sizeof(void*));
+
+       fprintf(stderr, "[%s] %d samples in %d input files\n", __func__, sm->n, n);
+       if (conf->fn_pos) hash = load_pos(conf->fn_pos, h);
+       // write the VCF header
+       if (conf->flag & MPLP_GLF) {
+               kstring_t s;
+               bh = calloc(1, sizeof(bcf_hdr_t));
+               s.l = s.m = 0; s.s = 0;
+               bp = bcf_open("-", (conf->flag&MPLP_NO_COMP)? "wu" : "w");
+               for (i = 0; i < h->n_targets; ++i) {
+                       kputs(h->target_name[i], &s);
+                       kputc('\0', &s);
+               }
+               bh->l_nm = s.l;
+               bh->name = malloc(s.l);
+               memcpy(bh->name, s.s, s.l);
+               s.l = 0;
+               for (i = 0; i < sm->n; ++i) {
+                       kputs(sm->smpl[i], &s); kputc('\0', &s);
+               }
+               bh->l_smpl = s.l;
+               bh->sname = malloc(s.l);
+               memcpy(bh->sname, s.s, s.l);
+               bh->l_txt = 0;
+               free(s.s);
+               bcf_hdr_sync(bh);
+               bcf_hdr_write(bp, bh);
+               bca = bcf_call_init(-1., conf->min_baseQ);
+               bcr = calloc(sm->n, sizeof(bcf_callret1_t));
+       }

        ref_tid = -1; ref = 0;
        iter = bam_mplp_init(n, mplp_func, (void**)data);
        while (bam_mplp_auto(iter, &tid, &pos, n_plp, plp) > 0) {
                if (conf->reg && (pos < beg0 || pos >= end0)) continue; // out of the region requested
        ref_tid = -1; ref = 0;
        iter = bam_mplp_init(n, mplp_func, (void**)data);
        while (bam_mplp_auto(iter, &tid, &pos, n_plp, plp) > 0) {
                if (conf->reg && (pos < beg0 || pos >= end0)) continue; // out of the region requested

+               if (hash) {
+                       khint_t k;
+                       k = kh_get(64, hash, (uint64_t)tid<<32 | pos);
+                       if (k == kh_end(hash)) continue;
+               }

                if (tid != ref_tid) {
                if (tid != ref_tid) {

-                       free(ref);
+                       free(ref); ref = 0;

                        if (conf->fai) ref = fai_fetch(conf->fai, h->target_name[tid], &ref_len);
                        if (conf->fai) ref = fai_fetch(conf->fai, h->target_name[tid], &ref_len);

+                       for (i = 0; i < n; ++i) data[i]->ref = ref, data[i]->ref_id = tid;

                        ref_tid = tid;
                }
                        ref_tid = tid;
                }

-               printf("%s\t%d\t%c", h->target_name[tid], pos + 1, (ref && pos < ref_len)? ref[pos] : 'N');
-               for (i = 0; i < n; ++i) {
-                       int j;
-                       printf("\t%d\t", n_plp[i]);
-                       if (n_plp[i] == 0) printf("*\t*");
-                       else {
-                               for (j = 0; j < n_plp[i]; ++j)
-                                       pileup_seq(plp[i] + j, pos, ref_len, ref);
-                               putchar('\t');
-                               for (j = 0; j < n_plp[i]; ++j) {
-                                       const bam_pileup1_t *p = plp[i] + j;
-                                       int c = bam1_qual(p->b)[p->qpos] + 33;
-                                       if (c > 126) c = 126;
-                                       putchar(c);
+               if (conf->flag & MPLP_GLF) {
+                       int _ref0, ref16;
+                       bcf1_t *b = calloc(1, sizeof(bcf1_t));
+                       group_smpl(&gplp, sm, &buf, n, fn, n_plp, plp);
+                       _ref0 = (ref && pos < ref_len)? ref[pos] : 'N';
+                       ref16 = bam_nt16_table[_ref0];
+                       for (i = 0; i < gplp.n; ++i)
+                               bcf_call_glfgen(gplp.n_plp[i], gplp.plp[i], ref16, bca, bcr + i);
+                       bcf_call_combine(gplp.n, bcr, ref16, &bc);
+                       bcf_call2bcf(tid, pos, &bc, b);
+                       bcf_write(bp, bh, b);
+                       bcf_destroy(b);
+               } else {
+                       printf("%s\t%d\t%c", h->target_name[tid], pos + 1, (ref && pos < ref_len)? ref[pos] : 'N');
+                       for (i = 0; i < n; ++i) {
+                               int j;
+                               printf("\t%d\t", n_plp[i]);
+                               if (n_plp[i] == 0) printf("*\t*");
+                               else {
+                                       for (j = 0; j < n_plp[i]; ++j)
+                                               pileup_seq(plp[i] + j, pos, ref_len, ref);
+                                       putchar('\t');
+                                       for (j = 0; j < n_plp[i]; ++j) {
+                                               const bam_pileup1_t *p = plp[i] + j;
+                                               int c = bam1_qual(p->b)[p->qpos] + 33;
+                                               if (c > 126) c = 126;
+                                               putchar(c);
+                                       }

                                }
                        }
                                }
                        }

+                       putchar('\n');

                }
                }

-               putchar('\n');

        }
        }

+
+       bcf_close(bp);
+       bam_smpl_destroy(sm); free(buf.s);
+       for (i = 0; i < gplp.n; ++i) free(gplp.plp[i]);
+       free(gplp.plp); free(gplp.n_plp); free(gplp.m_plp);
+       if (hash) { // free the hash table
+               khint_t k;
+               for (k = kh_begin(hash); k < kh_end(hash); ++k)
+                       if (kh_exist(hash, k)) free(kh_val(hash, k));
+               kh_destroy(64, hash);
+       }
+       bcf_hdr_destroy(bh); bcf_call_destroy(bca); free(bc.PL); free(bcr);

        bam_mplp_destroy(iter);
        bam_header_destroy(h);
        for (i = 0; i < n; ++i) {
        bam_mplp_destroy(iter);
        bam_header_destroy(h);
        for (i = 0; i < n; ++i) {


@@ -550,17 +766,44 @@ int bam_mpileup(int argc, char *argv[])
        int c;
        mplp_conf_t mplp;
        memset(&mplp, 0, sizeof(mplp_conf_t));
        int c;
        mplp_conf_t mplp;
        memset(&mplp, 0, sizeof(mplp_conf_t));

-       while ((c = getopt(argc, argv, "f:r:")) >= 0) {
+       mplp.max_mq = 60;
+       mplp.min_baseQ = 13;
+       mplp.capQ_thres = 0;
+       mplp.flag = MPLP_NO_ORPHAN | MPLP_REALN;
+       while ((c = getopt(argc, argv, "gf:r:l:M:q:Q:uaORC:B")) >= 0) {

                switch (c) {
                case 'f':
                        mplp.fai = fai_load(optarg);
                        if (mplp.fai == 0) return 1;
                        break;
                switch (c) {
                case 'f':
                        mplp.fai = fai_load(optarg);
                        if (mplp.fai == 0) return 1;
                        break;

-               case 'r': mplp.reg = strdup(optarg);
+               case 'r': mplp.reg = strdup(optarg); break;
+               case 'l': mplp.fn_pos = strdup(optarg); break;
+               case 'g': mplp.flag |= MPLP_GLF; break;
+               case 'u': mplp.flag |= MPLP_NO_COMP | MPLP_GLF; break;
+               case 'a': mplp.flag |= MPLP_NO_ORPHAN | MPLP_REALN; break;
+               case 'B': mplp.flag &= ~MPLP_REALN & ~MPLP_NO_ORPHAN; break;
+               case 'O': mplp.flag |= MPLP_NO_ORPHAN; break;
+               case 'R': mplp.flag |= MPLP_REALN; break;
+               case 'C': mplp.capQ_thres = atoi(optarg); break;
+               case 'M': mplp.max_mq = atoi(optarg); break;
+               case 'q': mplp.min_mq = atoi(optarg); break;
+               case 'Q': mplp.min_baseQ = atoi(optarg); break;

                }
        }
        if (argc == 1) {
                }
        }
        if (argc == 1) {

-               fprintf(stderr, "Usage: samtools mpileup [-r reg] [-f in.fa] in1.bam [in2.bam [...]]\n");
+               fprintf(stderr, "\n");
+               fprintf(stderr, "Usage:   samtools mpileup [options] in1.bam [in2.bam [...]]\n\n");
+               fprintf(stderr, "Options: -f FILE     reference sequence file [null]\n");
+               fprintf(stderr, "         -r STR      region in which pileup is generated [null]\n");
+               fprintf(stderr, "         -l FILE     list of positions (format: chr pos) [null]\n");
+               fprintf(stderr, "         -M INT      cap mapping quality at INT [%d]\n", mplp.max_mq);
+               fprintf(stderr, "         -Q INT      min base quality [%d]\n", mplp.min_baseQ);
+               fprintf(stderr, "         -q INT      filter out alignment with MQ smaller than INT [%d]\n", mplp.min_mq);
+               fprintf(stderr, "         -g          generate BCF output\n");
+               fprintf(stderr, "         -u          do not compress BCF output\n");
+               fprintf(stderr, "         -B          disable BAQ computation\n");
+               fprintf(stderr, "\n");
+               fprintf(stderr, "Notes: Assuming diploid individuals.\n\n");

                return 1;
        }
        mpileup(&mplp, argc - optind, argv + optind);
                return 1;
        }
        mpileup(&mplp, argc - optind, argv + optind);

diff --git a/bam_sort.c b/bam_sort.c
index 12b1b5455280e8d8a788a82929844083a4efd084..76ab793196ca76fc5a63d619e230cf0fb5f59f1e 100644 (file)
--- a/bam_sort.c
+++ b/bam_sort.c
@@ -1,6 +1,7 @@
 #include <stdlib.h>
 #include <ctype.h>
 #include <assert.h>
 #include <stdlib.h>
 #include <ctype.h>
 #include <assert.h>

+#include <errno.h>

 #include <stdio.h>
 #include <string.h>
 #include <unistd.h>
 #include <stdio.h>
 #include <string.h>
 #include <unistd.h>


@@ -59,6 +60,9 @@ static void swap_header_text(bam_header_t *h1, bam_header_t *h2)
        temps = h1->text, h1->text = h2->text, h2->text = temps;
 }
 
        temps = h1->text, h1->text = h2->text, h2->text = temps;
 }
 

+#define MERGE_RG     1
+#define MERGE_UNCOMP 2
+

 /*!
   @abstract    Merge multiple sorted BAM.
   @param  is_by_qname whether to sort by query name
 /*!
   @abstract    Merge multiple sorted BAM.
   @param  is_by_qname whether to sort by query name


@@ -71,7 +75,8 @@ static void swap_header_text(bam_header_t *h1, bam_header_t *h2)
   @discussion Padding information may NOT correctly maintained. This
   function is NOT thread safe.
  */
   @discussion Padding information may NOT correctly maintained. This
   function is NOT thread safe.
  */

-void bam_merge_core(int by_qname, const char *out, const char *headers, int n, char * const *fn, int add_RG)
+int bam_merge_core(int by_qname, const char *out, const char *headers, int n, char * const *fn,
+                                       int flag, const char *reg)

 {
        bamFile fpout, *fp;
        heap1_t *heap;
 {
        bamFile fpout, *fp;
        heap1_t *heap;


@@ -80,22 +85,25 @@ void bam_merge_core(int by_qname, const char *out, const char *headers, int n, c
        int i, j, *RG_len = 0;
        uint64_t idx = 0;
        char **RG = 0;
        int i, j, *RG_len = 0;
        uint64_t idx = 0;
        char **RG = 0;

+       bam_iter_t *iter = 0;

 
        if (headers) {
                tamFile fpheaders = sam_open(headers);
                if (fpheaders == 0) {
 
        if (headers) {
                tamFile fpheaders = sam_open(headers);
                if (fpheaders == 0) {

-                       fprintf(stderr, "[bam_merge_core] Cannot open file `%s'. Continue anyway.\n", headers);
-               } else {
-                       hheaders = sam_header_read(fpheaders);
-                       sam_close(fpheaders);
+                       const char *message = strerror(errno);
+                       fprintf(stderr, "[bam_merge_core] cannot open '%s': %s\n", headers, message);
+                       return -1;

                }
                }

+               hheaders = sam_header_read(fpheaders);
+               sam_close(fpheaders);

        }
 
        g_is_by_qname = by_qname;
        fp = (bamFile*)calloc(n, sizeof(bamFile));
        heap = (heap1_t*)calloc(n, sizeof(heap1_t));
        }
 
        g_is_by_qname = by_qname;
        fp = (bamFile*)calloc(n, sizeof(bamFile));
        heap = (heap1_t*)calloc(n, sizeof(heap1_t));

+       iter = (bam_iter_t*)calloc(n, sizeof(bam_iter_t));

        // prepare RG tag
        // prepare RG tag

-       if (add_RG) {
+       if (flag & MERGE_RG) {

                RG = (char**)calloc(n, sizeof(void*));
                RG_len = (int*)calloc(n, sizeof(int));
                for (i = 0; i != n; ++i) {
                RG = (char**)calloc(n, sizeof(void*));
                RG_len = (int*)calloc(n, sizeof(int));
                for (i = 0; i != n; ++i) {


@@ -111,7 +119,6 @@ void bam_merge_core(int by_qname, const char *out, const char *headers, int n, c
        }
        // read the first
        for (i = 0; i != n; ++i) {
        }
        // read the first
        for (i = 0; i != n; ++i) {

-               heap1_t *h;

                bam_header_t *hin;
                fp[i] = bam_open(fn[i], "r");
                if (fp[i] == 0) {
                bam_header_t *hin;
                fp[i] = bam_open(fn[i], "r");
                if (fp[i] == 0) {


@@ -120,7 +127,7 @@ void bam_merge_core(int by_qname, const char *out, const char *headers, int n, c
                        for (j = 0; j < i; ++j) bam_close(fp[j]);
                        free(fp); free(heap);
                        // FIXME: possible memory leak
                        for (j = 0; j < i; ++j) bam_close(fp[j]);
                        free(fp); free(heap);
                        // FIXME: possible memory leak

-                       return;
+                       return -1;

                }
                hin = bam_header_read(fp[i]);
                if (i == 0) { // the first SAM
                }
                hin = bam_header_read(fp[i]);
                if (i == 0) { // the first SAM


@@ -130,11 +137,15 @@ void bam_merge_core(int by_qname, const char *out, const char *headers, int n, c
                                // check that they are consistent with the existing binary list
                                // of reference information.
                                if (hheaders->n_targets > 0) {
                                // check that they are consistent with the existing binary list
                                // of reference information.
                                if (hheaders->n_targets > 0) {

-                                       if (hout->n_targets != hheaders->n_targets)
+                                       if (hout->n_targets != hheaders->n_targets) {

                                                fprintf(stderr, "[bam_merge_core] number of @SQ headers in `%s' differs from number of target sequences", headers);
                                                fprintf(stderr, "[bam_merge_core] number of @SQ headers in `%s' differs from number of target sequences", headers);

+                                               if (!reg) return -1;
+                                       }

                                        for (j = 0; j < hout->n_targets; ++j)
                                        for (j = 0; j < hout->n_targets; ++j)

-                                               if (strcmp(hout->target_name[j], hheaders->target_name[j]) != 0)
+                                               if (strcmp(hout->target_name[j], hheaders->target_name[j]) != 0) {

                                                        fprintf(stderr, "[bam_merge_core] @SQ header '%s' in '%s' differs from target sequence", hheaders->target_name[j], headers);
                                                        fprintf(stderr, "[bam_merge_core] @SQ header '%s' in '%s' differs from target sequence", hheaders->target_name[j], headers);

+                                                       if (!reg) return -1;
+                                               }

                                }
                                swap_header_text(hout, hheaders);
                                bam_header_destroy(hheaders);
                                }
                                swap_header_text(hout, hheaders);
                                bam_header_destroy(hheaders);


@@ -142,39 +153,62 @@ void bam_merge_core(int by_qname, const char *out, const char *headers, int n, c
                        }
                } else { // validate multiple baf
                        if (hout->n_targets != hin->n_targets) {
                        }
                } else { // validate multiple baf
                        if (hout->n_targets != hin->n_targets) {

-                               fprintf(stderr, "[bam_merge_core] file '%s' has different number of target sequences. Abort!\n", fn[i]);
-                               exit(1);
-                       }
-                       for (j = 0; j < hout->n_targets; ++j) {
-                               if (strcmp(hout->target_name[j], hin->target_name[j])) {
-                                       fprintf(stderr, "[bam_merge_core] different target sequence name: '%s' != '%s' in file '%s'. Abort!\n",
-                                                       hout->target_name[j], hin->target_name[j], fn[i]);
-                                       exit(1);
+                               fprintf(stderr, "[bam_merge_core] file '%s' has different number of target sequences. Continue anyway!\n", fn[i]);
+                       } else {
+                               for (j = 0; j < hout->n_targets; ++j) {
+                                       if (strcmp(hout->target_name[j], hin->target_name[j])) {
+                                               fprintf(stderr, "[bam_merge_core] different target sequence name: '%s' != '%s' in file '%s'. Continue anyway!\n",
+                                                               hout->target_name[j], hin->target_name[j], fn[i]);
+                                       }

                                }
                        }
                        bam_header_destroy(hin);
                }
                                }
                        }
                        bam_header_destroy(hin);
                }

-               h = heap + i;
+       }
+
+       if (reg) {
+               int tid, beg, end;
+               if (bam_parse_region(hout, reg, &tid, &beg, &end) < 0) {
+                       fprintf(stderr, "[%s] Malformated region string or undefined reference name\n", __func__);
+                       return -1;
+               }
+               for (i = 0; i < n; ++i) {
+                       bam_index_t *idx;
+                       idx = bam_index_load(fn[i]);
+                       iter[i] = bam_iter_query(idx, tid, beg, end);
+                       bam_index_destroy(idx);
+               }
+       }
+
+       for (i = 0; i < n; ++i) {
+               heap1_t *h = heap + i;

                h->i = i;
                h->b = (bam1_t*)calloc(1, sizeof(bam1_t));
                h->i = i;
                h->b = (bam1_t*)calloc(1, sizeof(bam1_t));

-               if (bam_read1(fp[i], h->b) >= 0) {
+               if (bam_iter_read(fp[i], iter[i], h->b) >= 0) {

                        h->pos = ((uint64_t)h->b->core.tid<<32) | (uint32_t)h->b->core.pos<<1 | bam1_strand(h->b);
                        h->idx = idx++;
                }
                else h->pos = HEAP_EMPTY;
        }
                        h->pos = ((uint64_t)h->b->core.tid<<32) | (uint32_t)h->b->core.pos<<1 | bam1_strand(h->b);
                        h->idx = idx++;
                }
                else h->pos = HEAP_EMPTY;
        }

-       fpout = strcmp(out, "-")? bam_open(out, "w") : bam_dopen(fileno(stdout), "w");
-       assert(fpout);
+       if (flag & MERGE_UNCOMP) {
+               fpout = strcmp(out, "-")? bam_open(out, "wu") : bam_dopen(fileno(stdout), "wu");
+       } else {
+               fpout = strcmp(out, "-")? bam_open(out, "w") : bam_dopen(fileno(stdout), "w");
+       }
+       if (fpout == 0) {
+               fprintf(stderr, "[%s] fail to create the output file.\n", __func__);
+               return -1;
+       }

        bam_header_write(fpout, hout);
        bam_header_destroy(hout);
 
        ks_heapmake(heap, n, heap);
        while (heap->pos != HEAP_EMPTY) {
                bam1_t *b = heap->b;
        bam_header_write(fpout, hout);
        bam_header_destroy(hout);
 
        ks_heapmake(heap, n, heap);
        while (heap->pos != HEAP_EMPTY) {
                bam1_t *b = heap->b;

-               if (add_RG && bam_aux_get(b, "RG") == 0)
+               if ((flag & MERGE_RG) && bam_aux_get(b, "RG") == 0)

                        bam_aux_append(b, "RG", 'Z', RG_len[heap->i] + 1, (uint8_t*)RG[heap->i]);
                bam_write1_core(fpout, &b->core, b->data_len, b->data);
                        bam_aux_append(b, "RG", 'Z', RG_len[heap->i] + 1, (uint8_t*)RG[heap->i]);
                bam_write1_core(fpout, &b->core, b->data_len, b->data);

-               if ((j = bam_read1(fp[heap->i], b)) >= 0) {
+               if ((j = bam_iter_read(fp[heap->i], iter[heap->i], b)) >= 0) {

                        heap->pos = ((uint64_t)b->core.tid<<32) | (uint32_t)b->core.pos<<1 | bam1_strand(b);
                        heap->idx = idx++;
                } else if (j == -1) {
                        heap->pos = ((uint64_t)b->core.tid<<32) | (uint32_t)b->core.pos<<1 | bam1_strand(b);
                        heap->idx = idx++;
                } else if (j == -1) {


@@ -185,24 +219,31 @@ void bam_merge_core(int by_qname, const char *out, const char *headers, int n, c
                ks_heapadjust(heap, 0, n, heap);
        }
 
                ks_heapadjust(heap, 0, n, heap);
        }
 

-       if (add_RG) {
+       if (flag & MERGE_RG) {

                for (i = 0; i != n; ++i) free(RG[i]);
                free(RG); free(RG_len);
        }
                for (i = 0; i != n; ++i) free(RG[i]);
                free(RG); free(RG_len);
        }

-       for (i = 0; i != n; ++i) bam_close(fp[i]);
+       for (i = 0; i != n; ++i) {
+               bam_iter_destroy(iter[i]);
+               bam_close(fp[i]);
+       }

        bam_close(fpout);
        bam_close(fpout);

-       free(fp); free(heap);
+       free(fp); free(heap); free(iter);
+       return 0;

 }
 }

+

 int bam_merge(int argc, char *argv[])
 {
 int bam_merge(int argc, char *argv[])
 {

-       int c, is_by_qname = 0, add_RG = 0;
-       char *fn_headers = NULL;
+       int c, is_by_qname = 0, flag = 0, ret = 0;
+       char *fn_headers = NULL, *reg = 0;

 
 

-       while ((c = getopt(argc, argv, "h:nr")) >= 0) {
+       while ((c = getopt(argc, argv, "h:nruR:")) >= 0) {

                switch (c) {
                switch (c) {

-               case 'r': add_RG = 1; break;
+               case 'r': flag |= MERGE_RG; break;

                case 'h': fn_headers = strdup(optarg); break;
                case 'n': is_by_qname = 1; break;
                case 'h': fn_headers = strdup(optarg); break;
                case 'n': is_by_qname = 1; break;

+               case 'u': flag |= MERGE_UNCOMP; break;
+               case 'R': reg = strdup(optarg); break;

                }
        }
        if (optind + 2 >= argc) {
                }
        }
        if (optind + 2 >= argc) {


@@ -210,15 +251,18 @@ int bam_merge(int argc, char *argv[])
                fprintf(stderr, "Usage:   samtools merge [-nr] [-h inh.sam] <out.bam> <in1.bam> <in2.bam> [...]\n\n");
                fprintf(stderr, "Options: -n       sort by read names\n");
                fprintf(stderr, "         -r       attach RG tag (inferred from file names)\n");
                fprintf(stderr, "Usage:   samtools merge [-nr] [-h inh.sam] <out.bam> <in1.bam> <in2.bam> [...]\n\n");
                fprintf(stderr, "Options: -n       sort by read names\n");
                fprintf(stderr, "         -r       attach RG tag (inferred from file names)\n");

+               fprintf(stderr, "         -u       uncompressed BAM output\n");
+               fprintf(stderr, "         -R STR   merge file in the specified region STR [all]\n");

                fprintf(stderr, "         -h FILE  copy the header in FILE to <out.bam> [in1.bam]\n\n");
                fprintf(stderr, "Note: Samtools' merge does not reconstruct the @RG dictionary in the header. Users\n");
                fprintf(stderr, "      must provide the correct header with -h, or uses Picard which properly maintains\n");
                fprintf(stderr, "      the header dictionary in merging.\n\n");
                return 1;
        }
                fprintf(stderr, "         -h FILE  copy the header in FILE to <out.bam> [in1.bam]\n\n");
                fprintf(stderr, "Note: Samtools' merge does not reconstruct the @RG dictionary in the header. Users\n");
                fprintf(stderr, "      must provide the correct header with -h, or uses Picard which properly maintains\n");
                fprintf(stderr, "      the header dictionary in merging.\n\n");
                return 1;
        }

-       bam_merge_core(is_by_qname, argv[optind], fn_headers, argc - optind - 1, argv + optind + 1, add_RG);
+       if (bam_merge_core(is_by_qname, argv[optind], fn_headers, argc - optind - 1, argv + optind + 1, flag, reg) < 0) ret = 1;
+       free(reg);

        free(fn_headers);
        free(fn_headers);

-       return 0;
+       return ret;

 }
 
 typedef bam1_t *bam1_p;
 }
 
 typedef bam1_t *bam1_p;


@@ -313,7 +357,7 @@ void bam_sort_core_ext(int is_by_qname, const char *fn, const char *prefix, size
                        fns[i] = (char*)calloc(strlen(prefix) + 20, 1);
                        sprintf(fns[i], "%s.%.4d.bam", prefix, i);
                }
                        fns[i] = (char*)calloc(strlen(prefix) + 20, 1);
                        sprintf(fns[i], "%s.%.4d.bam", prefix, i);
                }

-               bam_merge_core(is_by_qname, fnout, 0, n, fns, 0);
+               bam_merge_core(is_by_qname, fnout, 0, n, fns, 0, 0);

                free(fnout);
                for (i = 0; i < n; ++i) {
                        unlink(fns[i]);
                free(fnout);
                for (i = 0; i < n; ++i) {
                        unlink(fns[i]);

diff --git a/bam_tview.c b/bam_tview.c
index 7b326fc40e7bf276b7adc3848c0667d5628a68ad..e48afa7bd4add26404fecc04d6548e7b3989927b 100644 (file)
--- a/bam_tview.c
+++ b/bam_tview.c
@@ -109,7 +109,7 @@ int tv_pl_func(uint32_t tid, uint32_t pos, int n, const bam_pileup1_t *pl, void
                                                        if (tv->is_dot && toupper(c) == toupper(rb)) c = bam1_strand(p->b)? ',' : '.';
                                                }
                                        }
                                                        if (tv->is_dot && toupper(c) == toupper(rb)) c = bam1_strand(p->b)? ',' : '.';
                                                }
                                        }

-                               } else c = '*';
+                               } else c = p->is_refskip? (bam1_strand(p->b)? '<' : '>') : '*';

                        } else { // padding
                                if (j > p->indel) c = '*';
                                else { // insertion
                        } else { // padding
                                if (j > p->indel) c = '*';
                                else { // insertion


@@ -292,7 +292,7 @@ static void tv_win_goto(tview_t *tv, int *tid, int *pos)
                } else if (c == KEY_ENTER || c == '\012' || c == '\015') {
                        int _tid = -1, _beg, _end;
                        if (str[0] == '=') {
                } else if (c == KEY_ENTER || c == '\012' || c == '\015') {
                        int _tid = -1, _beg, _end;
                        if (str[0] == '=') {

-                               _beg = strtol(str+1, &p, 10);
+                               _beg = strtol(str+1, &p, 10) - 1;

                                if (_beg > 0) {
                                        *pos = _beg;
                                        return;
                                if (_beg > 0) {
                                        *pos = _beg;
                                        return;

diff --git a/bamtk.c b/bamtk.c
index 94c4d3fdb34be8698e21e9cf20166cadd1ae06c8..082767c8d6bf0a29be8f1b73101b8a7c43e29a08 100644 (file)
--- a/bamtk.c
+++ b/bamtk.c
@@ -9,7 +9,7 @@
 #endif
 
 #ifndef PACKAGE_VERSION
 #endif
 
 #ifndef PACKAGE_VERSION

-#define PACKAGE_VERSION "0.1.8 (r613)"
+#define PACKAGE_VERSION "0.1.9 (r783)"

 #endif
 
 int bam_taf2baf(int argc, char *argv[]);
 #endif
 
 int bam_taf2baf(int argc, char *argv[]);

diff --git a/bcftools/Makefile b/bcftools/Makefile
new file mode 100644 (file)
index 0000000..8b890ba
--- /dev/null
+++ b/bcftools/Makefile
@@ -0,0 +1,51 @@
+CC=                    gcc
+CFLAGS=                -g -Wall -O2 #-m64 #-arch ppc
+DFLAGS=                -D_FILE_OFFSET_BITS=64 -D_USE_KNETFILE
+LOBJS=         bcf.o vcf.o bcfutils.o prob1.o ld.o kfunc.o index.o fet.o bcf2qcall.o
+OMISC=         ..
+AOBJS=         call1.o main.o $(OMISC)/kstring.o $(OMISC)/bgzf.o $(OMISC)/knetfile.o
+PROG=          bcftools
+INCLUDES=      
+SUBDIRS=       .
+
+.SUFFIXES:.c .o
+
+.c.o:
+               $(CC) -c $(CFLAGS) $(DFLAGS) -I.. $(INCLUDES) $< -o $@
+
+all-recur lib-recur clean-recur cleanlocal-recur install-recur:
+               @target=`echo $@ | sed s/-recur//`; \
+               wdir=`pwd`; \
+               list='$(SUBDIRS)'; for subdir in $$list; do \
+                       cd $$subdir; \
+                       $(MAKE) CC="$(CC)" DFLAGS="$(DFLAGS)" CFLAGS="$(CFLAGS)" \
+                               INCLUDES="$(INCLUDES)" LIBPATH="$(LIBPATH)" $$target || exit 1; \
+                       cd $$wdir; \
+               done;
+
+all:$(PROG)
+
+lib:libbcf.a
+
+libbcf.a:$(LOBJS)
+               $(AR) -cru $@ $(LOBJS)
+
+bcftools:lib $(AOBJS)
+               $(CC) $(CFLAGS) -o $@ $(AOBJS) -lm $(LIBPATH) -lz -L. -lbcf
+
+bcf.o:bcf.h
+vcf.o:bcf.h
+index.o:bcf.h
+bcfutils.o:bcf.h
+prob1.o:prob1.h bcf.h
+call1.o:prob1.h bcf.h
+bcf2qcall.o:bcf.h
+main.o:bcf.h
+
+bcf.pdf:bcf.tex
+               pdflatex bcf
+
+cleanlocal:
+               rm -fr gmon.out *.o a.out *.dSYM $(PROG) *~ *.a bcf.aux bcf.log bcf.pdf *.class libbcf.*.dylib libbcf.so*
+
+clean:cleanlocal-recur

diff --git a/bcftools/README b/bcftools/README
new file mode 100644 (file)
index 0000000..1d7159d
--- /dev/null
+++ b/bcftools/README
@@ -0,0 +1,36 @@
+The view command of bcftools calls variants, tests Hardy-Weinberg
+equilibrium (HWE), tests allele balances and estimates allele frequency.
+
+This command calls a site as a potential variant if P(ref|D,F) is below
+0.9 (controlled by the -p option), where D is data and F is the prior
+allele frequency spectrum (AFS).
+
+The view command performs two types of allele balance tests, both based
+on Fisher's exact test for 2x2 contingency tables with the row variable
+being reference allele or not. In the first table, the column variable
+is strand. Two-tail P-value is taken. We test if variant bases tend to
+come from one strand. In the second table, the column variable is
+whether a base appears in the first or the last 11bp of the read.
+One-tail P-value is taken. We test if variant bases tend to occur
+towards the end of reads, which is usually an indication of
+misalignment.
+
+Site allele frequency is estimated in two ways. In the first way, the
+frequency is esimated as \argmax_f P(D|f) under the assumption of
+HWE. Prior AFS is not used. In the second way, the frequency is
+estimated as the posterior expectation of allele counts \sum_k
+kP(k|D,F), dividied by the total number of haplotypes. HWE is not
+assumed, but the estimate depends on the prior AFS. The two estimates
+largely agree when the signal is strong, but may differ greatly on weak
+sites as in this case, the prior plays an important role.
+
+To test HWE, we calculate the posterior distribution of genotypes
+(ref-hom, het and alt-hom). Chi-square test is performed. It is worth
+noting that the model used here is prior dependent and assumes HWE,
+which is different from both models for allele frequency estimate. The
+new model actually yields a third estimate of site allele frequency.
+
+The estimate allele frequency spectrum is printed to stderr per 64k
+sites. The estimate is in fact only the first round of a EM
+procedure. The second model (not the model for HWE testing) is used to
+estimate the AFS.
\ No newline at end of file

diff --git a/bcftools/bcf-fix.pl b/bcftools/bcf-fix.pl
new file mode 100755 (executable)
index 0000000..61c6136
--- /dev/null
+++ b/bcftools/bcf-fix.pl
@@ -0,0 +1,101 @@
+#!/usr/bin/perl -w
+
+use strict;
+use warnings;
+use Carp;
+
+my $opts = parse_params();
+bcf_fix();
+
+exit;
+
+#--------------------------------
+
+sub error
+{
+    my (@msg) = @_;
+    if ( scalar @msg ) { confess @msg; }
+    die
+        "Usage: bcftools view test.bcf | bcf-fix.pl > test.vcf\n",
+        "Options:\n",
+        "   -h, -?, --help                  This help message.\n",
+        "\n";
+}
+
+
+sub parse_params
+{
+    my $opts = {};
+    while (my $arg=shift(@ARGV))
+    {
+        if ( $arg eq '-?' || $arg eq '-h' || $arg eq '--help' ) { error(); }
+        error("Unknown parameter \"$arg\". Run -h for help.\n");
+    }
+    return $opts;
+}
+
+sub bcf_fix
+{
+    while (my $line=<STDIN>)
+    {
+        if ( $line=~/^#CHROM/ )
+        {
+            print 
+qq[##INFO=<ID=DP4,Number=4,Type=Integer,Description="Read depth for 1) forward reference bases; 2) reverse ref; 3) forward non-ref; 4) reverse non-ref">
+##INFO=<ID=PV4,Number=4,Type=Float,Description="P-values for 1) strand bias (exact test); 2) baseQ bias (t-test); 3) mapQ bias (t); 4) tail distance bias (t)">
+##INFO=<ID=AF1,Number=1,Type=Float,Description="EM estimate of site allele frequency without prior">
+##INFO=<ID=AFE,Number=1,Type=Float,Description="Posterior expectation of site allele frequency (with prior)">
+##INFO=<ID=HWE,Number=1,Type=Float,Description="P-value for Hardy-Weinberg equillibrium (chi-square test)">
+##INFO=<ID=MQ,Number=1,Type=Integer,Descriptin="RMS mapping quality">
+##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
+##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype Quality">
+##FORMAT=<ID=GL,Number=3,Type=Float,Description="Likelihoods for RR,RA,AA genotypes (R=ref,A=alt)">
+];
+            print $line;
+        }
+        elsif ( $line=~/^#/ )
+        {
+            print $line;
+        }
+        else
+        {
+            my @items = split(/\t/,$line);
+            my @tags = split(/:/,$items[8]);    # FORMAT tags
+
+            my $nidx=2;
+            my @idxs;   # Mapping which defines new ordering: $idxs[$inew]=$iold; GT comes first, PL second
+            for (my $i=0; $i<@tags; $i++)
+            {
+                if ( $tags[$i] eq 'GT' ) { $idxs[0]=$i; }
+                elsif ( $tags[$i] eq 'PL' ) { $idxs[1]=$i; }
+                else { $idxs[$nidx++]=$i; }
+            }
+            if ( !exists($tags[0]) or !exists($tags[1]) ) { error("FIXME: expected GT and PL in the format field.\n"); }
+
+            # First fix the FORMAT column
+            $items[8] = 'GT:GL';
+            for (my $i=2; $i<@tags; $i++)
+            {
+                $items[8] .= ':'.$tags[$idxs[$i]];
+            }
+
+            # Now all the genotype columns
+            for (my $iitem=9; $iitem<@items; $iitem++)
+            {
+                @tags = split(/:/,$items[$iitem]);
+                $items[$iitem] = $tags[$idxs[0]] .':';
+
+                # GL=-PL/10
+                my ($a,$b,$c) = split(/,/,$tags[$idxs[1]]);
+                $items[$iitem] .= sprintf "%.2f,%.2f,%.2f",-$a/10.,-$b/10.,-$c/10.;
+
+                for (my $itag=2; $itag<@tags; $itag++)
+                {
+                    $items[$iitem] .= ':'.$tags[$idxs[$itag]];
+                }
+            }
+            print join("\t",@items);
+        }
+    }
+}
+

diff --git a/bcftools/bcf.c b/bcftools/bcf.c
new file mode 100644 (file)
index 0000000..9044b93
--- /dev/null
+++ b/bcftools/bcf.c
@@ -0,0 +1,306 @@
+#include <string.h>
+#include <ctype.h>
+#include <stdio.h>
+#include "kstring.h"
+#include "bcf.h"
+
+bcf_t *bcf_open(const char *fn, const char *mode)
+{
+       bcf_t *b;
+       b = calloc(1, sizeof(bcf_t));
+       if (strchr(mode, 'w')) {
+               b->fp = strcmp(fn, "-")? bgzf_open(fn, mode) : bgzf_fdopen(fileno(stdout), mode);
+       } else {
+               b->fp = strcmp(fn, "-")? bgzf_open(fn, mode) : bgzf_fdopen(fileno(stdin), mode);
+       }
+       b->fp->owned_file = 1;
+       return b;
+}
+
+int bcf_close(bcf_t *b)
+{
+       int ret;
+       if (b == 0) return 0;
+       ret = bgzf_close(b->fp);
+       free(b);
+       return ret;
+}
+
+int bcf_hdr_write(bcf_t *b, const bcf_hdr_t *h)
+{
+       if (b == 0 || h == 0) return -1;
+       bgzf_write(b->fp, "BCF\4", 4);
+       bgzf_write(b->fp, &h->l_nm, 4);
+       bgzf_write(b->fp, h->name, h->l_nm);
+       bgzf_write(b->fp, &h->l_smpl, 4);
+       bgzf_write(b->fp, h->sname, h->l_smpl);
+       bgzf_write(b->fp, &h->l_txt, 4);
+       bgzf_write(b->fp, h->txt, h->l_txt);
+       bgzf_flush(b->fp);
+       return 16 + h->l_nm + h->l_smpl + h->l_txt;
+}
+
+bcf_hdr_t *bcf_hdr_read(bcf_t *b)
+{
+       uint8_t magic[4];
+       bcf_hdr_t *h;
+       if (b == 0) return 0;
+       h = calloc(1, sizeof(bcf_hdr_t));
+       bgzf_read(b->fp, magic, 4);
+       bgzf_read(b->fp, &h->l_nm, 4);
+       h->name = malloc(h->l_nm);
+       bgzf_read(b->fp, h->name, h->l_nm);
+       bgzf_read(b->fp, &h->l_smpl, 4);
+       h->sname = malloc(h->l_smpl);
+       bgzf_read(b->fp, h->sname, h->l_smpl);
+       bgzf_read(b->fp, &h->l_txt, 4);
+       h->txt = malloc(h->l_txt);
+       bgzf_read(b->fp, h->txt, h->l_txt);
+       bcf_hdr_sync(h);
+       return h;
+}
+
+void bcf_hdr_destroy(bcf_hdr_t *h)
+{
+       if (h == 0) return;
+       free(h->name); free(h->sname); free(h->txt); free(h->ns); free(h->sns);
+       free(h);
+}
+
+static inline char **cnt_null(int l, char *str, int *_n)
+{
+       int n = 0;
+       char *p, **list;
+       *_n = 0;
+       if (l == 0 || str == 0) return 0;
+       for (p = str; p != str + l; ++p)
+               if (*p == 0) ++n;
+       *_n = n;
+       list = calloc(n, sizeof(void*));
+       list[0] = str;
+       for (p = str, n = 1; p < str + l - 1; ++p)
+               if (*p == 0) list[n++] = p + 1;
+       return list;
+}
+
+int bcf_hdr_sync(bcf_hdr_t *b)
+{
+       if (b == 0) return -1;
+       if (b->ns) free(b->ns);
+       if (b->sns) free(b->sns);
+       if (b->l_nm) b->ns = cnt_null(b->l_nm, b->name, &b->n_ref);
+       else b->ns = 0, b->n_ref = 0;
+       b->sns = cnt_null(b->l_smpl, b->sname, &b->n_smpl);
+       return 0;
+}
+
+int bcf_sync(bcf1_t *b)
+{
+       char *p, *tmp[5];
+       int i, n, n_smpl = b->n_smpl;
+       ks_tokaux_t aux;
+       // set ref, alt, flt, info, fmt
+       b->ref = b->alt = b->flt = b->info = b->fmt = 0;
+       for (p = b->str, n = 0; p < b->str + b->l_str; ++p)
+               if (*p == 0 && p+1 != b->str + b->l_str) tmp[n++] = p + 1;
+       if (n != 5) {
+               fprintf(stderr, "[%s] incorrect number of fields (%d != 5). Corrupted file?\n", __func__, n);
+               return -1;
+       }
+       b->ref = tmp[0]; b->alt = tmp[1]; b->flt = tmp[2]; b->info = tmp[3]; b->fmt = tmp[4];
+       // set n_alleles
+       if (*b->alt == 0) b->n_alleles = 1;
+       else {
+               for (p = b->alt, n = 1; *p; ++p)
+                       if (*p == ',') ++n;
+               b->n_alleles = n + 1;
+       }
+       // set n_gi and gi[i].fmt
+       for (p = b->fmt, n = 1; *p; ++p)
+               if (*p == ':') ++n;
+       if (n > b->m_gi) {
+               int old_m = b->m_gi;
+               b->m_gi = n;
+               kroundup32(b->m_gi);
+               b->gi = realloc(b->gi, b->m_gi * sizeof(bcf_ginfo_t));
+               memset(b->gi + old_m, 0, (b->m_gi - old_m) * sizeof(bcf_ginfo_t));
+       }
+       b->n_gi = n;
+       for (p = kstrtok(b->fmt, ":", &aux), n = 0; p; p = kstrtok(0, 0, &aux))
+               b->gi[n++].fmt = bcf_str2int(p, aux.p - p);
+       // set gi[i].len
+       for (i = 0; i < b->n_gi; ++i) {
+               if (b->gi[i].fmt == bcf_str2int("PL", 2)) {
+                       b->gi[i].len = b->n_alleles * (b->n_alleles + 1) / 2;
+               } else if (b->gi[i].fmt == bcf_str2int("DP", 2) || b->gi[i].fmt == bcf_str2int("HQ", 2)) {
+                       b->gi[i].len = 2;
+               } else if (b->gi[i].fmt == bcf_str2int("GQ", 2) || b->gi[i].fmt == bcf_str2int("GT", 2)) {
+                       b->gi[i].len = 1;
+               } else if (b->gi[i].fmt == bcf_str2int("GL", 2)) {
+                       b->gi[i].len = b->n_alleles * (b->n_alleles + 1) / 2 * 4;
+               }
+               b->gi[i].data = realloc(b->gi[i].data, n_smpl * b->gi[i].len);
+       }
+       return 0;
+}
+
+int bcf_write(bcf_t *bp, const bcf_hdr_t *h, const bcf1_t *b)
+{
+       int i, l = 0;
+       if (b == 0) return -1;
+       bgzf_write(bp->fp, &b->tid, 4);
+       bgzf_write(bp->fp, &b->pos, 4);
+       bgzf_write(bp->fp, &b->qual, 4);
+       bgzf_write(bp->fp, &b->l_str, 4);
+       bgzf_write(bp->fp, b->str, b->l_str);
+       l = 12 + b->l_str;
+       for (i = 0; i < b->n_gi; ++i) {
+               bgzf_write(bp->fp, b->gi[i].data, b->gi[i].len * h->n_smpl);
+               l += b->gi[i].len * h->n_smpl;
+       }
+       return l;
+}
+
+int bcf_read(bcf_t *bp, const bcf_hdr_t *h, bcf1_t *b)
+{
+       int i, l = 0;
+       if (b == 0) return -1;
+       if (bgzf_read(bp->fp, &b->tid, 4) == 0) return -1;
+       b->n_smpl = h->n_smpl;
+       bgzf_read(bp->fp, &b->pos, 4);
+       bgzf_read(bp->fp, &b->qual, 4);
+       bgzf_read(bp->fp, &b->l_str, 4);
+       if (b->l_str > b->m_str) {
+               b->m_str = b->l_str;
+               kroundup32(b->m_str);
+               b->str = realloc(b->str, b->m_str);
+       }
+       bgzf_read(bp->fp, b->str, b->l_str);
+       l = 12 + b->l_str;
+       bcf_sync(b);
+       for (i = 0; i < b->n_gi; ++i) {
+               bgzf_read(bp->fp, b->gi[i].data, b->gi[i].len * h->n_smpl);
+               l += b->gi[i].len * h->n_smpl;
+       }
+       return l;
+}
+
+int bcf_destroy(bcf1_t *b)
+{
+       int i;
+       if (b == 0) return -1;
+       free(b->str);
+       for (i = 0; i < b->m_gi; ++i)
+               free(b->gi[i].data);
+       free(b->gi);
+       free(b);
+       return 0;
+}
+
+static inline void fmt_str(const char *p, kstring_t *s)
+{
+       if (*p == 0) kputc('.', s);
+       else kputs(p, s);
+}
+
+void bcf_fmt_core(const bcf_hdr_t *h, bcf1_t *b, kstring_t *s)
+{
+       int i, j, x;
+       s->l = 0;
+       if (h->n_ref) kputs(h->ns[b->tid], s);
+       else kputw(b->tid, s);
+       kputc('\t', s);
+       kputw(b->pos + 1, s); kputc('\t', s);
+       fmt_str(b->str, s); kputc('\t', s);
+       fmt_str(b->ref, s); kputc('\t', s);
+       fmt_str(b->alt, s); kputc('\t', s);
+       ksprintf(s, "%.3g", b->qual); kputc('\t', s);
+       fmt_str(b->flt, s); kputc('\t', s);
+       fmt_str(b->info, s);
+       if (b->fmt[0]) {
+               kputc('\t', s);
+               fmt_str(b->fmt, s);
+       }
+       x = b->n_alleles * (b->n_alleles + 1) / 2;
+       if (b->n_gi == 0) return;
+       for (j = 0; j < h->n_smpl; ++j) {
+               kputc('\t', s);
+               for (i = 0; i < b->n_gi; ++i) {
+                       if (i) kputc(':', s);
+                       if (b->gi[i].fmt == bcf_str2int("PL", 2)) {
+                               uint8_t *d = (uint8_t*)b->gi[i].data + j * x;
+                               int k;
+                               for (k = 0; k < x; ++k) {
+                                       if (k > 0) kputc(',', s);
+                                       kputw(d[k], s);
+                               }
+                       } else if (b->gi[i].fmt == bcf_str2int("DP", 2)) {
+                               kputw(((uint16_t*)b->gi[i].data)[j], s);
+                       } else if (b->gi[i].fmt == bcf_str2int("GQ", 2)) {
+                               kputw(((uint8_t*)b->gi[i].data)[j], s);
+                       } else if (b->gi[i].fmt == bcf_str2int("GT", 2)) {
+                               int y = ((uint8_t*)b->gi[i].data)[j];
+                               if (y>>7&1) {
+                                       kputsn("./.", 3, s);
+                               } else {
+                                       kputc('0' + (y>>3&7), s);
+                                       kputc("/|"[y>>6&1], s);
+                                       kputc('0' + (y&7), s);
+                               }
+                       } else if (b->gi[i].fmt == bcf_str2int("GL", 2)) {
+                               float *d = (float*)b->gi[i].data + j * x;
+                               int k;
+                               //printf("- %lx\n", d);
+                               for (k = 0; k < x; ++k) {
+                                       if (k > 0) kputc(',', s);
+                                       ksprintf(s, "%.2f", d[k]);
+                               }
+                       }
+               }
+       }
+}
+
+char *bcf_fmt(const bcf_hdr_t *h, bcf1_t *b)
+{
+       kstring_t s;
+       s.l = s.m = 0; s.s = 0;
+       bcf_fmt_core(h, b, &s);
+       return s.s;
+}
+
+int bcf_append_info(bcf1_t *b, const char *info, int l)
+{
+       int shift = b->fmt - b->str;
+       int l_fmt = b->l_str - shift;
+       char *ori = b->str;
+       if (b->l_str + l > b->m_str) { // enlarge if necessary
+               b->m_str = b->l_str + l;
+               kroundup32(b->m_str);
+               b->str = realloc(b->str, b->m_str);
+       }
+       memmove(b->str + shift + l, b->str + shift, l_fmt); // move the FORMAT field
+       memcpy(b->str + shift - 1, info, l); // append to the INFO field
+       b->str[shift + l - 1] = '\0';
+       b->fmt = b->str + shift + l;
+       b->l_str += l;
+       if (ori != b->str) bcf_sync(b); // synchronize when realloc changes the pointer
+       return 0;
+}
+
+int bcf_cpy(bcf1_t *r, const bcf1_t *b)
+{
+       char *t1 = r->str;
+       bcf_ginfo_t *t2 = r->gi;
+       int i, t3 = r->m_str, t4 = r->m_gi;
+       *r = *b;
+       r->str = t1; r->gi = t2; r->m_str = t3; r->m_gi = t4;
+       if (r->m_str < b->m_str) {
+               r->m_str = b->m_str;
+               r->str = realloc(r->str, r->m_str);
+       }
+       memcpy(r->str, b->str, r->m_str);
+       bcf_sync(r); // calling bcf_sync() is simple but inefficient
+       for (i = 0; i < r->n_gi; ++i)
+               memcpy(r->gi[i].data, b->gi[i].data, r->n_smpl * r->gi[i].len);
+       return 0;
+}

diff --git a/bcftools/bcf.h b/bcftools/bcf.h
new file mode 100644 (file)
index 0000000..3657895
--- /dev/null
+++ b/bcftools/bcf.h
@@ -0,0 +1,161 @@
+/* The MIT License
+
+   Copyright (c) 2010 Broad Institute
+
+   Permission is hereby granted, free of charge, to any person obtaining
+   a copy of this software and associated documentation files (the
+   "Software"), to deal in the Software without restriction, including
+   without limitation the rights to use, copy, modify, merge, publish,
+   distribute, sublicense, and/or sell copies of the Software, and to
+   permit persons to whom the Software is furnished to do so, subject to
+   the following conditions:
+
+   The above copyright notice and this permission notice shall be
+   included in all copies or substantial portions of the Software.
+
+   THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+   EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+   MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+   NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+   BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+   ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+   CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+   SOFTWARE.
+*/
+
+/* Contact: Heng Li <lh3@live.co.uk> */
+
+#ifndef BCF_H
+#define BCF_H
+
+#include <stdint.h>
+#include <zlib.h>
+#include "bgzf.h"
+
+/*
+  A member in the structs below is said to "primary" if its content
+  cannot be inferred from other members in any of structs below; a
+  member is said to be "derived" if its content can be derived from
+  other members. For example, bcf1_t::str is primary as this comes from
+  the input data, while bcf1_t::info is derived as it can always be
+  correctly set if we know bcf1_t::str. Derived members are for quick
+  access to the content and must be synchronized with the primary data.
+ */
+
+typedef struct {
+       uint32_t fmt; // format of the block, set by bcf_str2int(). 
+       int len; // length of data for each individual
+       void *data; // concatenated data
+       // derived info: fmt, len (<-bcf1_t::fmt)
+} bcf_ginfo_t;
+
+typedef struct {
+       int32_t tid, pos; // refID and 0-based position
+       int32_t l_str, m_str; // length and the allocated size of ->str
+       float qual; // SNP quality
+       char *str; // concatenated string of variable length strings in VCF (from col.2 to col.7)
+       char *ref, *alt, *flt, *info, *fmt; // they all point to ->str; no memory allocation
+       int n_gi, m_gi; // number and the allocated size of geno fields
+       bcf_ginfo_t *gi; // array of geno fields
+       int n_alleles, n_smpl; // number of alleles and samples
+       // derived info: ref, alt, flt, info, fmt (<-str), n_gi (<-fmt), n_alleles (<-alt), n_smpl (<-bcf_hdr_t::n_smpl)
+} bcf1_t;
+
+typedef struct {
+       int32_t n_ref, n_smpl; // number of reference sequences and samples
+       int32_t l_nm; // length of concatenated sequence names; 0 padded
+       int32_t l_smpl; // length of concatenated sample names; 0 padded
+       int32_t l_txt; // length of header text (lines started with ##)
+       char *name, *sname, *txt; // concatenated sequence names, sample names and header text
+       char **ns, **sns; // array of sequence and sample names; point to name and sname, respectively
+       // derived info: n_ref (<-name), n_smpl (<-sname), ns (<-name), sns (<-sname)
+} bcf_hdr_t;
+
+typedef struct {
+       int is_vcf; // if the file in operation is a VCF
+       void *v; // auxillary data structure for VCF
+       BGZF *fp; // file handler for BCF
+} bcf_t;
+
+struct __bcf_idx_t;
+typedef struct __bcf_idx_t bcf_idx_t;
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+       // open a BCF file; for BCF file only
+       bcf_t *bcf_open(const char *fn, const char *mode);
+       // close file
+       int bcf_close(bcf_t *b);
+       // read one record from BCF; return -1 on end-of-file, and <-1 for errors
+       int bcf_read(bcf_t *bp, const bcf_hdr_t *h, bcf1_t *b);
+       // call this function if b->str is changed
+       int bcf_sync(bcf1_t *b);
+       // write a BCF record
+       int bcf_write(bcf_t *bp, const bcf_hdr_t *h, const bcf1_t *b);
+       // read the BCF header; BCF only
+       bcf_hdr_t *bcf_hdr_read(bcf_t *b);
+       // write the BCF header
+       int bcf_hdr_write(bcf_t *b, const bcf_hdr_t *h);
+       // set bcf_hdr_t::ns and bcf_hdr_t::sns
+       int bcf_hdr_sync(bcf_hdr_t *b);
+       // destroy the header
+       void bcf_hdr_destroy(bcf_hdr_t *h);
+       // destroy a record
+       int bcf_destroy(bcf1_t *b);
+       // BCF->VCF conversion
+       char *bcf_fmt(const bcf_hdr_t *h, bcf1_t *b);
+       // append more info
+       int bcf_append_info(bcf1_t *b, const char *info, int l);
+       // copy
+       int bcf_cpy(bcf1_t *r, const bcf1_t *b);
+
+       // open a VCF or BCF file if "b" is set in "mode"
+       bcf_t *vcf_open(const char *fn, const char *mode);
+       // close a VCF/BCF file
+       int vcf_close(bcf_t *bp);
+       // read the VCF/BCF header
+       bcf_hdr_t *vcf_hdr_read(bcf_t *bp);
+       // read a VCF/BCF record; return -1 on end-of-file and <-1 for errors
+       int vcf_read(bcf_t *bp, bcf_hdr_t *h, bcf1_t *b);
+       // write the VCF header
+       int vcf_hdr_write(bcf_t *bp, const bcf_hdr_t *h);
+       // write a VCF record
+       int vcf_write(bcf_t *bp, bcf_hdr_t *h, bcf1_t *b);
+
+       // keep the first n alleles and discard the rest
+       int bcf_shrink_alt(bcf1_t *b, int n);
+       // convert GL to PL
+       int bcf_gl2pl(bcf1_t *b);
+
+       // string hash table
+       void *bcf_build_refhash(bcf_hdr_t *h);
+       void bcf_str2id_destroy(void *_hash);
+       int bcf_str2id_add(void *_hash, const char *str);
+       int bcf_str2id(void *_hash, const char *str);
+       void *bcf_str2id_init();
+
+       // indexing related functions
+       int bcf_idx_build(const char *fn);
+       uint64_t bcf_idx_query(const bcf_idx_t *idx, int tid, int beg);
+       int bcf_parse_region(void *str2id, const char *str, int *tid, int *begin, int *end);
+       bcf_idx_t *bcf_idx_load(const char *fn);
+       void bcf_idx_destroy(bcf_idx_t *idx);
+
+#ifdef __cplusplus
+}
+#endif
+
+static inline uint32_t bcf_str2int(const char *str, int l)
+{
+       int i;
+       uint32_t x = 0;
+       for (i = 0; i < l && i < 4; ++i) {
+               if (str[i] == 0) return x;
+               x = x<<8 | str[i];
+       }
+       return x;
+}
+
+#endif

diff --git a/bcftools/bcf.tex b/bcftools/bcf.tex
new file mode 100644 (file)
index 0000000..5ca1e28
--- /dev/null
+++ b/bcftools/bcf.tex
@@ -0,0 +1,63 @@
+\documentclass[10pt,pdftex]{article}
+\usepackage{color}
+\definecolor{gray}{rgb}{0.7,0.7,0.7}
+
+\setlength{\topmargin}{0.0cm}
+\setlength{\textheight}{21.5cm}
+\setlength{\oddsidemargin}{0cm} 
+\setlength{\textwidth}{16.5cm}
+\setlength{\columnsep}{0.6cm}
+
+\begin{document}
+
+\begin{center}
+\begin{tabular}{|l|l|l|l|l|}
+\hline
+\multicolumn{2}{|c|}{\bf Field} & \multicolumn{1}{c|}{\bf Descrption} & \multicolumn{1}{c|}{\bf Type} & \multicolumn{1}{c|}{\bf Value} \\\hline\hline
+\multicolumn{2}{|l|}{\tt magic} & Magic string & {\tt char[4]} & {\tt BCF\char92 4} \\\hline
+\multicolumn{2}{|l|}{\tt l\_nm} & Length of concatenated sequence names & {\tt int32\_t} & \\\hline
+\multicolumn{2}{|l|}{\tt name} & Concatenated names, {\tt NULL} padded & {\tt char[l\_nm]} & \\\hline
+\multicolumn{2}{|l|}{\tt l\_smpl} & Length of concatenated sample names & {\tt int32\_t} & \\\hline
+\multicolumn{2}{|l|}{\tt sname} & Concatenated sample names & {\tt char[l\_smpl]} & \\\hline
+\multicolumn{2}{|l|}{\tt l\_txt} & Length of the meta text (double-hash lines)& {\tt int32\_t} & \\\hline
+\multicolumn{2}{|l|}{\tt text} & Meta text, {\tt NULL} terminated & {\tt char[l\_txt]} & \\\hline
+\multicolumn{5}{|c|}{\it \color{gray}{List of records until the end of the file}}\\\cline{2-5}
+& {\tt seq\_id} & Reference sequence ID & {\tt int32\_t} & \\\cline{2-5}
+& {\tt pos} & Position & {\tt int32\_t} & \\\cline{2-5}
+& {\tt qual} & Variant quality & {\tt float} & \\\cline{2-5}
+& {\tt l\_str} & Length of str & {\tt int32\_t} & \\\cline{2-5}
+& {\tt str} & {\tt ID+REF+ALT+FILTER+INFO+FORMAT}, {\tt NULL} padded & {\tt char[slen]} &\\\cline{2-5}
+& \multicolumn{4}{c|}{Blocks of data; \#blocks and formats defined by {\tt FORMAT} (table below)}\\
+\hline
+\end{tabular}
+\end{center}
+
+\begin{center}
+\begin{tabular}{cll}
+\hline
+\multicolumn{1}{l}{\bf Field} & \multicolumn{1}{l}{\bf Type} & \multicolumn{1}{l}{\bf Description} \\\hline
+{\tt DP} & {\tt uint16\_t[n]} & Read depth \\
+{\tt GL} & {\tt float[n*x]} & Log10 likelihood of data; $x=\frac{m(m+1)}{2}$, $m=\#\{alleles\}$\\
+{\tt GT} & {\tt uint8\_t[n]} & {\tt phase\char60\char60 6 | allele1\char60\char60 3 | allele2} \\
+{\tt GQ} & {\tt uint8\_t[n]} & {Genotype quality}\\
+{\tt HQ} & {\tt uint8\_t[n*2]} & {Haplotype quality}\\
+{\tt PL} & {\tt uint8\_t[n*x]} & {Phred-scaled likelihood of data}\\
+\emph{misc} & {\tt int32\_t+char*} & {\tt NULL} padded concatenated strings (integer equal to the length) \\
+\hline
+\end{tabular}
+\end{center}
+
+\begin{itemize}
+\item The file is {\tt BGZF} compressed.
+\item All integers are little-endian.
+\item In a string, a missing value `.' is an empty C string ``{\tt
+    \char92 0}'' (not ``{\tt .\char92 0}'')
+\item For {\tt GL} and {\tt PL}, likelihoods of genotypes appear in the
+  order of alleles in {\tt REF} and then {\tt ALT}. For example, if {\tt
+    REF=C}, {\tt ALT=T,A}, likelihoods appear in the order of {\tt
+    CC,CT,CA,TT,TA,AA}.
+\item {\tt GL} is an extension to and is backward compatible with the
+  {\tt GL} genotype field in {\tt VCFv4.0}.
+\end{itemize}
+
+\end{document}
\ No newline at end of file

diff --git a/bcftools/bcf2qcall.c b/bcftools/bcf2qcall.c
new file mode 100644 (file)
index 0000000..8634c9e
--- /dev/null
+++ b/bcftools/bcf2qcall.c
@@ -0,0 +1,91 @@
+#include <errno.h>
+#include <math.h>
+#include <string.h>
+#include <stdlib.h>
+#include "bcf.h"
+
+static int8_t nt4_table[256] = {
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4 /*'-'*/, 4, 4,
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 0, 4, 1,  4, 4, 4, 2,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  3, 4, 4, 4, -1, 4, 4, 4,  4, 4, 4, 4, 
+       4, 0, 4, 1,  4, 4, 4, 2,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  3, 4, 4, 4, -1, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4
+};
+
+static int read_I16(bcf1_t *b, int anno[16])
+{
+       char *p;
+       int i;
+       if ((p = strstr(b->info, "I16=")) == 0) return -1;
+       p += 4;
+       for (i = 0; i < 16; ++i) {
+               anno[i] = strtol(p, &p, 10);
+               if (anno[i] == 0 && (errno == EINVAL || errno == ERANGE)) return -2;
+               ++p;
+       }
+       return 0;
+}
+
+int bcf_2qcall(bcf_hdr_t *h, bcf1_t *b)
+{
+       int a[4], k, g[10], l, map[4], k1, j, i, i0, anno[16], dp, mq, d_rest;
+       char *s;
+       if (b->ref[1] != 0 || b->n_alleles > 4) return -1; // ref is not a single base
+       for (i = 0; i < b->n_gi; ++i)
+               if (b->gi[i].fmt == bcf_str2int("PL", 2)) break;
+       if (i == b->n_gi) return -1; // no PL
+       if (read_I16(b, anno) != 0) return -1; // no I16; FIXME: can be improved
+       d_rest = dp = anno[0] + anno[1] + anno[2] + anno[3];
+       if (dp == 0) return -1; // depth is zero
+       mq = (int)(sqrt((double)(anno[9] + anno[11]) / dp) + .499);
+       i0 = i;
+       a[0] = nt4_table[(int)b->ref[0]];
+       if (a[0] > 3) return -1; // ref is not A/C/G/T
+       a[1] = a[2] = a[3] = -2; // -1 has a special meaning
+       if (b->alt[0] == 0) return -1; // no alternate allele
+       map[0] = map[1] = map[2] = map[3] = -2;
+       map[a[0]] = 0;
+       for (k = 0, s = b->alt, k1 = -1; k < 3 && *s; ++k, s += 2) {
+               if (s[1] != ',' && s[1] != 0) return -1; // ALT is not single base
+               a[k+1] = nt4_table[(int)*s];
+               if (a[k+1] >= 0) map[a[k+1]] = k+1;
+               else k1 = k+1;
+               if (s[1] == 0) break;
+       }
+       for (k = 0; k < 4; ++k)
+               if (map[k] < 0) map[k] = k1;
+       for (i = 0; i < h->n_smpl; ++i) {
+               int d;
+               uint8_t *p = b->gi[i0].data + i * b->gi[i0].len;
+               for (j = 0; j < b->gi[i0].len; ++j)
+                       if (p[j]) break;
+               d = (int)((double)d_rest / (h->n_smpl - i) + .499);
+               if (d == 0) d = 1;
+               if (j == b->gi[i0].len) d = 0;
+               d_rest -= d;
+               for (k = j = 0; k < 4; ++k) {
+                       for (l = k; l < 4; ++l) {
+                               int t, x = map[k], y = map[l];
+                               if (x > y) t = x, x = y, y = t;
+                               g[j++] = p[x * b->n_alleles - x * (x-1) / 2 + (y - x)];
+                       }
+               }
+               printf("%s\t%d\t%c", h->ns[b->tid], b->pos+1, *b->ref);
+               printf("\t%d\t%d\t0", d, mq);
+               for (j = 0; j < 10; ++j)
+                       printf("\t%d", g[j]);
+               printf("\t%s\n", h->sns[i]);
+       }
+       return 0;
+}

diff --git a/bcftools/bcftools.1 b/bcftools/bcftools.1
new file mode 100644 (file)
index 0000000..ebff301
--- /dev/null
+++ b/bcftools/bcftools.1
@@ -0,0 +1,115 @@
+.TH bcftools 1 "2 October 2010" "bcftools" "Bioinformatics tools"
+.SH NAME
+.PP
+bcftools - Utilities for the Binary Call Format (BCF) and VCF.
+.SH SYNOPSIS
+.PP
+bcftools index in.bcf
+.PP
+bcftools view in.bcf chr2:100-200 > out.vcf
+.PP
+bcftools view -vc in.bcf > out.vcf 2> out.afs
+
+.SH DESCRIPTION
+.PP
+Bcftools is a toolkit for processing VCF/BCF files, calling variants and
+estimating site allele frequencies and allele frequency spectrums.
+
+.SH COMMANDS AND OPTIONS
+
+.TP 10
+.B view
+.B bcftools view
+.RB [ \-cbuSAGgHvNQ ]
+.RB [ \-1
+.IR nGroup1 ]
+.RB [ \-l
+.IR listFile ]
+.RB [ \-t
+.IR mutRate ]
+.RB [ \-p
+.IR varThres ]
+.RB [ \-P
+.IR prior ]
+.I in.bcf
+.RI [ region ]
+
+Convert between BCF and VCF, call variant candidates and estimate allele
+frequencies.
+
+.B OPTIONS:
+.RS
+.TP 10
+.B -b
+Output in the BCF format. The default is VCF.
+.TP
+.B -c
+Call variants.
+.TP
+.B -v
+Output variant sites only (force -c)
+.TP
+.B -g
+Call per-sample genotypes at variant sites (force -c)
+.TP
+.B -u
+Uncompressed BCF output (force -b).
+.TP
+.B -S
+The input is VCF instead of BCF.
+.TP
+.B -A
+Retain all possible alternate alleles at variant sites. By default, this
+command discards unlikely alleles.
+.TP
+.B -G
+Suppress all individual genotype information.
+.TP
+.B -H
+Perform Hardy-Weiberg Equilibrium test. This will add computation time, sometimes considerably.
+.TP
+.B -N
+Skip sites where the REF field is not A/C/G/T
+.TP
+.B -Q
+Output the QCALL likelihood format
+.TP
+.BI "-1 " INT
+Number of group-1 samples. This option is used for dividing input into
+two groups for comparing. A zero value disables this functionality. [0]
+.TP
+.BI "-l " FILE
+List of sites at which information are outputted [all sites]
+.TP
+.BI "-t " FLOAT
+Scaled muttion rate for variant calling [0.001]
+.TP
+.BI "-p " FLOAT
+A site is considered to be a variant if P(ref|D)<FLOAT [0.5]
+.TP
+.BI "-P " STR
+Prior or initial allele frequency spectrum. If STR can be
+.IR full ,
+.IR cond2 ,
+.I flat
+or the file consisting of error output from a previous variant calling
+run.
+.RE
+
+.TP
+.B index
+.B bcftools index
+.I in.bcf
+
+Index sorted BCF for random access.
+.RE
+
+.TP
+.B cat
+.B bcftools cat
+.I in1.bcf
+.RI [ "in2.bcf " [ ... "]]]"
+
+Concatenate BCF files. The input files are required to be sorted and
+have identical samples appearing in the same order.
+.RE

diff --git a/bcftools/bcfutils.c b/bcftools/bcfutils.c
new file mode 100644 (file)
index 0000000..4d6835d
--- /dev/null
+++ b/bcftools/bcfutils.c
@@ -0,0 +1,109 @@
+#include "bcf.h"
+#include "kstring.h"
+#include "khash.h"
+KHASH_MAP_INIT_STR(str2id, int)
+
+void *bcf_build_refhash(bcf_hdr_t *h)
+{
+       khash_t(str2id) *hash;
+       int i, ret;
+       hash = kh_init(str2id);
+       for (i = 0; i < h->n_ref; ++i) {
+               khint_t k;
+               k = kh_put(str2id, hash, h->ns[i], &ret); // FIXME: check ret
+               kh_val(hash, k) = i;
+       }
+       return hash;
+}
+
+void *bcf_str2id_init()
+{
+       return kh_init(str2id);
+}
+
+void bcf_str2id_destroy(void *_hash)
+{
+       khash_t(str2id) *hash = (khash_t(str2id)*)_hash;
+       if (hash) kh_destroy(str2id, hash); // Note that strings are not freed.
+}
+
+int bcf_str2id(void *_hash, const char *str)
+{
+       khash_t(str2id) *hash = (khash_t(str2id)*)_hash;
+       khint_t k;
+       if (!hash) return -1;
+       k = kh_get(str2id, hash, str);
+       return k == kh_end(hash)? -1 : kh_val(hash, k);
+}
+
+int bcf_str2id_add(void *_hash, const char *str)
+{
+       khint_t k;
+       int ret;
+       khash_t(str2id) *hash = (khash_t(str2id)*)_hash;
+       if (!hash) return -1;
+       k = kh_put(str2id, hash, str, &ret);
+       if (ret == 0) return kh_val(hash, k);
+       kh_val(hash, k) = kh_size(hash) - 1;
+       return kh_val(hash, k);
+}
+
+int bcf_shrink_alt(bcf1_t *b, int n)
+{
+       char *p;
+       int i, j, k, *z, n_smpl = b->n_smpl;
+       if (b->n_alleles <= n) return -1;
+       if (n > 1) {
+               for (p = b->alt, k = 1; *p; ++p)
+                       if (*p == ',' && ++k == n) break;
+               *p = '\0';
+       } else p = b->alt, *p = '\0';
+       ++p;
+       memmove(p, b->flt, b->str + b->l_str - b->flt);
+       b->l_str -= b->flt - p;
+       z = alloca(sizeof(int) / 2 * n * (n+1));
+       for (i = k = 0; i < n; ++i)
+               for (j = 0; j < n - i; ++j)
+                       z[k++] = i * b->n_alleles + j;
+       for (i = 0; i < b->n_gi; ++i) {
+               bcf_ginfo_t *g = b->gi + i;
+               if (g->fmt == bcf_str2int("PL", 2)) {
+                       int l, x = b->n_alleles * (b->n_alleles + 1) / 2;
+                       uint8_t *d = (uint8_t*)g->data;
+                       g->len = n * (n + 1) / 2;
+                       for (l = k = 0; l < n_smpl; ++l) {
+                               uint8_t *dl = d + l * x;
+                               for (j = 0; j < g->len; ++j) d[k++] = dl[z[j]];
+                       }
+               } // FIXME: to add GL
+       }
+       b->n_alleles = n;
+       bcf_sync(b);
+       return 0;
+}
+
+int bcf_gl2pl(bcf1_t *b)
+{
+       char *p;
+       int i, n_smpl = b->n_smpl;
+       bcf_ginfo_t *g;
+       float *d0;
+       uint8_t *d1;
+       if (strstr(b->fmt, "PL")) return -1;
+       if ((p = strstr(b->fmt, "GL")) == 0) return -1;
+       *p = 'P';
+       for (i = 0; i < b->n_gi; ++i)
+               if (b->gi[i].fmt == bcf_str2int("GL", 2))
+                       break;
+       g = b->gi + i;
+       g->fmt = bcf_str2int("PL", 2);
+       g->len /= 4; // 4 == sizeof(float)
+       d0 = (float*)g->data; d1 = (uint8_t*)g->data;
+       for (i = 0; i < n_smpl * g->len; ++i) {
+               int x = (int)(-10. * d0[i] + .499);
+               if (x > 255) x = 255;
+               if (x < 0) x = 0;
+               d1[i] = x;
+       }
+       return 0;
+}

diff --git a/bcftools/call1.c b/bcftools/call1.c
new file mode 100644 (file)
index 0000000..2b28452
--- /dev/null
+++ b/bcftools/call1.c
@@ -0,0 +1,372 @@
+#include <unistd.h>
+#include <stdlib.h>
+#include <math.h>
+#include <zlib.h>
+#include <errno.h>
+#include "bcf.h"
+#include "prob1.h"
+#include "kstring.h"
+
+#include "khash.h"
+KHASH_SET_INIT_INT64(set64)
+
+#include "kseq.h"
+KSTREAM_INIT(gzFile, gzread, 16384)
+
+#define VC_NO_GENO 2
+#define VC_BCFOUT  4
+#define VC_CALL    8
+#define VC_VARONLY 16
+#define VC_VCFIN   32
+#define VC_UNCOMP  64
+#define VC_HWE     128
+#define VC_KEEPALT 256
+#define VC_ACGT_ONLY 512
+#define VC_QCALL   1024
+#define VC_CALL_GT 2048
+#define VC_ADJLD   4096
+
+typedef struct {
+       int flag, prior_type, n1;
+       char *fn_list, *prior_file;
+       double theta, pref;
+} viewconf_t;
+
+khash_t(set64) *bcf_load_pos(const char *fn, bcf_hdr_t *_h)
+{
+       void *str2id;
+       gzFile fp;
+       kstream_t *ks;
+       int ret, dret, lineno = 1;
+       kstring_t *str;
+       khash_t(set64) *hash = 0;
+
+       hash = kh_init(set64);
+       str2id = bcf_build_refhash(_h);
+       str = calloc(1, sizeof(kstring_t));
+       fp = strcmp(fn, "-")? gzopen(fn, "r") : gzdopen(fileno(stdin), "r");
+       ks = ks_init(fp);
+       while (ks_getuntil(ks, 0, str, &dret) >= 0) {
+               int tid = bcf_str2id(str2id, str->s);
+               if (tid >= 0 && dret != '\n') {
+                       if (ks_getuntil(ks, 0, str, &dret) >= 0) {
+                               uint64_t x = (uint64_t)tid<<32 | (atoi(str->s) - 1);
+                               kh_put(set64, hash, x, &ret);
+                       } else break;
+               } else fprintf(stderr, "[%s] %s is not a reference name (line %d).\n", __func__, str->s, lineno);
+               if (dret != '\n') while ((dret = ks_getc(ks)) > 0 && dret != '\n');
+               if (dret < 0) break;
+               ++lineno;
+       }
+       bcf_str2id_destroy(str2id);
+       ks_destroy(ks);
+       gzclose(fp);
+       free(str->s); free(str);
+       return hash;
+}
+
+static double test_hwe(const double g[3])
+{
+       extern double kf_gammaq(double p, double x);
+       double fexp, chi2, f[3], n;
+       int i;
+       n = g[0] + g[1] + g[2];
+       fexp = (2. * g[2] + g[1]) / (2. * n);
+       if (fexp > 1. - 1e-10) fexp = 1. - 1e-10;
+       if (fexp < 1e-10) fexp = 1e-10;
+       f[0] = n * (1. - fexp) * (1. - fexp);
+       f[1] = n * 2. * fexp * (1. - fexp);
+       f[2] = n * fexp * fexp;
+       for (i = 0, chi2 = 0.; i < 3; ++i)
+               chi2 += (g[i] - f[i]) * (g[i] - f[i]) / f[i];
+       return kf_gammaq(.5, chi2 / 2.);
+}
+
+typedef struct {
+       double p[4];
+       int mq, depth, is_tested, d[4];
+} anno16_t;
+
+static double ttest(int n1, int n2, int a[4])
+{
+       extern double kf_betai(double a, double b, double x);
+       double t, v, u1, u2;
+       if (n1 == 0 || n2 == 0 || n1 + n2 < 3) return 1.0;
+       u1 = (double)a[0] / n1; u2 = (double)a[2] / n2;
+       if (u1 <= u2) return 1.;
+       t = (u1 - u2) / sqrt(((a[1] - n1 * u1 * u1) + (a[3] - n2 * u2 * u2)) / (n1 + n2 - 2) * (1./n1 + 1./n2));
+       v = n1 + n2 - 2;
+//     printf("%d,%d,%d,%d,%lf,%lf,%lf\n", a[0], a[1], a[2], a[3], t, u1, u2);
+       return t < 0.? 1. : .5 * kf_betai(.5*v, .5, v/(v+t*t));
+}
+
+static int test16_core(int anno[16], anno16_t *a)
+{
+       extern double kt_fisher_exact(int n11, int n12, int n21, int n22, double *_left, double *_right, double *two);
+       double left, right;
+       int i;
+       a->p[0] = a->p[1] = a->p[2] = a->p[3] = 1.;
+       memcpy(a->d, anno, 4 * sizeof(int));
+       a->depth = anno[0] + anno[1] + anno[2] + anno[3];
+       a->is_tested = (anno[0] + anno[1] > 0 && anno[2] + anno[3] > 0);
+       if (a->depth == 0) return -1;
+       a->mq = (int)(sqrt((anno[9] + anno[11]) / a->depth) + .499);
+       kt_fisher_exact(anno[0], anno[1], anno[2], anno[3], &left, &right, &a->p[0]);
+       for (i = 1; i < 4; ++i)
+               a->p[i] = ttest(anno[0] + anno[1], anno[2] + anno[3], anno+4*i);
+       return 0;
+}
+
+static int test16(bcf1_t *b, anno16_t *a)
+{
+       char *p;
+       int i, anno[16];
+       a->p[0] = a->p[1] = a->p[2] = a->p[3] = 1.;
+       a->d[0] = a->d[1] = a->d[2] = a->d[3] = 0.;
+       a->mq = a->depth = a->is_tested = 0;
+       if ((p = strstr(b->info, "I16=")) == 0) return -1;
+       p += 4;
+       for (i = 0; i < 16; ++i) {
+               anno[i] = strtol(p, &p, 10);
+               if (anno[i] == 0 && (errno == EINVAL || errno == ERANGE)) return -2;
+               ++p;
+       }
+       return test16_core(anno, a);
+}
+
+static void rm_info(bcf1_t *b, const char *key)
+{
+       char *p, *q;
+       if ((p = strstr(b->info, key)) == 0) return;
+       for (q = p; *q && *q != ';'; ++q);
+       if (p > b->info && *(p-1) == ';') --p;
+       memmove(p, q, b->l_str - (q - b->str));
+       b->l_str -= q - p;
+       bcf_sync(b);
+}
+
+static int update_bcf1(int n_smpl, bcf1_t *b, const bcf_p1aux_t *pa, const bcf_p1rst_t *pr, double pref, int flag)
+{
+       kstring_t s;
+       int is_var = (pr->p_ref < pref);
+       double p_hwe, r = is_var? pr->p_ref : 1. - pr->p_ref;
+       anno16_t a;
+
+       p_hwe = pr->g[0] >= 0.? test_hwe(pr->g) : 1.0; // only do HWE g[] is calculated
+       test16(b, &a);
+       rm_info(b, "I16=");
+
+       memset(&s, 0, sizeof(kstring_t));
+       kputc('\0', &s); kputs(b->ref, &s); kputc('\0', &s);
+       kputs(b->alt, &s); kputc('\0', &s); kputc('\0', &s);
+       kputs(b->info, &s);
+       if (b->info[0]) kputc(';', &s);
+       ksprintf(&s, "AF1=%.3lf;AFE=%.3lf", 1.-pr->f_em, 1.-pr->f_exp);
+       ksprintf(&s, ";DP4=%d,%d,%d,%d;MQ=%d", a.d[0], a.d[1], a.d[2], a.d[3], a.mq);
+       if (a.is_tested) {
+               if (pr->pc[0] >= 0.) ksprintf(&s, ";PC4=%lg,%lg,%lg,%lg", pr->pc[0], pr->pc[1], pr->pc[2], pr->pc[3]);
+               ksprintf(&s, ";PV4=%.2lg,%.2lg,%.2lg,%.2lg", a.p[0], a.p[1], a.p[2], a.p[3]);
+       }
+       if (pr->g[0] >= 0. && p_hwe <= .2)
+               ksprintf(&s, ";GC=%.2lf,%.2lf,%.2lf;HWE=%.3lf", pr->g[2], pr->g[1], pr->g[0], p_hwe);
+       kputc('\0', &s);
+       kputs(b->fmt, &s); kputc('\0', &s);
+       free(b->str);
+       b->m_str = s.m; b->l_str = s.l; b->str = s.s;
+       b->qual = r < 1e-100? 99 : -4.343 * log(r);
+       if (b->qual > 99) b->qual = 99;
+       bcf_sync(b);
+       if (!is_var) bcf_shrink_alt(b, 1);
+       else if (!(flag&VC_KEEPALT))
+               bcf_shrink_alt(b, pr->rank0 < 2? 2 : pr->rank0+1);
+       if (is_var && (flag&VC_CALL_GT)) { // call individual genotype
+               int i, x, old_n_gi = b->n_gi;
+               s.m = b->m_str; s.l = b->l_str - 1; s.s = b->str;
+               kputs(":GT:GQ", &s); kputc('\0', &s);
+               b->m_str = s.m; b->l_str = s.l; b->str = s.s;
+               bcf_sync(b);
+               for (i = 0; i < b->n_smpl; ++i) {
+                       x = bcf_p1_call_gt(pa, pr->f_em, i);
+                       ((uint8_t*)b->gi[old_n_gi].data)[i] = (x&3) == 0? 1<<3|1 : (x&3) == 1? 1 : 0;
+                       ((uint8_t*)b->gi[old_n_gi+1].data)[i] = x>>2;
+               }
+       }
+       return is_var;
+}
+
+double bcf_ld_freq(const bcf1_t *b0, const bcf1_t *b1, double f[4]);
+
+int bcfview(int argc, char *argv[])
+{
+       extern int bcf_2qcall(bcf_hdr_t *h, bcf1_t *b);
+       bcf_t *bp, *bout = 0;
+       bcf1_t *b, *blast;
+       int c;
+       uint64_t n_processed = 0;
+       viewconf_t vc;
+       bcf_p1aux_t *p1 = 0;
+       bcf_hdr_t *h;
+       int tid, begin, end;
+       char moder[4], modew[4];
+       khash_t(set64) *hash = 0;
+
+       tid = begin = end = -1;
+       memset(&vc, 0, sizeof(viewconf_t));
+       vc.prior_type = vc.n1 = -1; vc.theta = 1e-3; vc.pref = 0.5;
+       while ((c = getopt(argc, argv, "N1:l:cHAGvbSuP:t:p:QgL")) >= 0) {
+               switch (c) {
+               case '1': vc.n1 = atoi(optarg); break;
+               case 'l': vc.fn_list = strdup(optarg); break;
+               case 'N': vc.flag |= VC_ACGT_ONLY; break;
+               case 'G': vc.flag |= VC_NO_GENO; break;
+               case 'A': vc.flag |= VC_KEEPALT; break;
+               case 'b': vc.flag |= VC_BCFOUT; break;
+               case 'S': vc.flag |= VC_VCFIN; break;
+               case 'c': vc.flag |= VC_CALL; break;
+               case 'v': vc.flag |= VC_VARONLY | VC_CALL; break;
+               case 'u': vc.flag |= VC_UNCOMP | VC_BCFOUT; break;
+               case 'H': vc.flag |= VC_HWE; break;
+               case 'g': vc.flag |= VC_CALL_GT | VC_CALL; break;
+               case 't': vc.theta = atof(optarg); break;
+               case 'p': vc.pref = atof(optarg); break;
+               case 'Q': vc.flag |= VC_QCALL; break;
+               case 'L': vc.flag |= VC_ADJLD; break;
+               case 'P':
+                       if (strcmp(optarg, "full") == 0) vc.prior_type = MC_PTYPE_FULL;
+                       else if (strcmp(optarg, "cond2") == 0) vc.prior_type = MC_PTYPE_COND2;
+                       else if (strcmp(optarg, "flat") == 0) vc.prior_type = MC_PTYPE_FLAT;
+                       else vc.prior_file = strdup(optarg);
+                       break;
+               }
+       }
+       if (argc == optind) {
+               fprintf(stderr, "\n");
+               fprintf(stderr, "Usage:   bcftools view [options] <in.bcf> [reg]\n\n");
+               fprintf(stderr, "Options: -c        SNP calling\n");
+               fprintf(stderr, "         -v        output potential variant sites only (force -c)\n");
+               fprintf(stderr, "         -g        call genotypes at variant sites (force -c)\n");
+               fprintf(stderr, "         -b        output BCF instead of VCF\n");
+               fprintf(stderr, "         -u        uncompressed BCF output (force -b)\n");
+               fprintf(stderr, "         -S        input is VCF\n");
+               fprintf(stderr, "         -A        keep all possible alternate alleles at variant sites\n");
+               fprintf(stderr, "         -G        suppress all individual genotype information\n");
+               fprintf(stderr, "         -H        perform Hardy-Weinberg test (slower)\n");
+               fprintf(stderr, "         -N        skip sites where REF is not A/C/G/T\n");
+               fprintf(stderr, "         -Q        output the QCALL likelihood format\n");
+               fprintf(stderr, "         -L        calculate LD for adjacent sites\n");
+               fprintf(stderr, "         -1 INT    number of group-1 samples [0]\n");
+               fprintf(stderr, "         -l FILE   list of sites to output [all sites]\n");
+               fprintf(stderr, "         -t FLOAT  scaled mutation rate [%.4lg]\n", vc.theta);
+               fprintf(stderr, "         -p FLOAT  variant if P(ref|D)<FLOAT [%.3lg]\n", vc.pref);
+               fprintf(stderr, "         -P STR    type of prior: full, cond2, flat [full]\n");
+               fprintf(stderr, "\n");
+               return 1;
+       }
+
+       b = calloc(1, sizeof(bcf1_t));
+       blast = calloc(1, sizeof(bcf1_t));
+       strcpy(moder, "r");
+       if (!(vc.flag & VC_VCFIN)) strcat(moder, "b");
+       strcpy(modew, "w");
+       if (vc.flag & VC_BCFOUT) strcat(modew, "b");
+       if (vc.flag & VC_UNCOMP) strcat(modew, "u");
+       bp = vcf_open(argv[optind], moder);
+       h = vcf_hdr_read(bp);
+       bout = vcf_open("-", modew);
+       if (!(vc.flag & VC_QCALL)) vcf_hdr_write(bout, h);
+       if (vc.flag & VC_CALL) {
+               p1 = bcf_p1_init(h->n_smpl);
+               if (vc.prior_file) {
+                       if (bcf_p1_read_prior(p1, vc.prior_file) < 0) {
+                               fprintf(stderr, "[%s] fail to read the prior AFS.\n", __func__);
+                               return 1;
+                       }
+               } else bcf_p1_init_prior(p1, vc.prior_type, vc.theta);
+               if (vc.n1 > 0) {
+                       bcf_p1_set_n1(p1, vc.n1);
+                       bcf_p1_init_subprior(p1, vc.prior_type, vc.theta);
+               }
+       }
+       if (vc.fn_list) hash = bcf_load_pos(vc.fn_list, h);
+       if (optind + 1 < argc && !(vc.flag&VC_VCFIN)) {
+               void *str2id = bcf_build_refhash(h);
+               if (bcf_parse_region(str2id, argv[optind+1], &tid, &begin, &end) >= 0) {
+                       bcf_idx_t *idx;
+                       idx = bcf_idx_load(argv[optind]);
+                       if (idx) {
+                               uint64_t off;
+                               off = bcf_idx_query(idx, tid, begin);
+                               if (off == 0) {
+                                       fprintf(stderr, "[%s] no records in the query region.\n", __func__);
+                                       return 1; // FIXME: a lot of memory leaks...
+                               }
+                               bgzf_seek(bp->fp, off, SEEK_SET);
+                               bcf_idx_destroy(idx);
+                       }
+               }
+       }
+       while (vcf_read(bp, h, b) > 0) {
+               if (vc.flag & VC_ACGT_ONLY) {
+                       int x;
+                       if (b->ref[0] == 0 || b->ref[1] != 0) continue;
+                       x = toupper(b->ref[0]);
+                       if (x != 'A' && x != 'C' && x != 'G' && x != 'T') continue;
+               }
+               if (hash) {
+                       uint64_t x = (uint64_t)b->tid<<32 | b->pos;
+                       khint_t k = kh_get(set64, hash, x);
+                       if (kh_size(hash) == 0) break;
+                       if (k == kh_end(hash)) continue;
+                       kh_del(set64, hash, k);
+               }
+               if (tid >= 0) {
+                       int l = strlen(b->ref);
+                       l = b->pos + (l > 0? l : 1);
+                       if (b->tid != tid || b->pos >= end) break;
+                       if (!(l > begin && end > b->pos)) continue;
+               }
+               ++n_processed;
+               if (vc.flag & VC_QCALL) { // output QCALL format; STOP here
+                       bcf_2qcall(h, b);
+                       continue;
+               }
+               if (vc.flag & (VC_CALL|VC_ADJLD)) bcf_gl2pl(b);
+               if (vc.flag & VC_CALL) { // call variants
+                       bcf_p1rst_t pr;
+                       bcf_p1_cal(b, p1, &pr); // pr.g[3] is not calculated here
+                       if (vc.flag&VC_HWE) bcf_p1_cal_g3(p1, pr.g);
+                       if (n_processed % 100000 == 0) {
+                               fprintf(stderr, "[%s] %ld sites processed.\n", __func__, (long)n_processed);
+                               bcf_p1_dump_afs(p1);
+                       }
+                       if (pr.p_ref >= vc.pref && (vc.flag & VC_VARONLY)) continue;
+                       update_bcf1(h->n_smpl, b, p1, &pr, vc.pref, vc.flag);
+               }
+               if (vc.flag & VC_ADJLD) { // compute LD
+                       double f[4], r2;
+                       if ((r2 = bcf_ld_freq(blast, b, f)) >= 0) {
+                               kstring_t s;
+                               s.m = s.l = 0; s.s = 0;
+                               if (*b->info) kputc(';', &s);
+                               ksprintf(&s, "NEIR=%.3lf;NEIF=%.3lf,%.3lf", r2, f[0]+f[2], f[0]+f[1]);
+                               bcf_append_info(b, s.s, s.l);
+                               free(s.s);
+                       }
+                       bcf_cpy(blast, b);
+               }
+               if (vc.flag & VC_NO_GENO) { // do not output GENO fields
+                       b->n_gi = 0;
+                       b->fmt[0] = '\0';
+               }
+               vcf_write(bout, h, b);
+       }
+       if (vc.prior_file) free(vc.prior_file);
+       if (vc.flag & VC_CALL) bcf_p1_dump_afs(p1);
+       bcf_hdr_destroy(h);
+       bcf_destroy(b); bcf_destroy(blast);
+       vcf_close(bp); vcf_close(bout);
+       if (hash) kh_destroy(set64, hash);
+       if (vc.fn_list) free(vc.fn_list);
+       if (p1) bcf_p1_destroy(p1);
+       return 0;
+}

diff --git a/bcftools/fet.c b/bcftools/fet.c
new file mode 100644 (file)
index 0000000..845f8c2
--- /dev/null
+++ b/bcftools/fet.c
@@ -0,0 +1,110 @@
+#include <math.h>
+#include <stdlib.h>
+
+/* This program is implemented with ideas from this web page:
+ *
+ *   http://www.langsrud.com/fisher.htm
+ */
+
+// log\binom{n}{k}
+static double lbinom(int n, int k)
+{
+       if (k == 0 || n == k) return 0;
+       return lgamma(n+1) - lgamma(k+1) - lgamma(n-k+1);
+}
+
+// n11  n12  | n1_
+// n21  n22  | n2_
+//-----------+----
+// n_1  n_2  | n
+
+// hypergeometric distribution
+static double hypergeo(int n11, int n1_, int n_1, int n)
+{
+       return exp(lbinom(n1_, n11) + lbinom(n-n1_, n_1-n11) - lbinom(n, n_1));
+}
+
+typedef struct {
+       int n11, n1_, n_1, n;
+       double p;
+} hgacc_t;
+
+// incremental version of hypergenometric distribution
+static double hypergeo_acc(int n11, int n1_, int n_1, int n, hgacc_t *aux)
+{
+       if (n1_ || n_1 || n) {
+               aux->n11 = n11; aux->n1_ = n1_; aux->n_1 = n_1; aux->n = n;
+       } else { // then only n11 changed; the rest fixed
+               if (n11%11 && n11 + aux->n - aux->n1_ - aux->n_1) {
+                       if (n11 == aux->n11 + 1) { // incremental
+                               aux->p *= (double)(aux->n1_ - aux->n11) / n11
+                                       * (aux->n_1 - aux->n11) / (n11 + aux->n - aux->n1_ - aux->n_1);
+                               aux->n11 = n11;
+                               return aux->p;
+                       }
+                       if (n11 == aux->n11 - 1) { // incremental
+                               aux->p *= (double)aux->n11 / (aux->n1_ - n11)
+                                       * (aux->n11 + aux->n - aux->n1_ - aux->n_1) / (aux->n_1 - n11);
+                               aux->n11 = n11;
+                               return aux->p;
+                       }
+               }
+               aux->n11 = n11;
+       }
+       aux->p = hypergeo(aux->n11, aux->n1_, aux->n_1, aux->n);
+       return aux->p;
+}
+
+double kt_fisher_exact(int n11, int n12, int n21, int n22, double *_left, double *_right, double *two)
+{
+       int i, j, max, min;
+       double p, q, left, right;
+       hgacc_t aux;
+       int n1_, n_1, n;
+
+       n1_ = n11 + n12; n_1 = n11 + n21; n = n11 + n12 + n21 + n22; // calculate n1_, n_1 and n
+       max = (n_1 < n1_) ? n_1 : n1_; // max n11, for right tail
+       min = (n1_ + n_1 - n < 0) ? 0 : (n1_ + n_1 - n < 0); // min n11, for left tail
+       *two = *_left = *_right = 1.;
+       if (min == max) return 1.; // no need to do test
+       q = hypergeo_acc(n11, n1_, n_1, n, &aux); // the probability of the current table
+       // left tail
+       p = hypergeo_acc(min, 0, 0, 0, &aux);
+       for (left = 0., i = min + 1; p < 0.99999999 * q; ++i) // loop until underflow
+               left += p, p = hypergeo_acc(i, 0, 0, 0, &aux);
+       --i;
+       if (p < 1.00000001 * q) left += p;
+       else --i;
+       // right tail
+       p = hypergeo_acc(max, 0, 0, 0, &aux);
+       for (right = 0., j = max - 1; p < 0.99999999 * q; --j) // loop until underflow
+               right += p, p = hypergeo_acc(j, 0, 0, 0, &aux);
+       if (p < 1.00000001 * q) right += p;
+       else ++j;
+       // two-tail
+       *two = left + right;
+       if (*two > 1.) *two = 1.;
+       // adjust left and right
+       if (abs(i - n11) < abs(j - n11)) right = 1. - left + q;
+       else left = 1.0 - right + q;
+       *_left = left; *_right = right;
+       return q;
+}
+
+#ifdef FET_MAIN
+#include <stdio.h>
+
+int main(int argc, char *argv[])
+{
+       char id[1024];
+       int n11, n12, n21, n22;
+       double left, right, twotail, prob;
+
+       while (scanf("%s%d%d%d%d", id, &n11, &n12, &n21, &n22) == 5) {
+               prob = kt_fisher_exact(n11, n12, n21, n22, &left, &right, &twotail);
+               printf("%s\t%d\t%d\t%d\t%d\t%.6g\t%.6g\t%.6g\t%.6g\n", id, n11, n12, n21, n22,
+                               prob, left, right, twotail);
+       }
+       return 0;
+}
+#endif

diff --git a/bcftools/index.c b/bcftools/index.c
new file mode 100644 (file)
index 0000000..014856d
--- /dev/null
+++ b/bcftools/index.c
@@ -0,0 +1,335 @@
+#include <assert.h>
+#include <ctype.h>
+#include <sys/stat.h>
+#include "bam_endian.h"
+#include "kstring.h"
+#include "bcf.h"
+#ifdef _USE_KNETFILE
+#include "knetfile.h"
+#endif
+
+#define TAD_LIDX_SHIFT 13
+
+typedef struct {
+       int32_t n, m;
+       uint64_t *offset;
+} bcf_lidx_t;
+
+struct __bcf_idx_t {
+       int32_t n;
+       bcf_lidx_t *index2;
+};
+
+/************
+ * indexing *
+ ************/
+
+static inline void insert_offset2(bcf_lidx_t *index2, int _beg, int _end, uint64_t offset)
+{
+       int i, beg, end;
+       beg = _beg >> TAD_LIDX_SHIFT;
+       end = (_end - 1) >> TAD_LIDX_SHIFT;
+       if (index2->m < end + 1) {
+               int old_m = index2->m;
+               index2->m = end + 1;
+               kroundup32(index2->m);
+               index2->offset = (uint64_t*)realloc(index2->offset, index2->m * 8);
+               memset(index2->offset + old_m, 0, 8 * (index2->m - old_m));
+       }
+       if (beg == end) {
+               if (index2->offset[beg] == 0) index2->offset[beg] = offset;
+       } else {
+               for (i = beg; i <= end; ++i)
+                       if (index2->offset[i] == 0) index2->offset[i] = offset;
+       }
+       if (index2->n < end + 1) index2->n = end + 1;
+}
+
+bcf_idx_t *bcf_idx_core(bcf_t *bp, bcf_hdr_t *h)
+{
+       bcf_idx_t *idx;
+       int32_t last_coor, last_tid;
+       uint64_t last_off;
+       kstring_t *str;
+       BGZF *fp = bp->fp;
+       bcf1_t *b;
+       int ret;
+
+       b = calloc(1, sizeof(bcf1_t));
+       str = calloc(1, sizeof(kstring_t));
+       idx = (bcf_idx_t*)calloc(1, sizeof(bcf_idx_t));
+       idx->n = h->n_ref;
+       idx->index2 = calloc(h->n_ref, sizeof(bcf_lidx_t));
+
+       last_tid = 0xffffffffu;
+       last_off = bgzf_tell(fp); last_coor = 0xffffffffu;
+       while ((ret = bcf_read(bp, h, b)) > 0) {
+               int end, tmp;
+               if (last_tid != b->tid) { // change of chromosomes
+                       last_tid = b->tid;
+               } else if (last_coor > b->pos) {
+                       fprintf(stderr, "[bcf_idx_core] the input is out of order\n");
+                       free(str->s); free(str); free(idx); bcf_destroy(b);
+                       return 0;
+               }
+               tmp = strlen(b->ref);
+               end = b->pos + (tmp > 0? tmp : 1);
+               insert_offset2(&idx->index2[b->tid], b->pos, end, last_off);
+               last_off = bgzf_tell(fp);
+               last_coor = b->pos;
+       }
+       free(str->s); free(str); bcf_destroy(b);
+       return idx;
+}
+
+void bcf_idx_destroy(bcf_idx_t *idx)
+{
+       int i;
+       if (idx == 0) return;
+       for (i = 0; i < idx->n; ++i) free(idx->index2[i].offset);
+       free(idx->index2);
+       free(idx);
+}
+
+/******************
+ * index file I/O *
+ ******************/
+
+void bcf_idx_save(const bcf_idx_t *idx, BGZF *fp)
+{
+       int32_t i, ti_is_be;
+       ti_is_be = bam_is_big_endian();
+       bgzf_write(fp, "BCI\4", 4);
+       if (ti_is_be) {
+               uint32_t x = idx->n;
+               bgzf_write(fp, bam_swap_endian_4p(&x), 4);
+       } else bgzf_write(fp, &idx->n, 4);
+       for (i = 0; i < idx->n; ++i) {
+               bcf_lidx_t *index2 = idx->index2 + i;
+               // write linear index (index2)
+               if (ti_is_be) {
+                       int x = index2->n;
+                       bgzf_write(fp, bam_swap_endian_4p(&x), 4);
+               } else bgzf_write(fp, &index2->n, 4);
+               if (ti_is_be) { // big endian
+                       int x;
+                       for (x = 0; (int)x < index2->n; ++x)
+                               bam_swap_endian_8p(&index2->offset[x]);
+                       bgzf_write(fp, index2->offset, 8 * index2->n);
+                       for (x = 0; (int)x < index2->n; ++x)
+                               bam_swap_endian_8p(&index2->offset[x]);
+               } else bgzf_write(fp, index2->offset, 8 * index2->n);
+       }
+}
+
+static bcf_idx_t *bcf_idx_load_core(BGZF *fp)
+{
+       int i, ti_is_be;
+       char magic[4];
+       bcf_idx_t *idx;
+       ti_is_be = bam_is_big_endian();
+       if (fp == 0) {
+               fprintf(stderr, "[%s] fail to load index.\n", __func__);
+               return 0;
+       }
+       bgzf_read(fp, magic, 4);
+       if (strncmp(magic, "BCI\4", 4)) {
+               fprintf(stderr, "[%s] wrong magic number.\n", __func__);
+               return 0;
+       }
+       idx = (bcf_idx_t*)calloc(1, sizeof(bcf_idx_t)); 
+       bgzf_read(fp, &idx->n, 4);
+       if (ti_is_be) bam_swap_endian_4p(&idx->n);
+       idx->index2 = (bcf_lidx_t*)calloc(idx->n, sizeof(bcf_lidx_t));
+       for (i = 0; i < idx->n; ++i) {
+               bcf_lidx_t *index2 = idx->index2 + i;
+               int j;
+               bgzf_read(fp, &index2->n, 4);
+               if (ti_is_be) bam_swap_endian_4p(&index2->n);
+               index2->m = index2->n;
+               index2->offset = (uint64_t*)calloc(index2->m, 8);
+               bgzf_read(fp, index2->offset, index2->n * 8);
+               if (ti_is_be)
+                       for (j = 0; j < index2->n; ++j) bam_swap_endian_8p(&index2->offset[j]);
+       }
+       return idx;
+}
+
+bcf_idx_t *bcf_idx_load_local(const char *fnidx)
+{
+       BGZF *fp;
+       fp = bgzf_open(fnidx, "r");
+       if (fp) {
+               bcf_idx_t *idx = bcf_idx_load_core(fp);
+               bgzf_close(fp);
+               return idx;
+       } else return 0;
+}
+
+#ifdef _USE_KNETFILE
+static void download_from_remote(const char *url)
+{
+       const int buf_size = 1 * 1024 * 1024;
+       char *fn;
+       FILE *fp;
+       uint8_t *buf;
+       knetFile *fp_remote;
+       int l;
+       if (strstr(url, "ftp://") != url && strstr(url, "http://") != url) return;
+       l = strlen(url);
+       for (fn = (char*)url + l - 1; fn >= url; --fn)
+               if (*fn == '/') break;
+       ++fn; // fn now points to the file name
+       fp_remote = knet_open(url, "r");
+       if (fp_remote == 0) {
+               fprintf(stderr, "[download_from_remote] fail to open remote file.\n");
+               return;
+       }
+       if ((fp = fopen(fn, "w")) == 0) {
+               fprintf(stderr, "[download_from_remote] fail to create file in the working directory.\n");
+               knet_close(fp_remote);
+               return;
+       }
+       buf = (uint8_t*)calloc(buf_size, 1);
+       while ((l = knet_read(fp_remote, buf, buf_size)) != 0)
+               fwrite(buf, 1, l, fp);
+       free(buf);
+       fclose(fp);
+       knet_close(fp_remote);
+}
+#else
+static void download_from_remote(const char *url)
+{
+       return;
+}
+#endif
+
+static char *get_local_version(const char *fn)
+{
+    struct stat sbuf;
+       char *fnidx = (char*)calloc(strlen(fn) + 5, 1);
+       strcat(strcpy(fnidx, fn), ".bci");
+       if ((strstr(fnidx, "ftp://") == fnidx || strstr(fnidx, "http://") == fnidx)) {
+               char *p, *url;
+               int l = strlen(fnidx);
+               for (p = fnidx + l - 1; p >= fnidx; --p)
+                       if (*p == '/') break;
+               url = fnidx; fnidx = strdup(p + 1);
+               if (stat(fnidx, &sbuf) == 0) {
+                       free(url);
+                       return fnidx;
+               }
+               fprintf(stderr, "[%s] downloading the index file...\n", __func__);
+               download_from_remote(url);
+               free(url);
+       }
+    if (stat(fnidx, &sbuf) == 0) return fnidx;
+       free(fnidx); return 0;
+}
+
+bcf_idx_t *bcf_idx_load(const char *fn)
+{
+       bcf_idx_t *idx;
+    char *fname = get_local_version(fn);
+       if (fname == 0) return 0;
+       idx = bcf_idx_load_local(fname);
+    free(fname);
+       return idx;
+}
+
+int bcf_idx_build2(const char *fn, const char *_fnidx)
+{
+       char *fnidx;
+       BGZF *fpidx;
+       bcf_t *bp;
+       bcf_idx_t *idx;
+       bcf_hdr_t *h;
+       if ((bp = bcf_open(fn, "r")) == 0) {
+               fprintf(stderr, "[bcf_idx_build2] fail to open the BAM file.\n");
+               return -1;
+       }
+       h = bcf_hdr_read(bp);
+       idx = bcf_idx_core(bp, h);
+       bcf_close(bp);
+       if (_fnidx == 0) {
+               fnidx = (char*)calloc(strlen(fn) + 5, 1);
+               strcpy(fnidx, fn); strcat(fnidx, ".bci");
+       } else fnidx = strdup(_fnidx);
+       fpidx = bgzf_open(fnidx, "w");
+       if (fpidx == 0) {
+               fprintf(stderr, "[bcf_idx_build2] fail to create the index file.\n");
+               free(fnidx);
+               return -1;
+       }
+       bcf_idx_save(idx, fpidx);
+       bcf_idx_destroy(idx);
+       bgzf_close(fpidx);
+       free(fnidx);
+       return 0;
+}
+
+int bcf_idx_build(const char *fn)
+{
+       return bcf_idx_build2(fn, 0);
+}
+
+/********************************************
+ * parse a region in the format chr:beg-end *
+ ********************************************/
+
+int bcf_parse_region(void *str2id, const char *str, int *tid, int *begin, int *end)
+{
+       char *s, *p;
+       int i, l, k;
+       l = strlen(str);
+       p = s = (char*)malloc(l+1);
+       /* squeeze out "," */
+       for (i = k = 0; i != l; ++i)
+               if (str[i] != ',' && !isspace(str[i])) s[k++] = str[i];
+       s[k] = 0;
+       for (i = 0; i != k; ++i) if (s[i] == ':') break;
+       s[i] = 0;
+       if ((*tid = bcf_str2id(str2id, s)) < 0) {
+               free(s);
+               return -1;
+       }
+       if (i == k) { /* dump the whole sequence */
+               *begin = 0; *end = 1<<29; free(s);
+               return 0;
+       }
+       for (p = s + i + 1; i != k; ++i) if (s[i] == '-') break;
+       *begin = atoi(p);
+       if (i < k) {
+               p = s + i + 1;
+               *end = atoi(p);
+       } else *end = 1<<29;
+       if (*begin > 0) --*begin;
+       free(s);
+       if (*begin > *end) return -1;
+       return 0;
+}
+
+/*******************************
+ * retrieve a specified region *
+ *******************************/
+
+uint64_t bcf_idx_query(const bcf_idx_t *idx, int tid, int beg)
+{
+       uint64_t min_off, *offset;
+       int i;
+       if (beg < 0) beg = 0;
+       offset = idx->index2[tid].offset;
+       for (i = beg>>TAD_LIDX_SHIFT; i < idx->index2[tid].n && offset[i] == 0; ++i);
+       min_off = (i == idx->index2[tid].n)? offset[idx->index2[tid].n-1] : offset[i];
+       return min_off;
+}
+
+int bcf_main_index(int argc, char *argv[])
+{
+       if (argc == 1) {
+               fprintf(stderr, "Usage: bcftools index <in.bcf>\n");
+               return 1;
+       }
+       bcf_idx_build(argv[1]);
+       return 0;
+}

diff --git a/bcftools/kfunc.c b/bcftools/kfunc.c
new file mode 100644 (file)
index 0000000..a637b6c
--- /dev/null
+++ b/bcftools/kfunc.c
@@ -0,0 +1,162 @@
+#include <math.h>
+
+
+/* Log gamma function
+ * \log{\Gamma(z)}
+ * AS245, 2nd algorithm, http://lib.stat.cmu.edu/apstat/245
+ */
+double kf_lgamma(double z)
+{
+       double x = 0;
+       x += 0.1659470187408462e-06 / (z+7);
+       x += 0.9934937113930748e-05 / (z+6);
+       x -= 0.1385710331296526     / (z+5);
+       x += 12.50734324009056      / (z+4);
+       x -= 176.6150291498386      / (z+3);
+       x += 771.3234287757674      / (z+2);
+       x -= 1259.139216722289      / (z+1);
+       x += 676.5203681218835      / z;
+       x += 0.9999999999995183;
+       return log(x) - 5.58106146679532777 - z + (z-0.5) * log(z+6.5);
+}
+
+/* complementary error function
+ * \frac{2}{\sqrt{\pi}} \int_x^{\infty} e^{-t^2} dt
+ * AS66, 2nd algorithm, http://lib.stat.cmu.edu/apstat/66
+ */
+double kf_erfc(double x)
+{
+       const double p0 = 220.2068679123761;
+       const double p1 = 221.2135961699311;
+       const double p2 = 112.0792914978709;
+       const double p3 = 33.912866078383;
+       const double p4 = 6.37396220353165;
+       const double p5 = .7003830644436881;
+       const double p6 = .03526249659989109;
+       const double q0 = 440.4137358247522;
+       const double q1 = 793.8265125199484;
+       const double q2 = 637.3336333788311;
+       const double q3 = 296.5642487796737;
+       const double q4 = 86.78073220294608;
+       const double q5 = 16.06417757920695;
+       const double q6 = 1.755667163182642;
+       const double q7 = .08838834764831844;
+       double expntl, z, p;
+       z = fabs(x) * M_SQRT2;
+       if (z > 37.) return x > 0.? 0. : 2.;
+       expntl = exp(z * z * - .5);
+       if (z < 10. / M_SQRT2) // for small z
+           p = expntl * ((((((p6 * z + p5) * z + p4) * z + p3) * z + p2) * z + p1) * z + p0)
+                       / (((((((q7 * z + q6) * z + q5) * z + q4) * z + q3) * z + q2) * z + q1) * z + q0);
+       else p = expntl / 2.506628274631001 / (z + 1. / (z + 2. / (z + 3. / (z + 4. / (z + .65)))));
+       return x > 0.? 2. * p : 2. * (1. - p);
+}
+
+/* The following computes regularized incomplete gamma functions.
+ * Formulas are taken from Wiki, with additional input from Numerical
+ * Recipes in C (for modified Lentz's algorithm) and AS245
+ * (http://lib.stat.cmu.edu/apstat/245).
+ *
+ * A good online calculator is available at:
+ *
+ *   http://www.danielsoper.com/statcalc/calc23.aspx
+ *
+ * It calculates upper incomplete gamma function, which equals
+ * kf_gammaq(s,z)*tgamma(s).
+ */
+
+#define KF_GAMMA_EPS 1e-14
+#define KF_TINY 1e-290
+
+// regularized lower incomplete gamma function, by series expansion
+static double _kf_gammap(double s, double z)
+{
+       double sum, x;
+       int k;
+       for (k = 1, sum = x = 1.; k < 100; ++k) {
+               sum += (x *= z / (s + k));
+               if (x / sum < KF_GAMMA_EPS) break;
+       }
+       return exp(s * log(z) - z - kf_lgamma(s + 1.) + log(sum));
+}
+// regularized upper incomplete gamma function, by continued fraction
+static double _kf_gammaq(double s, double z)
+{
+       int j;
+       double C, D, f;
+       f = 1. + z - s; C = f; D = 0.;
+       // Modified Lentz's algorithm for computing continued fraction
+       // See Numerical Recipes in C, 2nd edition, section 5.2
+       for (j = 1; j < 100; ++j) {
+               double a = j * (s - j), b = (j<<1) + 1 + z - s, d;
+               D = b + a * D;
+               if (D < KF_TINY) D = KF_TINY;
+               C = b + a / C;
+               if (C < KF_TINY) C = KF_TINY;
+               D = 1. / D;
+               d = C * D;
+               f *= d;
+               if (fabs(d - 1.) < KF_GAMMA_EPS) break;
+       }
+       return exp(s * log(z) - z - kf_lgamma(s) - log(f));
+}
+
+double kf_gammap(double s, double z)
+{
+       return z <= 1. || z < s? _kf_gammap(s, z) : 1. - _kf_gammaq(s, z);
+}
+
+double kf_gammaq(double s, double z)
+{
+       return z <= 1. || z < s? 1. - _kf_gammap(s, z) : _kf_gammaq(s, z);
+}
+
+/* Regularized incomplete beta function. The method is taken from
+ * Numerical Recipe in C, 2nd edition, section 6.4. The following web
+ * page calculates the incomplete beta function, which equals
+ * kf_betai(a,b,x) * gamma(a) * gamma(b) / gamma(a+b):
+ *
+ *   http://www.danielsoper.com/statcalc/calc36.aspx
+ */
+static double kf_betai_aux(double a, double b, double x)
+{
+       double C, D, f;
+       int j;
+       if (x == 0.) return 0.;
+       if (x == 1.) return 1.;
+       f = 1.; C = f; D = 0.;
+       // Modified Lentz's algorithm for computing continued fraction
+       for (j = 1; j < 200; ++j) {
+               double aa, d;
+               int m = j>>1;
+               aa = (j&1)? -(a + m) * (a + b + m) * x / ((a + 2*m) * (a + 2*m + 1))
+                       : m * (b - m) * x / ((a + 2*m - 1) * (a + 2*m));
+               D = 1. + aa * D;
+               if (D < KF_TINY) D = KF_TINY;
+               C = 1. + aa / C;
+               if (C < KF_TINY) C = KF_TINY;
+               D = 1. / D;
+               d = C * D;
+               f *= d;
+               if (fabs(d - 1.) < KF_GAMMA_EPS) break;
+       }
+       return exp(kf_lgamma(a+b) - kf_lgamma(a) - kf_lgamma(b) + a * log(x) + b * log(1.-x)) / a / f;
+}
+double kf_betai(double a, double b, double x)
+{
+       return x < (a + 1.) / (a + b + 2.)? kf_betai_aux(a, b, x) : 1. - kf_betai_aux(b, a, 1. - x);
+}
+
+#ifdef KF_MAIN
+#include <stdio.h>
+int main(int argc, char *argv[])
+{
+       double x = 5.5, y = 3;
+       double a, b;
+       printf("erfc(%lg): %lg, %lg\n", x, erfc(x), kf_erfc(x));
+       printf("upper-gamma(%lg,%lg): %lg\n", x, y, kf_gammaq(y, x)*tgamma(y));
+       a = 2; b = 2; x = 0.5;
+       printf("incomplete-beta(%lg,%lg,%lg): %lg\n", a, b, x, kf_betai(a, b, x) / exp(kf_lgamma(a+b) - kf_lgamma(a) - kf_lgamma(b)));
+       return 0;
+}
+#endif

diff --git a/bcftools/ld.c b/bcftools/ld.c
new file mode 100644 (file)
index 0000000..aa7ec07
--- /dev/null
+++ b/bcftools/ld.c
@@ -0,0 +1,100 @@
+#include <stdlib.h>
+#include <string.h>
+#include <math.h>
+#include "bcf.h"
+
+static double g_q2p[256];
+
+#define LD_ITER_MAX 50
+#define LD_ITER_EPS 1e-4
+
+#define _G1(h, k) ((h>>1&1) + (k>>1&1))
+#define _G2(h, k) ((h&1) + (k&1))
+
+// 0: the previous site; 1: the current site
+static int freq_iter(int n, double *pdg[2], double f[4])
+{
+       double ff[4];
+       int i, k, h;
+       memset(ff, 0, 4 * sizeof(double));
+       for (i = 0; i < n; ++i) {
+               double *p[2], sum, tmp;
+               p[0] = pdg[0] + i * 3; p[1] = pdg[1] + i * 3;
+               for (k = 0, sum = 0.; k < 4; ++k)
+                       for (h = 0; h < 4; ++h)
+                               sum += f[k] * f[h] * p[0][_G1(k,h)] * p[1][_G2(k,h)];
+               for (k = 0; k < 4; ++k) {
+                       tmp = f[0] * (p[0][_G1(0,k)] * p[1][_G2(0,k)] + p[0][_G1(k,0)] * p[1][_G2(k,0)])
+                               + f[1] * (p[0][_G1(1,k)] * p[1][_G2(1,k)] + p[0][_G1(k,1)] * p[1][_G2(k,1)])
+                               + f[2] * (p[0][_G1(2,k)] * p[1][_G2(2,k)] + p[0][_G1(k,2)] * p[1][_G2(k,2)])
+                               + f[3] * (p[0][_G1(3,k)] * p[1][_G2(3,k)] + p[0][_G1(k,3)] * p[1][_G2(k,3)]);
+                       ff[k] += f[k] * tmp / sum;
+               }
+       }
+       for (k = 0; k < 4; ++k) f[k] = ff[k] / (2 * n);
+       return 0;
+}
+
+double bcf_ld_freq(const bcf1_t *b0, const bcf1_t *b1, double f[4])
+{
+       const bcf1_t *b[2];
+       uint8_t *PL[2];
+       int i, j, PL_len[2], n_smpl;
+       double *pdg[2], flast[4], r;
+       // initialize g_q2p if necessary
+       if (g_q2p[0] == 0.)
+               for (i = 0; i < 256; ++i)
+                       g_q2p[i] = pow(10., -i / 10.);
+       // initialize others
+       if (b0->n_smpl != b1->n_smpl) return -1; // different number of samples
+       n_smpl = b0->n_smpl;
+       b[0] = b0; b[1] = b1;
+       f[0] = f[1] = f[2] = f[3] = -1.;
+       if (b[0]->n_alleles < 2 || b[1]->n_alleles < 2) return -1; // one allele only
+       // set PL and PL_len
+       for (j = 0; j < 2; ++j) {
+               const bcf1_t *bj = b[j];
+               for (i = 0; i < bj->n_gi; ++i) {
+                       if (bj->gi[i].fmt == bcf_str2int("PL", 2)) {
+                               PL[j] = (uint8_t*)bj->gi[i].data;
+                               PL_len[j] = bj->gi[i].len;
+                               break;
+                       }
+               }
+               if (i == bj->n_gi) return -1; // no PL
+       }
+       // fill pdg[2]
+       pdg[0] = malloc(3 * n_smpl * sizeof(double));
+       pdg[1] = malloc(3 * n_smpl * sizeof(double));
+       for (j = 0; j < 2; ++j) {
+               for (i = 0; i < n_smpl; ++i) {
+                       const uint8_t *pi = PL[j] + i * PL_len[j];
+                       double *p = pdg[j] + i * 3;
+                       p[0] = g_q2p[pi[b[j]->n_alleles]]; p[1] = g_q2p[pi[1]]; p[2] = g_q2p[pi[0]];
+               }
+       }
+       // iteration
+       f[0] = f[1] = f[2] = f[3] = 0.25; // this is a really bad guess...
+       for (j = 0; j < LD_ITER_MAX; ++j) {
+               double eps = 0;
+               memcpy(flast, f, 4 * sizeof(double));
+               freq_iter(n_smpl, pdg, f);
+               for (i = 0; i < 4; ++i) {
+                       double x = fabs(f[0] - flast[0]);
+                       if (x > eps) eps = x;
+               }
+               if (eps < LD_ITER_EPS) break;
+       }
+       // free
+       free(pdg[0]); free(pdg[1]);
+       { // calculate r^2
+               double p[2], q[2], D;
+               p[0] = f[0] + f[1]; q[0] = 1 - p[0];
+               p[1] = f[0] + f[2]; q[1] = 1 - p[1];
+               D = f[0] * f[3] - f[1] * f[2];
+               r = sqrt(D * D / (p[0] * p[1] * q[0] * q[1]));
+               // fprintf(stderr, "R(%lf,%lf,%lf,%lf)=%lf\n", f[0], f[1], f[2], f[3], r2);
+               if (isnan(r)) r = -1.;
+       }
+       return r;
+}

diff --git a/bcftools/main.c b/bcftools/main.c
new file mode 100644 (file)
index 0000000..7ffc2a0
--- /dev/null
+++ b/bcftools/main.c
@@ -0,0 +1,64 @@
+#include <string.h>
+#include <stdlib.h>
+#include <sys/stat.h>
+#include "bcf.h"
+
+int bcfview(int argc, char *argv[]);
+int bcf_main_index(int argc, char *argv[]);
+
+#define BUF_SIZE 0x10000
+
+int bcf_cat(int n, char * const *fn)
+{
+       int i;
+       bcf_t *out;
+       uint8_t *buf;
+       buf = malloc(BUF_SIZE);
+       out = bcf_open("-", "w");
+       for (i = 0; i < n; ++i) {
+               bcf_t *in;
+               bcf_hdr_t *h;
+               off_t end;
+               struct stat s;
+               in = bcf_open(fn[i], "r");
+               h = bcf_hdr_read(in);
+               if (i == 0) bcf_hdr_write(out, h);
+               bcf_hdr_destroy(h);
+#ifdef _USE_KNETFILE
+               fstat(knet_fileno(in->fp->x.fpr), &s);
+               end = s.st_size - 28;
+               while (knet_tell(in->fp->x.fpr) < end) {
+                       int size = knet_tell(in->fp->x.fpr) + BUF_SIZE < end? BUF_SIZE : end - knet_tell(in->fp->x.fpr);
+                       knet_read(in->fp->x.fpr, buf, size);
+                       fwrite(buf, 1, size, out->fp->x.fpw);
+               }
+#else
+               abort(); // FIXME: not implemented
+#endif
+               bcf_close(in);
+       }
+       bcf_close(out);
+       free(buf);
+       return 0;
+}
+
+int main(int argc, char *argv[])
+{
+       if (argc == 1) {
+               fprintf(stderr, "\n");
+               fprintf(stderr, "Usage:   bcftools <command> <arguments>\n\n");
+               fprintf(stderr, "Command: view      print, extract, convert and call SNPs from BCF\n");
+               fprintf(stderr, "         index     index BCF\n");
+               fprintf(stderr, "         cat       concatenate BCFs\n");
+               fprintf(stderr, "\n");
+               return 1;
+       }
+       if (strcmp(argv[1], "view") == 0) return bcfview(argc-1, argv+1);
+       else if (strcmp(argv[1], "index") == 0) return bcf_main_index(argc-1, argv+1);
+       else if (strcmp(argv[1], "cat") == 0) return bcf_cat(argc-2, argv+2);
+       else {
+               fprintf(stderr, "[main] Unrecognized command.\n");
+               return 1;
+       }
+       return 0;
+}

diff --git a/bcftools/prob1.c b/bcftools/prob1.c
new file mode 100644 (file)
index 0000000..e3b0f73
--- /dev/null
+++ b/bcftools/prob1.c
@@ -0,0 +1,394 @@
+#include <math.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stdio.h>
+#include <errno.h>
+#include "prob1.h"
+
+#include "kseq.h"
+KSTREAM_INIT(gzFile, gzread, 16384)
+
+#define MC_AVG_ERR 0.007
+#define MC_MAX_EM_ITER 16
+#define MC_EM_EPS 1e-4
+
+unsigned char seq_nt4_table[256] = {
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4 /*'-'*/, 4, 4,
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 0, 4, 1,  4, 4, 4, 2,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  3, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 0, 4, 1,  4, 4, 4, 2,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  3, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4, 
+       4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4,  4, 4, 4, 4
+};
+
+struct __bcf_p1aux_t {
+       int n, M, n1;
+       double *q2p, *pdg; // pdg -> P(D|g)
+       double *phi;
+       double *z, *zswap; // aux for afs
+       double *z1, *z2, *phi1, *phi2; // only calculated when n1 is set
+       double t, t1, t2;
+       double *afs, *afs1; // afs: accumulative AFS; afs1: site posterior distribution
+       const uint8_t *PL; // point to PL
+       int PL_len;
+};
+
+static void init_prior(int type, double theta, int M, double *phi)
+{
+       int i;
+       if (type == MC_PTYPE_COND2) {
+               for (i = 0; i <= M; ++i)
+                       phi[i] = 2. * (i + 1) / (M + 1) / (M + 2);
+       } else if (type == MC_PTYPE_FLAT) {
+               for (i = 0; i <= M; ++i)
+                       phi[i] = 1. / (M + 1);
+       } else {
+               double sum;
+               for (i = 0, sum = 0.; i < M; ++i)
+                       sum += (phi[i] = theta / (M - i));
+               phi[M] = 1. - sum;
+       }
+}
+
+void bcf_p1_init_prior(bcf_p1aux_t *ma, int type, double theta)
+{
+       init_prior(type, theta, ma->M, ma->phi);
+}
+
+void bcf_p1_init_subprior(bcf_p1aux_t *ma, int type, double theta)
+{
+       if (ma->n1 <= 0 || ma->n1 >= ma->M) return;
+       init_prior(type, theta, 2*ma->n1, ma->phi1);
+       init_prior(type, theta, 2*(ma->n - ma->n1), ma->phi2);
+}
+
+int bcf_p1_read_prior(bcf_p1aux_t *ma, const char *fn)
+{
+       gzFile fp;
+       kstring_t s;
+       kstream_t *ks;
+       long double sum;
+       int dret, k;
+       memset(&s, 0, sizeof(kstring_t));
+       fp = strcmp(fn, "-")? gzopen(fn, "r") : gzdopen(fileno(stdin), "r");
+       ks = ks_init(fp);
+       memset(ma->phi, 0, sizeof(double) * (ma->M + 1));
+       while (ks_getuntil(ks, '\n', &s, &dret) >= 0) {
+               if (strstr(s.s, "[afs] ") == s.s) {
+                       char *p = s.s + 6;
+                       for (k = 0; k <= ma->M; ++k) {
+                               int x;
+                               double y;
+                               x = strtol(p, &p, 10);
+                               if (x != k && (errno == EINVAL || errno == ERANGE)) return -1;
+                               ++p;
+                               y = strtod(p, &p);
+                               if (y == 0. && (errno == EINVAL || errno == ERANGE)) return -1;
+                               ma->phi[ma->M - k] += y;
+                       }
+               }
+       }
+       ks_destroy(ks);
+       gzclose(fp);
+       free(s.s);
+       for (sum = 0., k = 0; k <= ma->M; ++k) sum += ma->phi[k];
+       fprintf(stderr, "[prior]");
+       for (k = 0; k <= ma->M; ++k) ma->phi[k] /= sum;
+       for (k = 0; k <= ma->M; ++k) fprintf(stderr, " %d:%.3lg", k, ma->phi[ma->M - k]);
+       fputc('\n', stderr);
+       for (sum = 0., k = 1; k < ma->M; ++k) sum += ma->phi[ma->M - k] * (2.* k * (ma->M - k) / ma->M / (ma->M - 1));
+       fprintf(stderr, "[%s] heterozygosity=%lf, ", __func__, (double)sum);
+       for (sum = 0., k = 1; k <= ma->M; ++k) sum += k * ma->phi[ma->M - k] / ma->M;
+       fprintf(stderr, "theta=%lf\n", (double)sum);
+       return 0;
+}
+
+bcf_p1aux_t *bcf_p1_init(int n)
+{
+       bcf_p1aux_t *ma;
+       int i;
+       ma = calloc(1, sizeof(bcf_p1aux_t));
+       ma->n1 = -1;
+       ma->n = n; ma->M = 2 * n;
+       ma->q2p = calloc(256, sizeof(double));
+       ma->pdg = calloc(3 * ma->n, sizeof(double));
+       ma->phi = calloc(ma->M + 1, sizeof(double));
+       ma->phi1 = calloc(ma->M + 1, sizeof(double));
+       ma->phi2 = calloc(ma->M + 1, sizeof(double));
+       ma->z = calloc(2 * ma->n + 1, sizeof(double));
+       ma->zswap = calloc(2 * ma->n + 1, sizeof(double));
+       ma->z1 = calloc(ma->M + 1, sizeof(double)); // actually we do not need this large
+       ma->z2 = calloc(ma->M + 1, sizeof(double));
+       ma->afs = calloc(2 * ma->n + 1, sizeof(double));
+       ma->afs1 = calloc(2 * ma->n + 1, sizeof(double));
+       for (i = 0; i < 256; ++i)
+               ma->q2p[i] = pow(10., -i / 10.);
+       bcf_p1_init_prior(ma, MC_PTYPE_FULL, 1e-3); // the simplest prior
+       return ma;
+}
+
+int bcf_p1_set_n1(bcf_p1aux_t *b, int n1)
+{
+       if (n1 == 0 || n1 >= b->n) return -1;
+       b->n1 = n1;
+       return 0;
+}
+
+void bcf_p1_destroy(bcf_p1aux_t *ma)
+{
+       if (ma) {
+               free(ma->q2p); free(ma->pdg);
+               free(ma->phi); free(ma->phi1); free(ma->phi2);
+               free(ma->z); free(ma->zswap); free(ma->z1); free(ma->z2);
+               free(ma->afs); free(ma->afs1);
+               free(ma);
+       }
+}
+
+static int cal_pdg(const bcf1_t *b, bcf_p1aux_t *ma)
+{
+       int i, j, k;
+       long *p, tmp;
+       p = alloca(b->n_alleles * sizeof(long));
+       memset(p, 0, sizeof(long) * b->n_alleles);
+       for (j = 0; j < ma->n; ++j) {
+               const uint8_t *pi = ma->PL + j * ma->PL_len;
+               double *pdg = ma->pdg + j * 3;
+               pdg[0] = ma->q2p[pi[b->n_alleles]]; pdg[1] = ma->q2p[pi[1]]; pdg[2] = ma->q2p[pi[0]];
+               for (i = k = 0; i < b->n_alleles; ++i) {
+                       p[i] += (int)pi[k];
+                       k += b->n_alleles - i;
+               }
+       }
+       for (i = 0; i < b->n_alleles; ++i) p[i] = p[i]<<4 | i;
+       for (i = 1; i < b->n_alleles; ++i) // insertion sort
+               for (j = i; j > 0 && p[j] < p[j-1]; --j)
+                       tmp = p[j], p[j] = p[j-1], p[j-1] = tmp;
+       for (i = b->n_alleles - 1; i >= 0; --i)
+               if ((p[i]&0xf) == 0) break;
+       return i;
+}
+// f0 is the reference allele frequency
+static double mc_freq_iter(double f0, const bcf_p1aux_t *ma)
+{
+       double f, f3[3];
+       int i;
+       f3[0] = (1.-f0)*(1.-f0); f3[1] = 2.*f0*(1.-f0); f3[2] = f0*f0;
+       for (i = 0, f = 0.; i < ma->n; ++i) {
+               double *pdg;
+               pdg = ma->pdg + i * 3;
+               f += (pdg[1] * f3[1] + 2. * pdg[2] * f3[2])
+                       / (pdg[0] * f3[0] + pdg[1] * f3[1] + pdg[2] * f3[2]);
+       }
+       f /= ma->n * 2.;
+       return f;
+}
+
+int bcf_p1_call_gt(const bcf_p1aux_t *ma, double f0, int k)
+{
+       double sum, g[3];
+       double max, f3[3], *pdg = ma->pdg + k * 3;
+       int q, i, max_i;
+       f3[0] = (1.-f0)*(1.-f0); f3[1] = 2.*f0*(1.-f0); f3[2] = f0*f0;
+       for (i = 0, sum = 0.; i < 3; ++i)
+               sum += (g[i] = pdg[i] * f3[i]);
+       for (i = 0, max = -1., max_i = 0; i < 3; ++i) {
+               g[i] /= sum;
+               if (g[i] > max) max = g[i], max_i = i;
+       }
+       max = 1. - max;
+       if (max < 1e-308) max = 1e-308;
+       q = (int)(-4.343 * log(max) + .499);
+       if (q > 99) q = 99;
+       return q<<2|max_i;
+}
+
+#define TINY 1e-20
+
+static void mc_cal_y_core(bcf_p1aux_t *ma, int beg)
+{
+       double *z[2], *tmp, *pdg;
+       int _j, last_min, last_max;
+       z[0] = ma->z;
+       z[1] = ma->zswap;
+       pdg = ma->pdg;
+       memset(z[0], 0, sizeof(double) * (ma->M + 1));
+       memset(z[1], 0, sizeof(double) * (ma->M + 1));
+       z[0][0] = 1.;
+       last_min = last_max = 0;
+       ma->t = 0.;
+       for (_j = beg; _j < ma->n; ++_j) {
+               int k, j = _j - beg, _min = last_min, _max = last_max;
+               double p[3], sum;
+               pdg = ma->pdg + _j * 3;
+               p[0] = pdg[0]; p[1] = 2. * pdg[1]; p[2] = pdg[2];
+               for (; _min < _max && z[0][_min] < TINY; ++_min) z[0][_min] = z[1][_min] = 0.;
+               for (; _max > _min && z[0][_max] < TINY; --_max) z[0][_max] = z[1][_max] = 0.;
+               _max += 2;
+               if (_min == 0) 
+                       k = 0, z[1][k] = (2*j+2-k)*(2*j-k+1) * p[0] * z[0][k];
+               if (_min <= 1)
+                       k = 1, z[1][k] = (2*j+2-k)*(2*j-k+1) * p[0] * z[0][k] + k*(2*j+2-k) * p[1] * z[0][k-1];
+               for (k = _min < 2? 2 : _min; k <= _max; ++k)
+                       z[1][k] = (2*j+2-k)*(2*j-k+1) * p[0] * z[0][k]
+                               + k*(2*j+2-k) * p[1] * z[0][k-1]
+                               + k*(k-1)* p[2] * z[0][k-2];
+               for (k = _min, sum = 0.; k <= _max; ++k) sum += z[1][k];
+               ma->t += log(sum / ((2. * j + 2) * (2. * j + 1)));
+               for (k = _min; k <= _max; ++k) z[1][k] /= sum;
+               if (_min >= 1) z[1][_min-1] = 0.;
+               if (_min >= 2) z[1][_min-2] = 0.;
+               if (j < ma->n - 1) z[1][_max+1] = z[1][_max+2] = 0.;
+               if (_j == ma->n1 - 1) { // set pop1
+                       ma->t1 = ma->t;
+                       memcpy(ma->z1, z[1], sizeof(double) * (ma->n1 * 2 + 1));
+               }
+               tmp = z[0]; z[0] = z[1]; z[1] = tmp;
+               last_min = _min; last_max = _max;
+       }
+       if (z[0] != ma->z) memcpy(ma->z, z[0], sizeof(double) * (ma->M + 1));
+}
+
+static void mc_cal_y(bcf_p1aux_t *ma)
+{
+       if (ma->n1 > 0 && ma->n1 < ma->n) {
+               int k;
+               long double x;
+               memset(ma->z1, 0, sizeof(double) * (2 * ma->n1 + 1));
+               memset(ma->z2, 0, sizeof(double) * (2 * (ma->n - ma->n1) + 1));
+               ma->t1 = ma->t2 = 0.;
+               mc_cal_y_core(ma, ma->n1);
+               ma->t2 = ma->t;
+               memcpy(ma->z2, ma->z, sizeof(double) * (2 * (ma->n - ma->n1) + 1));
+               mc_cal_y_core(ma, 0);
+               // rescale z
+               x = expl(ma->t - (ma->t1 + ma->t2));
+               for (k = 0; k <= ma->M; ++k) ma->z[k] *= x;
+       } else mc_cal_y_core(ma, 0);
+}
+
+static void contrast(bcf_p1aux_t *ma, double pc[4]) // mc_cal_y() must be called before hand
+{
+       int k, n1 = ma->n1, n2 = ma->n - ma->n1;
+       long double sum1, sum2;
+       pc[0] = pc[1] = pc[2] = pc[3] = -1.;
+       if (n1 <= 0 || n2 <= 0) return;
+       for (k = 0, sum1 = 0.; k <= 2*n1; ++k) sum1 += ma->phi1[k] * ma->z1[k];
+       for (k = 0, sum2 = 0.; k <= 2*n2; ++k) sum2 += ma->phi2[k] * ma->z2[k];
+       pc[2] = ma->phi1[2*n1] * ma->z1[2*n1] / sum1;
+       pc[3] = ma->phi2[2*n2] * ma->z2[2*n2] / sum2;
+       for (k = 2; k < 4; ++k) {
+               pc[k] = pc[k] > .5? -(-4.343 * log(1. - pc[k] + TINY) + .499) : -4.343 * log(pc[k] + TINY) + .499;
+               pc[k] = (int)pc[k];
+               if (pc[k] > 99) pc[k] = 99;
+               if (pc[k] < -99) pc[k] = -99;
+       }
+       pc[0] = ma->phi2[2*n2] * ma->z2[2*n2] / sum2 * (1. - ma->phi1[2*n1] * ma->z1[2*n1] / sum1);
+       pc[1] = ma->phi1[2*n1] * ma->z1[2*n1] / sum1 * (1. - ma->phi2[2*n2] * ma->z2[2*n2] / sum2);
+       pc[0] = pc[0] == 1.? 99 : (int)(-4.343 * log(1. - pc[0]) + .499);
+       pc[1] = pc[1] == 1.? 99 : (int)(-4.343 * log(1. - pc[1]) + .499);
+}
+
+static double mc_cal_afs(bcf_p1aux_t *ma)
+{
+       int k;
+       long double sum = 0.;
+       memset(ma->afs1, 0, sizeof(double) * (ma->M + 1));
+       mc_cal_y(ma);
+       for (k = 0, sum = 0.; k <= ma->M; ++k)
+               sum += (long double)ma->phi[k] * ma->z[k];
+       for (k = 0; k <= ma->M; ++k) {
+               ma->afs1[k] = ma->phi[k] * ma->z[k] / sum;
+               if (isnan(ma->afs1[k]) || isinf(ma->afs1[k])) return -1.;
+       }
+       for (k = 0, sum = 0.; k <= ma->M; ++k) {
+               ma->afs[k] += ma->afs1[k];
+               sum += k * ma->afs1[k];
+       }
+       return sum / ma->M;
+}
+
+long double bcf_p1_cal_g3(bcf_p1aux_t *p1a, double g[3])
+{
+       long double pd = 0., g2[3];
+       int i, k;
+       memset(g2, 0, sizeof(long double) * 3);
+       for (k = 0; k < p1a->M; ++k) {
+               double f = (double)k / p1a->M, f3[3], g1[3];
+               long double z = 1.;
+               g1[0] = g1[1] = g1[2] = 0.;
+               f3[0] = (1. - f) * (1. - f); f3[1] = 2. * f * (1. - f); f3[2] = f * f;
+               for (i = 0; i < p1a->n; ++i) {
+                       double *pdg = p1a->pdg + i * 3;
+                       double x = pdg[0] * f3[0] + pdg[1] * f3[1] + pdg[2] * f3[2];
+                       z *= x;
+                       g1[0] += pdg[0] * f3[0] / x;
+                       g1[1] += pdg[1] * f3[1] / x;
+                       g1[2] += pdg[2] * f3[2] / x;
+               }
+               pd += p1a->phi[k] * z;
+               for (i = 0; i < 3; ++i)
+                       g2[i] += p1a->phi[k] * z * g1[i];
+       }
+       for (i = 0; i < 3; ++i) g[i] = g2[i] / pd;
+       return pd;
+}
+
+int bcf_p1_cal(bcf1_t *b, bcf_p1aux_t *ma, bcf_p1rst_t *rst)
+{
+       int i, k;
+       long double sum = 0.;
+       // set PL and PL_len
+       for (i = 0; i < b->n_gi; ++i) {
+               if (b->gi[i].fmt == bcf_str2int("PL", 2)) {
+                       ma->PL = (uint8_t*)b->gi[i].data;
+                       ma->PL_len = b->gi[i].len;
+                       break;
+               }
+       }
+       if (b->n_alleles < 2) return -1; // FIXME: find a better solution
+       // 
+       rst->rank0 = cal_pdg(b, ma);
+       rst->f_exp = mc_cal_afs(ma);
+       rst->p_ref = ma->afs1[ma->M];
+       // calculate f_flat and f_em
+       for (k = 0, sum = 0.; k <= ma->M; ++k)
+               sum += (long double)ma->z[k];
+       rst->f_flat = 0.;
+       for (k = 0; k <= ma->M; ++k) {
+               double p = ma->z[k] / sum;
+               rst->f_flat += k * p;
+       }
+       rst->f_flat /= ma->M;
+       { // calculate f_em
+               double flast = rst->f_flat;
+               for (i = 0; i < MC_MAX_EM_ITER; ++i) {
+                       rst->f_em = mc_freq_iter(flast, ma);
+                       if (fabs(rst->f_em - flast) < MC_EM_EPS) break;
+                       flast = rst->f_em;
+               }
+       }
+       rst->g[0] = rst->g[1] = rst->g[2] = -1.;
+       contrast(ma, rst->pc);
+       return 0;
+}
+
+void bcf_p1_dump_afs(bcf_p1aux_t *ma)
+{
+       int k;
+       fprintf(stderr, "[afs]");
+       for (k = 0; k <= ma->M; ++k)
+               fprintf(stderr, " %d:%.3lf", k, ma->afs[ma->M - k]);
+       fprintf(stderr, "\n");
+       memset(ma->afs, 0, sizeof(double) * (ma->M + 1));
+}

diff --git a/bcftools/prob1.h b/bcftools/prob1.h
new file mode 100644 (file)
index 0000000..7158fe2
--- /dev/null
+++ b/bcftools/prob1.h
@@ -0,0 +1,39 @@
+#ifndef BCF_PROB1_H
+#define BCF_PROB1_H
+
+#include "bcf.h"
+
+struct __bcf_p1aux_t;
+typedef struct __bcf_p1aux_t bcf_p1aux_t;
+
+typedef struct {
+       int rank0;
+       double f_em, f_exp, f_flat, p_ref;
+       double pc[4];
+       double g[3];
+} bcf_p1rst_t;
+
+#define MC_PTYPE_FULL  1
+#define MC_PTYPE_COND2 2
+#define MC_PTYPE_FLAT  3
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+       bcf_p1aux_t *bcf_p1_init(int n);
+       void bcf_p1_init_prior(bcf_p1aux_t *ma, int type, double theta);
+       void bcf_p1_init_subprior(bcf_p1aux_t *ma, int type, double theta);
+       void bcf_p1_destroy(bcf_p1aux_t *ma);
+       int bcf_p1_cal(bcf1_t *b, bcf_p1aux_t *ma, bcf_p1rst_t *rst);
+       int bcf_p1_call_gt(const bcf_p1aux_t *ma, double f0, int k);
+       void bcf_p1_dump_afs(bcf_p1aux_t *ma);
+       int bcf_p1_read_prior(bcf_p1aux_t *ma, const char *fn);
+       long double bcf_p1_cal_g3(bcf_p1aux_t *p1a, double g[3]);
+       int bcf_p1_set_n1(bcf_p1aux_t *b, int n1);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif

diff --git a/bcftools/vcf.c b/bcftools/vcf.c
new file mode 100644 (file)
index 0000000..ebca869
--- /dev/null
+++ b/bcftools/vcf.c
@@ -0,0 +1,212 @@
+#include <zlib.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include "bcf.h"
+#include "kstring.h"
+#include "kseq.h"
+KSTREAM_INIT(gzFile, gzread, 4096)
+
+typedef struct {
+       gzFile fp;
+       FILE *fpout;
+       kstream_t *ks;
+       void *refhash;
+       kstring_t line;
+       int max_ref;
+} vcf_t;
+
+bcf_hdr_t *vcf_hdr_read(bcf_t *bp)
+{
+       kstring_t meta, smpl;
+       int dret;
+       vcf_t *v;
+       bcf_hdr_t *h;
+       if (!bp->is_vcf) return bcf_hdr_read(bp);
+       h = calloc(1, sizeof(bcf_hdr_t));
+       v = (vcf_t*)bp->v;
+       v->line.l = 0;
+       memset(&meta, 0, sizeof(kstring_t));
+       memset(&smpl, 0, sizeof(kstring_t));
+       while (ks_getuntil(v->ks, '\n', &v->line, &dret) >= 0) {
+               if (v->line.l < 2) continue;
+               if (v->line.s[0] != '#') return 0; // no sample line
+               if (v->line.s[0] == '#' && v->line.s[1] == '#') {
+                       kputsn(v->line.s, v->line.l, &meta); kputc('\n', &meta);
+               } else if (v->line.s[0] == '#') {
+                       int k;
+                       ks_tokaux_t aux;
+                       char *p;
+                       for (p = kstrtok(v->line.s, "\t\n", &aux), k = 0; p; p = kstrtok(0, 0, &aux), ++k) {
+                               if (k >= 9) {
+                                       kputsn(p, aux.p - p, &smpl);
+                                       kputc('\0', &smpl);
+                               }
+                       }
+                       break;
+               }
+       }
+       kputc('\0', &meta);
+       h->name = 0;
+       h->sname = smpl.s; h->l_smpl = smpl.l;
+       h->txt = meta.s; h->l_txt = meta.l;
+       bcf_hdr_sync(h);
+       return h;
+}
+
+bcf_t *vcf_open(const char *fn, const char *mode)
+{
+       bcf_t *bp;
+       vcf_t *v;
+       if (strchr(mode, 'b')) return bcf_open(fn, mode);
+       bp = calloc(1, sizeof(bcf_t));
+       v = calloc(1, sizeof(vcf_t));
+       bp->is_vcf = 1;
+       bp->v = v;
+       v->refhash = bcf_str2id_init();
+       if (strchr(mode, 'r')) {
+               v->fp = strcmp(fn, "-")? gzopen(fn, "r") : gzdopen(fileno(stdin), "r");
+               v->ks = ks_init(v->fp);
+       } else if (strchr(mode, 'w'))
+               v->fpout = strcmp(fn, "-")? fopen(fn, "w") : stdout;
+       return bp;
+}
+
+int vcf_close(bcf_t *bp)
+{
+       vcf_t *v;
+       if (bp == 0) return -1;
+       if (!bp->is_vcf) return bcf_close(bp);
+       v = (vcf_t*)bp->v;
+       if (v->fp) {
+               ks_destroy(v->ks);
+               gzclose(v->fp);
+       }
+       if (v->fpout) fclose(v->fpout);
+       free(v->line.s);
+       bcf_str2id_destroy(v->refhash);
+       free(v);
+       free(bp);
+       return 0;
+}
+
+int vcf_hdr_write(bcf_t *bp, const bcf_hdr_t *h)
+{
+       vcf_t *v = (vcf_t*)bp->v;
+       int i, has_ref = 0, has_ver = 0;
+       if (!bp->is_vcf) return bcf_hdr_write(bp, h);
+       if (h->l_txt > 0) {
+               if (strstr(h->txt, "##fileformat=")) has_ver = 1;
+               if (has_ver == 0) fprintf(v->fpout, "##fileformat=VCFv4.0\n");
+               fwrite(h->txt, 1, h->l_txt - 1, v->fpout);
+               if (strstr(h->txt, "##SQ=")) has_ref = 1;
+       }
+       if (has_ver == 0) fprintf(v->fpout, "##fileformat=VCFv4.0\n");
+       fprintf(v->fpout, "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT");
+       for (i = 0; i < h->n_smpl; ++i)
+               fprintf(v->fpout, "\t%s", h->sns[i]);
+       fputc('\n', v->fpout);
+       return 0;
+}
+
+int vcf_write(bcf_t *bp, bcf_hdr_t *h, bcf1_t *b)
+{
+       vcf_t *v = (vcf_t*)bp->v;
+       extern void bcf_fmt_core(const bcf_hdr_t *h, bcf1_t *b, kstring_t *s);
+       if (!bp->is_vcf) return bcf_write(bp, h, b);
+       bcf_fmt_core(h, b, &v->line);
+       fwrite(v->line.s, 1, v->line.l, v->fpout);
+       fputc('\n', v->fpout);
+       return v->line.l + 1;
+}
+
+int vcf_read(bcf_t *bp, bcf_hdr_t *h, bcf1_t *b)
+{
+       int dret, k, i, sync = 0;
+       vcf_t *v = (vcf_t*)bp->v;
+       char *p, *q;
+       kstring_t str, rn;
+       ks_tokaux_t aux, a2;
+       if (!bp->is_vcf) return bcf_read(bp, h, b);
+       v->line.l = 0;
+       str.l = 0; str.m = b->m_str; str.s = b->str;
+       rn.l = rn.m = h->l_nm; rn.s = h->name;
+       if (ks_getuntil(v->ks, '\n', &v->line, &dret) < 0) return -1;
+       b->n_smpl = h->n_smpl;
+       for (p = kstrtok(v->line.s, "\t", &aux), k = 0; p; p = kstrtok(0, 0, &aux), ++k) {
+               *(char*)aux.p = 0;
+               if (k == 0) { // ref
+                       int tid = bcf_str2id(v->refhash, p);
+                       if (tid < 0) {
+                               tid = bcf_str2id_add(v->refhash, p);
+                               kputs(p, &rn); kputc('\0', &rn);
+                               sync = 1;
+                       }
+                       b->tid = tid;
+               } else if (k == 1) { // pos
+                       b->pos = atoi(p) - 1;
+               } else if (k == 5) { // qual
+                       b->qual = (p[0] >= '0' && p[0] <= '9')? atof(p) : 0;
+               } else if (k <= 8) { // variable length strings
+                       kputs(p, &str); kputc('\0', &str);
+                       b->l_str = str.l; b->m_str = str.m; b->str = str.s;
+                       if (k == 8) bcf_sync(b);
+               } else { // k > 9
+                       if (strncmp(p, "./.", 3) == 0) {
+                               for (i = 0; i < b->n_gi; ++i) {
+                                       if (b->gi[i].fmt == bcf_str2int("GT", 2)) {
+                                               ((uint8_t*)b->gi[i].data)[k-9] = 1<<7;
+                                       } else if (b->gi[i].fmt == bcf_str2int("GQ", 2)) {
+                                               ((uint8_t*)b->gi[i].data)[k-9] = 0;
+                                       } else if (b->gi[i].fmt == bcf_str2int("DP", 2)) {
+                                               ((uint16_t*)b->gi[i].data)[k-9] = 0;
+                                       } else if (b->gi[i].fmt == bcf_str2int("PL", 2)) {
+                                               int y = b->n_alleles * (b->n_alleles + 1) / 2;
+                                               memset((uint8_t*)b->gi[i].data + (k - 9) * y, 0, y);
+                                       } else if (b->gi[i].fmt == bcf_str2int("GL", 2)) {
+                                               int y = b->n_alleles * (b->n_alleles + 1) / 2;
+                                               memset((float*)b->gi[i].data + (k - 9) * y, 0, y * 4);
+                                       }
+                               }
+                               goto endblock;
+                       }
+                       for (q = kstrtok(p, ":", &a2), i = 0; q && i < b->n_gi; q = kstrtok(0, 0, &a2), ++i) {
+                               if (b->gi[i].fmt == bcf_str2int("GT", 2)) {
+                                       ((uint8_t*)b->gi[i].data)[k-9] = (q[0] - '0')<<3 | (q[2] - '0') | (q[1] == '/'? 0 : 1) << 6;
+                               } else if (b->gi[i].fmt == bcf_str2int("GQ", 2)) {
+                                       double _x = strtod(q, &q);
+                                       int x = (int)(_x + .499);
+                                       if (x > 255) x = 255;
+                                       ((uint8_t*)b->gi[i].data)[k-9] = x;
+                               } else if (b->gi[i].fmt == bcf_str2int("DP", 2)) {
+                                       int x = strtol(q, &q, 10);
+                                       if (x > 0xffff) x = 0xffff;
+                                       ((uint16_t*)b->gi[i].data)[k-9] = x;
+                               } else if (b->gi[i].fmt == bcf_str2int("PL", 2)) {
+                                       int x, y, j;
+                                       uint8_t *data = (uint8_t*)b->gi[i].data;
+                                       y = b->n_alleles * (b->n_alleles + 1) / 2;
+                                       for (j = 0; j < y; ++j) {
+                                               x = strtol(q, &q, 10);
+                                               if (x > 255) x = 255;
+                                               data[(k-9) * y + j] = x;
+                                               ++q;
+                                       }
+                               } else if (b->gi[i].fmt == bcf_str2int("GL", 2)) {
+                                       int j, y;
+                                       float x, *data = (float*)b->gi[i].data;
+                                       y = b->n_alleles * (b->n_alleles + 1) / 2;
+                                       for (j = 0; j < y; ++j) {
+                                               x = strtod(q, &q);
+                                               data[(k-9) * y + j] = x;
+                                               ++q;
+                                       }
+                               }
+                       }
+               endblock: i = i;
+               }
+       }
+       h->l_nm = rn.l; h->name = rn.s;
+       if (sync) bcf_hdr_sync(h);
+       return v->line.l + 1;
+}

diff --git a/bcftools/vcfutils.pl b/bcftools/vcfutils.pl
new file mode 100755 (executable)
index 0000000..d0b7971
--- /dev/null
+++ b/bcftools/vcfutils.pl
@@ -0,0 +1,477 @@
+#!/usr/bin/perl -w
+
+# Author: lh3
+
+use strict;
+use warnings;
+use Getopt::Std;
+
+&main;
+exit;
+
+sub main {
+  my $version = '0.1.0';
+  &usage if (@ARGV < 1);
+  my $command = shift(@ARGV);
+  my %func = (subsam=>\&subsam, listsam=>\&listsam, fillac=>\&fillac, qstats=>\&qstats, varFilter=>\&varFilter,
+                         hapmap2vcf=>\&hapmap2vcf, ucscsnp2vcf=>\&ucscsnp2vcf, filter4vcf=>\&filter4vcf, ldstats=>\&ldstats);
+  die("Unknown command \"$command\".\n") if (!defined($func{$command}));
+  &{$func{$command}};
+}
+
+sub subsam {
+  die(qq/Usage: vcfutils.pl subsam <in.vcf> [samples]\n/) if (@ARGV == 0);
+  my ($fh, %h);
+  my $fn = shift(@ARGV);
+  my @col;
+  open($fh, ($fn =~ /\.gz$/)? "gzip -dc $fn |" : $fn) || die;
+  $h{$_} = 1 for (@ARGV);
+  while (<$fh>) {
+       if (/^##/) {
+         print;
+       } elsif (/^#/) {
+         my @t = split;
+         my @s = @t[0..8]; # all fixed fields + FORMAT
+         for (9 .. $#t) {
+               if ($h{$t[$_]}) {
+                 push(@s, $t[$_]);
+                 push(@col, $_);
+               }
+         }
+         pop(@s) if (@s == 9); # no sample selected; remove the FORMAT field
+         print join("\t", @s), "\n";
+       } else {
+         my @t = split;
+         if (@col == 0) {
+               print join("\t", @t[0..7]), "\n";
+         } else {
+               print join("\t", @t[0..8], map {$t[$_]} @col), "\n";
+         }
+       }
+  }
+  close($fh);
+}
+
+sub listsam {
+  die(qq/Usage: vcfutils.pl listsam <in.vcf>\n/) if (@ARGV == 0 && -t STDIN);
+  while (<>) {
+       if (/^#/ && !/^##/) {
+         my @t = split;
+         print join("\n", @t[9..$#t]), "\n";
+         exit;
+       }
+  }
+}
+
+sub fillac {
+  die(qq/Usage: vcfutils.pl fillac <in.vcf>\n\nNote: The GT field MUST BE present and always appear as the first field.\n/) if (@ARGV == 0 && -t STDIN);
+  while (<>) {
+       if (/^#/) {
+         print;
+       } else {
+         my @t = split;
+         my @c = (0);
+         my $n = 0;
+         my $s = -1;
+         @_ = split(":", $t[8]);
+         for (0 .. $#_) {
+               if ($_[$_] eq 'GT') { $s = $_; last; }
+         }
+         if ($s < 0) {
+               print join("\t", @t), "\n";
+               next;
+         }
+         for (9 .. $#t) {
+               if ($t[$_] =~ /^0,0,0/) {
+               } elsif ($t[$_] =~ /^([^\s:]+:){$s}(\d+).(\d+)/) {
+                 ++$c[$2]; ++$c[$3];
+                 $n += 2;
+               }
+         }
+         my $AC = "AC=" . join("\t", @c[1..$#c]) . ";AN=$n";
+         my $info = $t[7];
+         $info =~ s/(;?)AC=(\d+)//;
+         $info =~ s/(;?)AN=(\d+)//;
+         if ($info eq '.') {
+               $info = $AC;
+         } else {
+               $info .= ";$AC";
+         }
+         $t[7] = $info;
+         print join("\t", @t), "\n";
+       }
+  }
+}
+
+sub ldstats {
+  my %opts = (t=>0.9);
+  getopts('t:', \%opts);
+  die("Usage: vcfutils.pl ldstats [-t $opts{t}] <in.vcf>\n") if (@ARGV == 0 && -t STDIN);
+  my $cutoff = $opts{t};
+  my ($last, $lastchr) = (0x7fffffff, '');
+  my ($x, $y, $n) = (0, 0, 0);
+  while (<>) {
+       if (/^([^#\s]+)\s(\d+)/) {
+         my ($chr, $pos) = ($1, $2);
+         if (/NEIR=([\d\.]+)/) {
+               ++$n;
+               ++$y, $x += $pos - $last if ($lastchr eq $chr && $pos > $last && $1 > $cutoff);
+         }
+         $last = $pos; $lastchr = $chr;
+       }
+  }
+  print "Number of SNP intervals in strong LD (r > $opts{t}): $y\n";
+  print "Fraction: ", $y/$n, "\n";
+  print "Length: $x\n";
+}
+
+sub qstats {
+  my %opts = (r=>'', s=>0.02, v=>undef);
+  getopts('r:s:v', \%opts);
+  die("Usage: vcfutils.pl qstats [-r ref.vcf] <in.vcf>\n
+Note: This command discards indels. Output: QUAL #non-indel #SNPs #transitions #joint ts/tv #joint/#ref #joint/#non-indel \n") if (@ARGV == 0 && -t STDIN);
+  my %ts = (AG=>1, GA=>1, CT=>1, TC=>1);
+  my %h = ();
+  my $is_vcf = defined($opts{v})? 1 : 0;
+  if ($opts{r}) { # read the reference positions
+       my $fh;
+       open($fh, $opts{r}) || die;
+       while (<$fh>) {
+         next if (/^#/);
+         if ($is_vcf) {
+               my @t = split;
+               $h{$t[0],$t[1]} = $t[4];
+         } else {
+               $h{$1,$2} = 1 if (/^(\S+)\s+(\d+)/);
+         }
+       }
+       close($fh);
+  }
+  my $hsize = scalar(keys %h);
+  my @a;
+  while (<>) {
+       next if (/^#/);
+       my @t = split;
+       next if (length($t[3]) != 1 || uc($t[3]) eq 'N');
+       $t[3] = uc($t[3]); $t[4] = uc($t[4]);
+       my @s = split(',', $t[4]);
+       $t[5] = 3 if ($t[5] < 0);
+       next if (length($s[0]) != 1);
+       my $hit;
+       if ($is_vcf) {
+         $hit = 0;
+         my $aa = $h{$t[0],$t[1]};
+         if (defined($aa)) {
+               my @aaa = split(",", $aa);
+               for (@aaa) {
+                 $hit = 1 if ($_ eq $s[0]);
+               }
+         }
+       } else {
+         $hit = defined($h{$t[0],$t[1]})? 1 : 0;
+       }
+       push(@a, [$t[5], ($t[4] eq '.' || $t[4] eq $t[3])? 0 : 1, $ts{$t[3].$s[0]}? 1 : 0, $hit]);
+  }
+  push(@a, [-1, 0, 0, 0]); # end marker
+  die("[qstats] No SNP data!\n") if (@a == 0);
+  @a = sort {$b->[0]<=>$a->[0]} @a;
+  my $next = $opts{s};
+  my $last = $a[0];
+  my @c = (0, 0, 0, 0);
+  my @lc;
+  $lc[1] = $lc[2] = 0;
+  for my $p (@a) {
+       if ($p->[0] == -1 || ($p->[0] != $last && $c[0]/@a > $next)) {
+         my @x;
+         $x[0] = sprintf("%.4f", $c[1]-$c[2]? $c[2] / ($c[1] - $c[2]) : 100);
+         $x[1] = sprintf("%.4f", $hsize? $c[3] / $hsize : 0);
+         $x[2] = sprintf("%.4f", $c[3] / $c[1]);
+         my $a = $c[1] - $lc[1];
+         my $b = $c[2] - $lc[2];
+         $x[3] = sprintf("%.4f", $a-$b? $b / ($a-$b) : 100);
+         print join("\t", $last, @c, @x), "\n";
+         $next = $c[0]/@a + $opts{s};
+         $lc[1] = $c[1]; $lc[2] = $c[2];
+       }
+       ++$c[0]; $c[1] += $p->[1]; $c[2] += $p->[2]; $c[3] += $p->[3];
+       $last = $p->[0];
+  }
+}
+
+sub varFilter {
+  my %opts = (d=>1, D=>10000, l=>30, Q=>25, q=>10, G=>25, s=>100, w=>10, W=>10, N=>2, p=>undef, F=>.001);
+  getopts('pq:d:D:l:Q:w:W:N:G:F:', \%opts);
+  die(qq/
+Usage:   vcfutils.pl varFilter [options] <in.vcf>
+
+Options: -Q INT    minimum RMS mapping quality for SNPs [$opts{Q}]
+         -q INT    minimum RMS mapping quality for gaps [$opts{q}]
+         -d INT    minimum read depth [$opts{d}]
+         -D INT    maximum read depth [$opts{D}]
+
+         -G INT    min indel score for nearby SNP filtering [$opts{G}]
+         -w INT    SNP within INT bp around a gap to be filtered [$opts{w}]
+
+         -W INT    window size for filtering dense SNPs [$opts{W}]
+         -N INT    max number of SNPs in a window [$opts{N}]
+
+         -l INT    window size for filtering adjacent gaps [$opts{l}]
+
+         -p        print filtered variants
+\n/) if (@ARGV == 0 && -t STDIN);
+
+  # calculate the window size
+  my ($ol, $ow, $oW) = ($opts{l}, $opts{w}, $opts{W});
+  my $max_dist = $ol > $ow? $ol : $ow;
+  $max_dist = $oW if ($max_dist < $oW);
+  # the core loop
+  my @staging; # (indel_filtering_score, flt_tag)
+  while (<>) {
+       my @t = split;
+       next if (/^#/);
+       next if ($t[4] eq '.'); # skip non-var sites
+       my $is_snp = 1;
+       if (length($t[3]) > 1) {
+         $is_snp = 0;
+       } else {
+         my @s = split(',', $t[4]);
+         for (@s) {
+               $is_snp = 0 if (length > 1);
+         }
+       }
+       # clear the out-of-range elements
+       while (@staging) {
+      # Still on the same chromosome and the first element's window still affects this position?
+         last if ($staging[0][3] eq $t[0] && $staging[0][4] + $staging[0][2] + $max_dist >= $t[1]);
+         varFilter_aux(shift(@staging), $opts{p}); # calling a function is a bit slower, not much
+       }
+       my ($flt, $score) = (0, -1);
+
+       # collect key annotations
+       my ($dp, $mq, $af) = (-1, -1, 1);
+       if ($t[7] =~ /DP=(\d+)/i) {
+         $dp = $1;
+       } elsif ($t[7] =~ /DP4=(\d+),(\d+),(\d+),(\d+)/i) {
+         $dp = $1 + $2 + $3 + $4;
+       }
+       if ($t[7] =~ /MQ=(\d+)/i) {
+         $mq = $1;
+       }
+       if ($t[7] =~ /AF=([^\s;=]+)/i) {
+         $af = $1;
+       } elsif ($t[7] =~ /AF1=([^\s;=]+)/i) {
+         $af = $1;
+       }
+       # the depth filter
+       if ($dp >= 0) {
+         if ($dp < $opts{d}) {
+               $flt = 2;
+         } elsif ($dp > $opts{D}) {
+               $flt = 3;
+         }
+       }
+
+       # site dependent filters
+       my $dlen = 0;
+       if ($flt == 0) {
+         if (!$is_snp) { # an indel
+        # If deletion, remember the length of the deletion
+               $dlen = length($t[3]) - 1;
+               $flt = 1 if ($mq < $opts{q});
+               # filtering SNPs
+               if ($t[5] >= $opts{G}) {
+                 for my $x (@staging) {
+            # Is it a SNP and is it outside the SNP filter window?
+                       next if ($x->[0] >= 0 || $x->[4] + $x->[2] + $ow < $t[1]);
+                       $x->[1] = 5 if ($x->[1] == 0);
+                 }
+               }
+               # the indel filtering score
+               $score = $t[5];
+               # check the staging list for indel filtering
+               for my $x (@staging) {
+          # Is it a SNP and is it outside the gap filter window
+                 next if ($x->[0] < 0 || $x->[4] + $x->[2] + $ol < $t[1]);
+                 if ($x->[0] < $score) {
+                       $x->[1] = 6;
+                 } else {
+                       $flt = 6; last;
+                 }
+               }
+         } else { # a SNP
+               $flt = 1 if ($mq < $opts{Q});
+               # check adjacent SNPs
+               my $k = 1;
+               for my $x (@staging) {
+                 ++$k if ($x->[0] < 0 && -($x->[0] + 1) > $opts{F} && $x->[4] + $x->[2] + $oW >= $t[1] && ($x->[1] == 0 || $x->[1] == 4 || $x->[1] == 5));
+               }
+               # filtering is necessary
+               if ($k > $opts{N}) {
+                 $flt = 4;
+                 for my $x (@staging) {
+                        $x->[1] = 4 if ($x->[0] < 0 && $x->[4] + $x->[2] + $oW >= $t[1] && $x->[1] == 0);
+                 }
+               } else { # then check gap filter
+                 for my $x (@staging) {
+                       next if ($x->[0] < 0 || $x->[4] + $x->[2] + $ow < $t[1]);
+                       if ($x->[0] >= $opts{G}) {
+                         $flt = 5; last;
+                       }
+                 }
+               }
+         }
+       }
+       push(@staging, [$score < 0? -$af-1 : $score, $flt, $dlen, @t]);
+  }
+  # output the last few elements in the staging list
+  while (@staging) {
+       varFilter_aux(shift @staging, $opts{p});
+  }
+}
+
+sub varFilter_aux {
+  my ($first, $is_print) = @_;
+  if ($first->[1] == 0) {
+       print join("\t", @$first[3 .. @$first-1]), "\n";
+  } elsif ($is_print) {
+       print STDERR join("\t", substr("UQdDWGgsiX", $first->[1], 1), @$first[3 .. @$first-1]), "\n";
+  }
+}
+
+sub filter4vcf {
+  my %opts = (d=>3, D=>2000, 1=>1e-4, 2=>1e-100, 3=>0, 4=>1e-4, Q=>10, q=>3);
+  getopts('d:D:1:2:3:4:Q:q:', \%opts);
+  die(qq/
+Usage:   vcfutils.pl filter4vcf [options] <in.vcf>
+
+Options: -d INT     min total depth (given DP or DP4) [$opts{d}]
+         -D INT     max total depth [$opts{D}]
+         -q INT     min SNP quality [$opts{q}]
+         -Q INT     min RMS mapQ (given MQ) [$opts{Q}]
+         -1 FLOAT   min P-value for strand bias (given PV4) [$opts{1}]
+         -2 FLOAT   min P-value for baseQ bias [$opts{2}]
+         -3 FLOAT   min P-value for mapQ bias [$opts{3}]
+         -4 FLOAT   min P-value for end distance bias [$opts{4}]\n
+/) if (@ARGV == 0 && -t STDIN);
+
+  my %ts = (AG=>1, GA=>1, CT=>1, TC=>1);
+
+  my @n = (0, 0);
+  while (<>) {
+       if (/^#/) {
+         print;
+         next;
+       }
+       next if (/PV4=([^,]+),([^,]+),([^,]+),([^,;\t]+)/ && ($1<$opts{1} || $2<$opts{2} || $3<$opts{3} || $4<$opts{4}));
+       my $depth = -1;
+       $depth = $1 if (/DP=(\d+)/);
+       $depth = $1+$2+$3+$4 if (/DP4=(\d+),(\d+),(\d+),(\d+)/);
+       next if ($depth > 0 && ($depth < $opts{d} || $depth > $opts{D}));
+       next if (/MQ=(\d+)/ && $1 < $opts{Q});
+       my @t = split;
+       next if ($t[5] >= 0 && $t[5] < $opts{q});
+       ++$n[0];
+       my @s = split(',', $t[4]);
+       ++$n[1] if ($ts{$t[3].$s[0]});
+       print;
+  }
+}
+
+sub ucscsnp2vcf {
+  die("Usage: vcfutils.pl <in.ucsc.snp>\n") if (@ARGV == 0 && -t STDIN);
+  print "##fileformat=VCFv4.0\n";
+  print join("\t", "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO"), "\n";
+  while (<>) {
+       my @t = split("\t");
+       my $indel = ($t[9] =~ /^[ACGT](\/[ACGT])+$/)? 0 : 1;
+       my $pos = $t[2] + 1;
+       my @alt;
+       push(@alt, $t[7]);
+       if ($t[6] eq '-') {
+         $t[9] = reverse($t[9]);
+         $t[9] =~ tr/ACGTRYMKWSNacgtrymkwsn/TGCAYRKMWSNtgcayrkmwsn/;
+       }
+       my @a = split("/", $t[9]);
+       for (@a) {
+         push(@alt, $_) if ($_ ne $alt[0]);
+       }
+       if ($indel) {
+         --$pos;
+         for (0 .. $#alt) {
+               $alt[$_] =~ tr/-//d;
+               $alt[$_] = "N$alt[$_]";
+         }
+       }
+       my $ref = shift(@alt);
+       my $af = $t[13] > 0? ";AF=$t[13]" : '';
+       my $valid = ($t[12] eq 'unknown')? '' : ";valid=$t[12]";
+       my $info = "molType=$t[10];class=$t[11]$valid$af";
+       print join("\t", $t[1], $pos, $t[4], $ref, join(",", @alt), 0, '.', $info), "\n";
+  }
+}
+
+sub hapmap2vcf {
+  die("Usage: vcfutils.pl <in.ucsc.snp> <in.hapmap>\n") if (@ARGV == 0);
+  my $fn = shift(@ARGV);
+  # parse UCSC SNP
+  warn("Parsing UCSC SNPs...\n");
+  my ($fh, %map);
+  open($fh, ($fn =~ /\.gz$/)? "gzip -dc $fn |" : $fn) || die;
+  while (<$fh>) {
+       my @t = split;
+       next if ($t[3] - $t[2] != 1); # not SNP
+       @{$map{$t[4]}} = @t[1,3,7];
+  }
+  close($fh);
+  # write VCF
+  warn("Writing VCF...\n");
+  print "##fileformat=VCFv4.0\n";
+  while (<>) {
+       my @t = split;
+       if ($t[0] eq 'rs#') { # the first line
+         print join("\t", "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT", @t[11..$#t]), "\n";
+       } else {
+         next unless ($map{$t[0]});
+         next if (length($t[1]) != 3); # skip non-SNPs
+         my $a = \@{$map{$t[0]}};
+         my $ref = $a->[2];
+         my @u = split('/', $t[1]);
+         if ($u[1] eq $ref) {
+               $u[1] = $u[0]; $u[0] = $ref;
+         } elsif ($u[0] ne $ref) { next; }
+         my $alt = $u[1];
+         my %w;
+         $w{$u[0]} = 0; $w{$u[1]} = 1;
+         my @s = (@$a[0,1], $t[0], $ref, $alt, 0, '.', '.', 'GT');
+         my $is_tri = 0;
+         for (@t[11..$#t]) {
+               if ($_ eq 'NN') {
+                 push(@s, './.');
+               } else {
+                 my @a = ($w{substr($_,0,1)}, $w{substr($_,1,1)});
+                 if (!defined($a[0]) || !defined($a[1])) {
+                       $is_tri = 1;
+                       last;
+                 }
+                 push(@s, "$a[0]/$a[1]");
+               }
+         }
+         next if ($is_tri);
+         print join("\t", @s), "\n";
+       }
+  }
+}
+
+sub usage {
+  die(qq/
+Usage:   vcfutils.pl <command> [<arguments>]\n
+Command: subsam       get a subset of samples
+         listsam      list the samples
+         fillac       fill the allele count field
+         qstats       SNP stats stratified by QUAL
+         varFilter    filtering short variants
+         filter4vcf   filtering VCFs produced by samtools+bcftools
+         hapmap2vcf   convert the hapmap format to VCF
+         ucscsnp2vcf  convert UCSC SNP SQL dump to VCF
+\n/);
+}

diff --git a/bgzf.c b/bgzf.c
index a6923daa0cc1cfa8d0d0bc2269255326be8439e3..66d6b024f5441f694faf1d875febe9fe8e3c248c 100644 (file)
--- a/bgzf.c
+++ b/bgzf.c
@@ -177,7 +177,7 @@ BGZF*
 bgzf_open(const char* __restrict path, const char* __restrict mode)
 {
     BGZF* fp = NULL;
 bgzf_open(const char* __restrict path, const char* __restrict mode)
 {
     BGZF* fp = NULL;

-    if (mode[0] == 'r' || mode[0] == 'R') { /* The reading mode is preferred. */
+    if (strchr(mode, 'r') || strchr(mode, 'R')) { /* The reading mode is preferred. */

 #ifdef _USE_KNETFILE
                knetFile *file = knet_open(path, mode);
                if (file == 0) return 0;
 #ifdef _USE_KNETFILE
                knetFile *file = knet_open(path, mode);
                if (file == 0) return 0;


@@ -194,14 +194,14 @@ bgzf_open(const char* __restrict path, const char* __restrict mode)
                if (fd == -1) return 0;
         fp = open_read(fd);
 #endif
                if (fd == -1) return 0;
         fp = open_read(fd);
 #endif

-    } else if (mode[0] == 'w' || mode[0] == 'W') {
+    } else if (strchr(mode, 'w') || strchr(mode, 'W')) {

                int fd, oflag = O_WRONLY | O_CREAT | O_TRUNC;
 #ifdef _WIN32
                oflag |= O_BINARY;
 #endif
                fd = open(path, oflag, 0666);
                if (fd == -1) return 0;
                int fd, oflag = O_WRONLY | O_CREAT | O_TRUNC;
 #ifdef _WIN32
                oflag |= O_BINARY;
 #endif
                fd = open(path, oflag, 0666);
                if (fd == -1) return 0;

-        fp = open_write(fd, strstr(mode, "u")? 1 : 0);
+        fp = open_write(fd, strchr(mode, 'u')? 1 : 0);

     }
     if (fp != NULL) fp->owned_file = 1;
     return fp;
     }
     if (fp != NULL) fp->owned_file = 1;
     return fp;

diff --git a/bgzip.c b/bgzip.c
index ac2a98ea35eb2503d94652918054575b341f5cc1..ebcafa20651fd2b3a9863bfe38dc590497099671 100644 (file)
--- a/bgzip.c
+++ b/bgzip.c
@@ -27,22 +27,24 @@
 #include <fcntl.h>
 #include <unistd.h>
 #include <errno.h>
 #include <fcntl.h>
 #include <unistd.h>
 #include <errno.h>

+#include <sys/select.h>
+#include <sys/stat.h>

 #include "bgzf.h"
 
 static const int WINDOW_SIZE = 64 * 1024;
 
 static int bgzip_main_usage()
 {
 #include "bgzf.h"
 
 static const int WINDOW_SIZE = 64 * 1024;
 
 static int bgzip_main_usage()
 {

-       printf("\n");
-       printf("Usage:   bgzip [options] [file] ...\n\n");
-       printf("Options: -c      write on standard output, keep original files unchanged\n");
-       printf("         -d      decompress\n");
-       // printf("         -l      list compressed file contents\n");
-       printf("         -b INT  decompress at virtual file pointer INT\n");
-       printf("         -s INT  decompress INT bytes in the uncompressed file\n");
-       printf("         -h      give this help\n");
-       printf("\n");
-       return 0;
+       fprintf(stderr, "\n");
+       fprintf(stderr, "Usage:   bgzip [options] [file] ...\n\n");
+       fprintf(stderr, "Options: -c      write on standard output, keep original files unchanged\n");
+       fprintf(stderr, "         -d      decompress\n");
+       fprintf(stderr, "         -f      overwrite files without asking\n");
+       fprintf(stderr, "         -b INT  decompress at virtual file pointer INT\n");
+       fprintf(stderr, "         -s INT  decompress INT bytes in the uncompressed file\n");
+       fprintf(stderr, "         -h      give this help\n");
+       fprintf(stderr, "\n");
+       return 1;

 }
 
 static int write_open(const char *fn, int is_forced)
 }
 
 static int write_open(const char *fn, int is_forced)


@@ -51,129 +53,154 @@ static int write_open(const char *fn, int is_forced)
        char c;
        if (!is_forced) {
                if ((fd = open(fn, O_WRONLY | O_CREAT | O_TRUNC | O_EXCL, 0666)) < 0 && errno == EEXIST) {
        char c;
        if (!is_forced) {
                if ((fd = open(fn, O_WRONLY | O_CREAT | O_TRUNC | O_EXCL, 0666)) < 0 && errno == EEXIST) {

-                       printf("bgzip: %s already exists; do you wish to overwrite (y or n)? ", fn);
+                       fprintf(stderr, "[bgzip] %s already exists; do you wish to overwrite (y or n)? ", fn);

                        scanf("%c", &c);
                        if (c != 'Y' && c != 'y') {
                        scanf("%c", &c);
                        if (c != 'Y' && c != 'y') {

-                               printf("bgzip: not overwritten\n");
+                               fprintf(stderr, "[bgzip] not overwritten\n");

                                exit(1);
                        }
                }
        }
        if (fd < 0) {
                if ((fd = open(fn, O_WRONLY | O_CREAT | O_TRUNC, 0666)) < 0) {
                                exit(1);
                        }
                }
        }
        if (fd < 0) {
                if ((fd = open(fn, O_WRONLY | O_CREAT | O_TRUNC, 0666)) < 0) {

-                       fprintf(stderr, "bgzip: %s: Fail to write\n", fn);
+                       fprintf(stderr, "[bgzip] %s: Fail to write\n", fn);

                        exit(1);
                }
        }
        return fd;
 }
 
                        exit(1);
                }
        }
        return fd;
 }
 

-static
-void
-fail(BGZF* fp)
+static void fail(BGZF* fp)

 {
 {

-    printf("Error: %s\n", fp->error);
+    fprintf(stderr, "Error: %s\n", fp->error);

     exit(1);
 }
 
 int main(int argc, char **argv)
 {
        int c, compress, pstdout, is_forced;
     exit(1);
 }
 
 int main(int argc, char **argv)
 {
        int c, compress, pstdout, is_forced;

-       BGZF *rz;
+       BGZF *fp;

        void *buffer;
        long start, end, size;
 
        compress = 1; pstdout = 0; start = 0; size = -1; end = -1; is_forced = 0;
        void *buffer;
        long start, end, size;
 
        compress = 1; pstdout = 0; start = 0; size = -1; end = -1; is_forced = 0;

-       while((c  = getopt(argc, argv, "cdlhfb:s:")) >= 0){
+       while((c  = getopt(argc, argv, "cdhfb:s:")) >= 0){

                switch(c){
                case 'h': return bgzip_main_usage();
                case 'd': compress = 0; break;
                case 'c': pstdout = 1; break;
                switch(c){
                case 'h': return bgzip_main_usage();
                case 'd': compress = 0; break;
                case 'c': pstdout = 1; break;

-                // case 'l': compress = 2; break;

                case 'b': start = atol(optarg); break;
                case 's': size = atol(optarg); break;
                case 'f': is_forced = 1; break;
                }
        }
        if (size >= 0) end = start + size;
                case 'b': start = atol(optarg); break;
                case 's': size = atol(optarg); break;
                case 'f': is_forced = 1; break;
                }
        }
        if (size >= 0) end = start + size;

-       if(end >= 0 && end < start){
-               fprintf(stderr, " -- Illegal region: [%ld, %ld] --\n", start, end);
+       if (end >= 0 && end < start) {
+               fprintf(stderr, "[bgzip] Illegal region: [%ld, %ld]\n", start, end);

                return 1;
        }
                return 1;
        }

-       if(compress == 1){
-               int f_src, f_dst = -1;
-               if(argc > optind){
-                       if((f_src = open(argv[optind], O_RDONLY)) < 0){
-                               fprintf(stderr, " -- Cannot open file: %s --\n", argv[optind]);
+       if (compress == 1) {
+               struct stat sbuf;
+               int f_src = fileno(stdin);
+               int f_dst = fileno(stdout);
+
+               if ( argc>optind )
+               {
+                       if ( stat(argv[optind],&sbuf)<0 ) 
+                       { 
+                               fprintf(stderr, "[bgzip] %s: %s\n", strerror(errno), argv[optind]);
+                               return 1; 
+                       }
+
+                       if ((f_src = open(argv[optind], O_RDONLY)) < 0) {
+                               fprintf(stderr, "[bgzip] %s: %s\n", strerror(errno), argv[optind]);

                                return 1;
                        }
                                return 1;
                        }

-                       if(pstdout){
+
+                       if (pstdout)

                                f_dst = fileno(stdout);
                                f_dst = fileno(stdout);

-                       } else {
-                               char *name = malloc(sizeof(strlen(argv[optind]) + 5));
+                       else
+                       {
+                               char *name = malloc(strlen(argv[optind]) + 5);

                                strcpy(name, argv[optind]);
                                strcat(name, ".gz");
                                f_dst = write_open(name, is_forced);
                                if (f_dst < 0) return 1;
                                free(name);
                        }
                                strcpy(name, argv[optind]);
                                strcat(name, ".gz");
                                f_dst = write_open(name, is_forced);
                                if (f_dst < 0) return 1;
                                free(name);
                        }

-               } else if(pstdout){ 
-                       f_src = fileno(stdin);
-                       f_dst = fileno(stdout);
-               } else return bgzip_main_usage();
-               rz = bgzf_fdopen(f_dst, "w");
+               }
+               else if (!pstdout && isatty(fileno((FILE *)stdout)) )
+                       return bgzip_main_usage();
+
+               fp = bgzf_fdopen(f_dst, "w");

                buffer = malloc(WINDOW_SIZE);
                buffer = malloc(WINDOW_SIZE);

-               while((c = read(f_src, buffer, WINDOW_SIZE)) > 0) {
-                  if (bgzf_write(rz, buffer, c) < 0) {
-                    fail(rz);
-                  }
-                }
-                // f_dst will be closed here
-               if (bgzf_close(rz) < 0) {
-                  fail(rz);
-                }
-               if (argc > optind) unlink(argv[optind]);
+               while ((c = read(f_src, buffer, WINDOW_SIZE)) > 0)
+                       if (bgzf_write(fp, buffer, c) < 0) fail(fp);
+               // f_dst will be closed here
+               if (bgzf_close(fp) < 0) fail(fp);
+               if (argc > optind && !pstdout) unlink(argv[optind]);

                free(buffer);
                close(f_src);
                return 0;
        } else {
                free(buffer);
                close(f_src);
                return 0;
        } else {

-               if(argc <= optind) return bgzip_main_usage();
-                int f_dst;
-                if (argc > optind && !pstdout) {
-                  char *name;
-                  if (strstr(argv[optind], ".gz") - argv[optind] != strlen(argv[optind]) - 3) {
-                    printf("bgzip: %s: unknown suffix -- ignored\n", argv[optind]);
-                    return 1;
-                  }
-                  name = strdup(argv[optind]);
-                  name[strlen(name) - 3] = '\0';
-                  f_dst = write_open(name, is_forced);
-                  free(name);
-                } else f_dst = fileno(stdout);
-                rz = bgzf_open(argv[optind], "r");
-                if (rz == NULL) {
-                  printf("Could not open file: %s\n", argv[optind]);
-                  return 1;
-                }
-                buffer = malloc(WINDOW_SIZE);
-                if (bgzf_seek(rz, start, SEEK_SET) < 0) {
-                  fail(rz);
-                }
-                while(1){
-                  if(end < 0) c = bgzf_read(rz, buffer, WINDOW_SIZE);
-                  else c = bgzf_read(rz, buffer, (end - start > WINDOW_SIZE)? WINDOW_SIZE:(end - start));
-                  if(c == 0) break;
-                  if (c < 0) fail(rz);
-                  start += c;
-                  write(f_dst, buffer, c);
-                  if(end >= 0 && start >= end) break;
-                }
-                free(buffer);
-               if (bgzf_close(rz) < 0) {
-                  fail(rz);
-                }
-                if (!pstdout) unlink(argv[optind]);
+               struct stat sbuf;
+               int f_dst;
+
+               if ( argc>optind )
+               {
+                       if ( stat(argv[optind],&sbuf)<0 )
+                       {
+                               fprintf(stderr, "[bgzip] %s: %s\n", strerror(errno), argv[optind]);
+                               return 1;
+                       }
+                       char *name;
+                       int len = strlen(argv[optind]);
+                       if ( strcmp(argv[optind]+len-3,".gz") )
+                       {
+                               fprintf(stderr, "[bgzip] %s: unknown suffix -- ignored\n", argv[optind]);
+                               return 1;
+                       }
+                       fp = bgzf_open(argv[optind], "r");
+                       if (fp == NULL) {
+                               fprintf(stderr, "[bgzip] Could not open file: %s\n", argv[optind]);
+                               return 1;
+                       }
+
+                       if (pstdout) {
+                               f_dst = fileno(stdout);
+                       }
+                       else {
+                               name = strdup(argv[optind]);
+                               name[strlen(name) - 3] = '\0';
+                               f_dst = write_open(name, is_forced);
+                               free(name);
+                       }
+               }
+               else if (!pstdout && isatty(fileno((FILE *)stdin)) )
+                       return bgzip_main_usage();
+               else
+               {
+                       f_dst = fileno(stdout);
+                       fp = bgzf_fdopen(fileno(stdin), "r");
+                       if (fp == NULL) {
+                               fprintf(stderr, "[bgzip] Could not read from stdin: %s\n", strerror(errno));
+                               return 1;
+                       }
+               }
+               buffer = malloc(WINDOW_SIZE);
+               if (bgzf_seek(fp, start, SEEK_SET) < 0) fail(fp);
+               while (1) {
+                       if (end < 0) c = bgzf_read(fp, buffer, WINDOW_SIZE);
+                       else c = bgzf_read(fp, buffer, (end - start > WINDOW_SIZE)? WINDOW_SIZE:(end - start));
+                       if (c == 0) break;
+                       if (c < 0) fail(fp);
+                       start += c;
+                       write(f_dst, buffer, c);
+                       if (end >= 0 && start >= end) break;
+               }
+               free(buffer);
+               if (bgzf_close(fp) < 0) fail(fp);
+               if (!pstdout) unlink(argv[optind]);

                return 0;
        }
 }
                return 0;
        }
 }

-

diff --git a/errmod.c b/errmod.c
new file mode 100644 (file)
index 0000000..fba9a8d
--- /dev/null
+++ b/errmod.c
@@ -0,0 +1,130 @@
+#include <math.h>
+#include "errmod.h"
+#include "ksort.h"
+KSORT_INIT_GENERIC(uint16_t)
+
+typedef struct __errmod_coef_t {
+       double *fk, *beta, *lhet;
+} errmod_coef_t;
+
+typedef struct {
+       double fsum[16], bsum[16];
+       uint32_t c[16];
+} call_aux_t;
+
+static errmod_coef_t *cal_coef(double depcorr, double eta)
+{
+       int k, n, q;
+       long double sum, sum1;
+       double *lC;
+       errmod_coef_t *ec;
+
+       ec = calloc(1, sizeof(errmod_coef_t));
+       // initialize ->fk
+       ec->fk = (double*)calloc(256, sizeof(double));
+       ec->fk[0] = 1.0;
+       for (n = 1; n != 256; ++n)
+               ec->fk[n] = pow(1. - depcorr, n) * (1.0 - eta) + eta;
+       // initialize ->coef
+       ec->beta = (double*)calloc(256 * 256 * 64, sizeof(double));
+       lC = (double*)calloc(256 * 256, sizeof(double));
+       for (n = 1; n != 256; ++n) {
+               double lgn = lgamma(n+1);
+               for (k = 1; k <= n; ++k)
+                       lC[n<<8|k] = lgn - lgamma(k+1) - lgamma(n-k+1);
+       }
+       for (q = 1; q != 64; ++q) {
+               double e = pow(10.0, -q/10.0);
+               double le = log(e);
+               double le1 = log(1.0 - e);
+               for (n = 1; n <= 255; ++n) {
+                       double *beta = ec->beta + (q<<16|n<<8);
+                       sum1 = sum = 0.0;
+                       for (k = n; k >= 0; --k, sum1 = sum) {
+                               sum = sum1 + expl(lC[n<<8|k] + k*le + (n-k)*le1);
+                               beta[k] = -10. / M_LN10 * logl(sum1 / sum);
+                       }
+               }
+       }
+       // initialize ->lhet
+       ec->lhet = (double*)calloc(256 * 256, sizeof(double));
+       for (n = 0; n < 256; ++n)
+               for (k = 0; k < 256; ++k)
+                       ec->lhet[n<<8|k] = lC[n<<8|k] - M_LN2 * n;
+       free(lC);
+       return ec;
+}
+
+errmod_t *errmod_init(float depcorr)
+{
+       errmod_t *em;
+       em = (errmod_t*)calloc(1, sizeof(errmod_t));
+       em->depcorr = depcorr;
+       em->coef = cal_coef(depcorr, 0.03);
+       return em;
+}
+
+void errmod_destroy(errmod_t *em)
+{
+       if (em == 0) return;
+       free(em->coef->lhet); free(em->coef->fk); free(em->coef->beta);
+       free(em->coef); free(em);
+}
+// qual:6, strand:1, base:4
+int errmod_cal(const errmod_t *em, int n, int m, uint16_t *bases, float *q)
+{
+       call_aux_t aux;
+       int i, j, k, w[32];
+
+       if (m > m) return -1;
+       memset(q, 0, m * m * sizeof(float));
+       if (n == 0) return 0;
+       // calculate aux.esum and aux.fsum
+       if (n > 255) { // then sample 255 bases
+               ks_shuffle(uint16_t, n, bases);
+               n = 255;
+       }
+       ks_introsort(uint16_t, n, bases);
+       memset(w, 0, 32 * sizeof(int));
+       memset(&aux, 0, sizeof(call_aux_t));
+       for (j = n - 1; j >= 0; --j) { // calculate esum and fsum
+               uint16_t b = bases[j];
+               int q = b>>5 < 4? 4 : b>>5;
+               if (q > 63) q = 63;
+               k = b&0x1f;
+               aux.fsum[k&0xf] += em->coef->fk[w[k]];
+               aux.bsum[k&0xf] += em->coef->fk[w[k]] * em->coef->beta[q<<16|n<<8|aux.c[k&0xf]];
+               ++aux.c[k&0xf];
+               ++w[k];
+       }
+       // generate likelihood
+       for (j = 0; j != m; ++j) {
+               float tmp1, tmp3;
+               int tmp2, bar_e;
+               // homozygous
+               for (k = 0, tmp1 = tmp3 = 0.0, tmp2 = 0; k != m; ++k) {
+                       if (k == j) continue;
+                       tmp1 += aux.bsum[k]; tmp2 += aux.c[k]; tmp3 += aux.fsum[k];
+               }
+               if (tmp2) {
+                       bar_e = (int)(tmp1 / tmp3 + 0.499);
+                       if (bar_e > 63) bar_e = 63;
+                       q[j*m+j] = tmp1;
+               }
+               // heterozygous
+               for (k = j + 1; k < m; ++k) {
+                       int cjk = aux.c[j] + aux.c[k];
+                       for (i = 0, tmp2 = 0, tmp1 = tmp3 = 0.0; i < m; ++i) {
+                               if (i == j || i == k) continue;
+                               tmp1 += aux.bsum[i]; tmp2 += aux.c[i]; tmp3 += aux.fsum[i];
+                       }
+                       if (tmp2) {
+                               bar_e = (int)(tmp1 / tmp3 + 0.499);
+                               if (bar_e > 63) bar_e = 63;
+                               q[j*m+k] = q[k*m+j] = -4.343 * em->coef->lhet[cjk<<8|aux.c[k]] + tmp1;
+                       } else q[j*m+k] = q[k*m+j] = -4.343 * em->coef->lhet[cjk<<8|aux.c[k]]; // all the bases are either j or k
+               }
+               for (k = 0; k != m; ++k) if (q[j*m+k] < 0.0) q[j*m+k] = 0.0;
+       }
+       return 0;
+}

diff --git a/errmod.h b/errmod.h
new file mode 100644 (file)
index 0000000..e3e9a90
--- /dev/null
+++ b/errmod.h
@@ -0,0 +1,17 @@
+#ifndef ERRMOD_H
+#define ERRMOD_H
+
+#include <stdint.h>
+
+struct __errmod_coef_t;
+
+typedef struct {
+       double depcorr;
+       struct __errmod_coef_t *coef;
+} errmod_t;
+
+errmod_t *errmod_init(float depcorr);
+void errmod_destroy(errmod_t *em);
+int errmod_cal(const errmod_t *em, int n, int m, uint16_t *bases, float *q);
+
+#endif

diff --git a/examples/Makefile b/examples/Makefile
index 8f0386fcaa48730dc24a276546cbe377ffffc756..ec976aedecf120c62a997580832e39c40d9748ac 100644 (file)
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -1,4 +1,5 @@
-all:../libbam.a ../samtools ex1.glf ex1.pileup.gz ex1.bam.bai ex1f-rmduppe.bam ex1f-rmdupse.bam ex1.glfview.gz calDepth
+all:../libbam.a ../samtools ../bcftools/bcftools \
+       ex1.glf ex1.pileup.gz ex1.bam.bai ex1f-rmduppe.bam ex1f-rmdupse.bam ex1.glfview.gz ex1.bcf calDepth

                @echo; echo \# You can now launch the viewer with: \'samtools tview ex1.bam ex1.fa\'; echo;
 
 ex1.fa.fai:ex1.fa
                @echo; echo \# You can now launch the viewer with: \'samtools tview ex1.bam ex1.fa\'; echo;
 
 ex1.fa.fai:ex1.fa


@@ -18,7 +19,7 @@ ex1a.bam:ex1.bam
 ex1b.bam:ex1.bam
                ../samtools view -h ex1.bam | awk 'BEGIN{FS=OFS="\t"}{if(/^@/)print;else{$$1=$$1"b";print}}' | ../samtools view -bS - > $@
 ex1f.rg:
 ex1b.bam:ex1.bam
                ../samtools view -h ex1.bam | awk 'BEGIN{FS=OFS="\t"}{if(/^@/)print;else{$$1=$$1"b";print}}' | ../samtools view -bS - > $@
 ex1f.rg:

-               (echo "@RG      ID:ex1  LB:ex1"; echo "@RG      ID:ex1a LB:ex1"; echo "@RG      ID:ex1b LB:ex1b") > $@
+               (echo "@RG      ID:ex1  LB:ex1  SM:ex1"; echo "@RG      ID:ex1a LB:ex1  SM:ex1"; echo "@RG      ID:ex1b LB:ex1b SM:ex1b") > $@

 ex1f.bam:ex1.bam ex1a.bam ex1b.bam ex1f.rg
                ../samtools merge -rh ex1f.rg $@ ex1.bam ex1a.bam ex1b.bam
 ex1f-rmduppe.bam:ex1f.bam
 ex1f.bam:ex1.bam ex1a.bam ex1b.bam ex1f.rg
                ../samtools merge -rh ex1f.rg $@ ex1.bam ex1a.bam ex1b.bam
 ex1f-rmduppe.bam:ex1f.bam


@@ -26,6 +27,12 @@ ex1f-rmduppe.bam:ex1f.bam
 ex1f-rmdupse.bam:ex1f.bam
                ../samtools rmdup -S ex1f.bam $@
 
 ex1f-rmdupse.bam:ex1f.bam
                ../samtools rmdup -S ex1f.bam $@
 

+ex1.bcf:ex1.bam ex1.fa.fai
+               ../samtools mpileup -gf ex1.fa ex1.bam > $@
+
+../bcftools/bcftools:
+               (cd ../bcftools; make bcftools)
+

 ../samtools:
                (cd ..; make samtools)
 
 ../samtools:
                (cd ..; make samtools)
 


@@ -36,4 +43,8 @@ calDepth:../libbam.a calDepth.c
                gcc -g -Wall -O2 -I.. calDepth.c -o $@ -lm -lz -L.. -lbam
 
 clean:
                gcc -g -Wall -O2 -I.. calDepth.c -o $@ -lm -lz -L.. -lbam
 
 clean:

-               rm -fr *.bam *.bai *.glf* *.fai *.pileup* *~ calDepth *.dSYM ex1*.rg
\ No newline at end of file
+               rm -fr *.bam *.bai *.glf* *.fai *.pileup* *~ calDepth *.dSYM ex1*.rg ex1.bcf
+
+# ../samtools pileup ex1.bam|perl -ape '$_=$F[4];s/(\d+)(??{".{$1}"})|\^.//g;@_=(tr/A-Z//,tr/a-z//);$_=join("\t",@F[0,1],@_)."\n"'
+
+# ../samtools pileup -cf ex1.fa ex1.bam|perl -ape '$_=$F[8];s/\^.//g;s/(\d+)(??{".{$1}"})|\^.//g;@_=(tr/A-Za-z//,tr/,.//);$_=join("\t",@F[0,1],@_)."\n"'
\ No newline at end of file

diff --git a/examples/toy.fa b/examples/toy.fa
index 38312c1b47204fbf3d77e57d809552a9f8360222..afe990a63bce6584dcb00704713e4183355b5afa 100644 (file)
--- a/examples/toy.fa
+++ b/examples/toy.fa
@@ -1,2 +1,4 @@
 >ref
 AGCATGTTAGATAAGATAGCTGTGCTAGTAGGCAGTCAGCGCCAT
 >ref
 AGCATGTTAGATAAGATAGCTGTGCTAGTAGGCAGTCAGCGCCAT

+>ref2
+aggttttataaaacaattaagtctacagagcaactacgcg

diff --git a/examples/toy.sam b/examples/toy.sam
index baf738894ddc674f37da6c851f6aba2ee032ff4f..1aff2204e4078b9adc9943a9533a2626cca887c8 100644 (file)
--- a/examples/toy.sam
+++ b/examples/toy.sam
@@ -1,7 +1,14 @@
 @SQ    SN:ref  LN:45
 @SQ    SN:ref  LN:45

-r001   163     ref     7       30      8M2I4M1D3M      =       37      39      TTAGATAAAGGATACTG       *
-r002   0       ref     9       30      1S2I6M1P1I4M2I  *       0       0       AAAAGATAAGGATAAA        *
+@SQ    SN:ref2 LN:40
+r001   163     ref     7       30      8M4I4M1D3M      =       37      39      TTAGATAAAGAGGATACTG     *
+r002   0       ref     9       30      1S2I6M1P1I1P1I4M2I      *       0       0       AAAAGATAAGGGATAAA       *

 r003   0       ref     9       30      5H6M    *       0       0       AGCTAA  *
 r004   0       ref     16      30      6M14N1I5M       *       0       0       ATAGCTCTCAGC    *
 r003   16      ref     29      30      6H5M    *       0       0       TAGGC   *
 r003   0       ref     9       30      5H6M    *       0       0       AGCTAA  *
 r004   0       ref     16      30      6M14N1I5M       *       0       0       ATAGCTCTCAGC    *
 r003   16      ref     29      30      6H5M    *       0       0       TAGGC   *

-r001   83      ref     37      30      9M      =       7       -39     CAGCGCCAT       *
\ No newline at end of file
+r001   83      ref     37      30      9M      =       7       -39     CAGCGCCAT       *
+x1     0       ref2    1       30      20M     *       0       0       aggttttataaaacaaataa    ????????????????????
+x2     0       ref2    2       30      21M     *       0       0       ggttttataaaacaaataatt   ?????????????????????
+x3     0       ref2    6       30      9M4I13M *       0       0       ttataaaacAAATaattaagtctaca      ??????????????????????????
+x4     0       ref2    10      30      25M     *       0       0       CaaaTaattaagtctacagagcaac       ?????????????????????????
+x5     0       ref2    12      30      24M     *       0       0       aaTaattaagtctacagagcaact        ????????????????????????
+x6     0       ref2    14      30      23M     *       0       0       Taattaagtctacagagcaacta ???????????????????????

diff --git a/kaln.c b/kaln.c
index 9fa40d0be3999e66b885486d7a03e50994bfbfdd..680779113ce7c2220434727cf578845d1e840d40 100644 (file)
--- a/kaln.c
+++ b/kaln.c
@@ -27,6 +27,7 @@
 #include <stdio.h>
 #include <string.h>
 #include <stdint.h>
 #include <stdio.h>
 #include <string.h>
 #include <stdint.h>

+#include <math.h>

 #include "kaln.h"
 
 #define FROM_M 0
 #include "kaln.h"
 
 #define FROM_M 0


@@ -72,8 +73,8 @@ int aln_sm_blast[] = {
        -2, -2, -2, -2, -2
 };
 
        -2, -2, -2, -2, -2
 };
 

-ka_param_t ka_param_blast = {  5,  2,  2, aln_sm_blast, 5, 50 };
-ka_param_t ka_param_aa2aa = { 10,  2,  2, aln_sm_blosum62, 22, 50 };
+ka_param_t ka_param_blast = {  5,  2,   5, 2, aln_sm_blast, 5, 50 };
+ka_param_t ka_param_aa2aa = { 10,  2,  10, 2, aln_sm_blosum62, 22, 50 };

 
 static uint32_t *ka_path2cigar32(const path_t *path, int path_len, int *n_cigar)
 {
 
 static uint32_t *ka_path2cigar32(const path_t *path, int path_len, int *n_cigar)
 {


@@ -141,13 +142,13 @@ static uint32_t *ka_path2cigar32(const path_t *path, int path_len, int *n_cigar)
 }
 #define set_end_I(II, cur, p)                                                  \
 {                                                                                                              \
 }
 #define set_end_I(II, cur, p)                                                  \
 {                                                                                                              \

-       if (gap_end >= 0) {                                                                     \
-               if ((p)->M - gap_open > (p)->I) {                               \
+       if (gap_end_ext >= 0) {                                                         \
+               if ((p)->M - gap_end_open > (p)->I) {                   \

                        (cur)->It = FROM_M;                                                     \
                        (cur)->It = FROM_M;                                                     \

-                       (II) = (p)->M - gap_open - gap_end;                     \
+                       (II) = (p)->M - gap_end_open - gap_end_ext;     \

                } else {                                                                                \
                        (cur)->It = FROM_I;                                                     \
                } else {                                                                                \
                        (cur)->It = FROM_I;                                                     \

-                       (II) = (p)->I - gap_end;                                        \
+                       (II) = (p)->I - gap_end_ext;                            \

                }                                                                                               \
        } else set_I(II, cur, p);                                                       \
 }
                }                                                                                               \
        } else set_I(II, cur, p);                                                       \
 }


@@ -163,13 +164,13 @@ static uint32_t *ka_path2cigar32(const path_t *path, int path_len, int *n_cigar)
 }
 #define set_end_D(DD, cur, p)                                                  \
 {                                                                                                              \
 }
 #define set_end_D(DD, cur, p)                                                  \
 {                                                                                                              \

-       if (gap_end >= 0) {                                                                     \
-               if ((p)->M - gap_open > (p)->D) {                               \
+       if (gap_end_ext >= 0) {                                                         \
+               if ((p)->M - gap_end_open > (p)->D) {                   \

                        (cur)->Dt = FROM_M;                                                     \
                        (cur)->Dt = FROM_M;                                                     \

-                       (DD) = (p)->M - gap_open - gap_end;                     \
+                       (DD) = (p)->M - gap_end_open - gap_end_ext;     \

                } else {                                                                                \
                        (cur)->Dt = FROM_D;                                                     \
                } else {                                                                                \
                        (cur)->Dt = FROM_D;                                                     \

-                       (DD) = (p)->D - gap_end;                                        \
+                       (DD) = (p)->D - gap_end_ext;                            \

                }                                                                                               \
        } else set_D(DD, cur, p);                                                       \
 }
                }                                                                                               \
        } else set_D(DD, cur, p);                                                       \
 }


@@ -195,13 +196,14 @@ uint32_t *ka_global_core(uint8_t *seq1, int len1, uint8_t *seq2, int len2, const
        uint8_t type, ctype;
        uint32_t *cigar = 0;
 
        uint8_t type, ctype;
        uint32_t *cigar = 0;
 

-       int gap_open, gap_ext, gap_end, b;
+       int gap_open, gap_ext, gap_end_open, gap_end_ext, b;

        int *score_matrix, N_MATRIX_ROW;
 
        /* initialize some align-related parameters. just for compatibility */
        gap_open = ap->gap_open;
        gap_ext = ap->gap_ext;
        int *score_matrix, N_MATRIX_ROW;
 
        /* initialize some align-related parameters. just for compatibility */
        gap_open = ap->gap_open;
        gap_ext = ap->gap_ext;

-       gap_end = ap->gap_end;
+       gap_end_open = ap->gap_end_open;
+       gap_end_ext = ap->gap_end_ext;

        b = ap->band_width;
        score_matrix = ap->matrix;
        N_MATRIX_ROW = ap->row;
        b = ap->band_width;
        score_matrix = ap->matrix;
        N_MATRIX_ROW = ap->row;


@@ -368,3 +370,212 @@ uint32_t *ka_global_core(uint8_t *seq1, int len1, uint8_t *seq2, int len2, const
 
        return cigar;
 }
 
        return cigar;
 }

+
+/*****************************************
+ * Probabilistic banded glocal alignment *
+ *****************************************/
+
+static float g_qual2prob[256];
+
+#define EI .25
+#define EM .3333333333333
+#define set_u(u, b, i, k) { int x=(i)-(b); x=x>0?x:0; (u)=((k)-x+1)*3; }
+
+ka_probpar_t ka_probpar_def = { 0.001, 0.1, 10 };
+
+/*
+  The topology of the profile HMM:
+
+           /\             /\        /\             /\
+           I[1]           I[k-1]    I[k]           I[L]
+            ^   \      \    ^    \   ^   \      \   ^
+            |    \      \   |     \  |    \      \  |
+    M[0]   M[1] -> ... -> M[k-1] -> M[k] -> ... -> M[L]   M[L+1]
+                \      \/        \/      \/      /
+                 \     /\        /\      /\     /
+                       -> D[k-1] -> D[k] ->
+
+   M[0] points to every {M,I}[k] and every {M,I}[k] points M[L+1].
+
+   On input, _ref is the reference sequence and _query is the query
+   sequence. Both are sequences of 0/1/2/3/4 where 4 stands for an
+   ambiguous residue. iqual is the base quality. c sets the gap open
+   probability, gap extension probability and band width.
+
+   On output, state and q are arrays of length l_query. The higher 30
+   bits give the reference position the query base is matched to and the
+   lower two bits can be 0 (an alignment match) or 1 (an
+   insertion). q[i] gives the phred scaled posterior probability of
+   state[i] being wrong.
+ */
+int ka_prob_glocal(const uint8_t *_ref, int l_ref, const uint8_t *_query, int l_query, const uint8_t *iqual,
+                                  const ka_probpar_t *c, int *state, uint8_t *q)
+{
+       double **f, **b, *s, m[9], sI, sM, bI, bM, pb;
+       float *qual, *_qual;
+       const uint8_t *ref, *query;
+       int bw, bw2, i, k, is_diff = 0;
+
+       /*** initialization ***/
+       ref = _ref - 1; query = _query - 1; // change to 1-based coordinate
+       bw = l_ref > l_query? l_ref : l_query;
+       if (bw > c->bw) bw = c->bw;
+       if (bw < abs(l_ref - l_query)) bw = abs(l_ref - l_query);
+       bw2 = bw * 2 + 1;
+       // allocate the forward and backward matrices f[][] and b[][] and the scaling array s[]
+       f = calloc(l_query+1, sizeof(void*));
+       b = calloc(l_query+1, sizeof(void*));
+       for (i = 0; i <= l_query; ++i) {
+               f[i] = calloc(bw2 * 3 + 6, sizeof(double)); // FIXME: this is over-allocated for very short seqs
+               b[i] = calloc(bw2 * 3 + 6, sizeof(double));
+       }
+       s = calloc(l_query+2, sizeof(double)); // s[] is the scaling factor to avoid underflow
+       // initialize qual
+       _qual = calloc(l_query, sizeof(float));
+       if (g_qual2prob[0] == 0)
+               for (i = 0; i < 256; ++i)
+                       g_qual2prob[i] = pow(10, -i/10.);
+       for (i = 0; i < l_query; ++i) _qual[i] = g_qual2prob[iqual? iqual[i] : 30];
+       qual = _qual - 1;
+       // initialize transition probability
+       sM = sI = 1. / (2 * l_query + 2); // the value here seems not to affect results; FIXME: need proof
+       m[0*3+0] = (1 - c->d - c->d) * (1 - sM); m[0*3+1] = m[0*3+2] = c->d * (1 - sM);
+       m[1*3+0] = (1 - c->e) * (1 - sI); m[1*3+1] = c->e * (1 - sI); m[1*3+2] = 0.;
+       m[2*3+0] = 1 - c->e; m[2*3+1] = 0.; m[2*3+2] = c->e;
+       bM = (1 - c->d) / l_query; bI = c->d / l_query; // (bM+bI)*l_query==1
+       /*** forward ***/
+       // f[0]
+       set_u(k, bw, 0, 0);
+       f[0][k] = s[0] = 1.;
+       { // f[1]
+               double *fi = f[1], sum;
+               int beg = 1, end = l_ref < bw + 1? l_ref : bw + 1, _beg, _end;
+               for (k = beg, sum = 0.; k <= end; ++k) {
+                       int u;
+                       double e = (ref[k] > 3 || query[1] > 3)? 1. : ref[k] == query[1]? 1. - qual[1] : qual[1] * EM;
+                       set_u(u, bw, 1, k);
+                       fi[u+0] = e * bM; fi[u+1] = EI * bI;
+                       sum += fi[u] + fi[u+1];
+               }
+               // rescale
+               s[1] = sum;
+               set_u(_beg, bw, 1, beg); set_u(_end, bw, 1, end); _end += 2;
+               for (k = _beg; k <= _end; ++k) fi[k] /= sum;
+       }
+       // f[2..l_query]
+       for (i = 2; i <= l_query; ++i) {
+               double *fi = f[i], *fi1 = f[i-1], sum, qli = qual[i];
+               int beg = 1, end = l_ref, x, _beg, _end;
+               uint8_t qyi = query[i];
+               x = i - bw; beg = beg > x? beg : x; // band start
+               x = i + bw; end = end < x? end : x; // band end
+               for (k = beg, sum = 0.; k <= end; ++k) {
+                       int u, v11, v01, v10;
+                       double e;
+                       e = (ref[k] > 3 || qyi > 3)? 1. : ref[k] == qyi? 1. - qli : qli * EM;
+                       set_u(u, bw, i, k); set_u(v11, bw, i-1, k-1); set_u(v10, bw, i-1, k); set_u(v01, bw, i, k-1);
+                       fi[u+0] = e * (m[0] * fi1[v11+0] + m[3] * fi1[v11+1] + m[6] * fi1[v11+2]);
+                       fi[u+1] = EI * (m[1] * fi1[v10+0] + m[4] * fi1[v10+1]);
+                       fi[u+2] = m[2] * fi[v01+0] + m[8] * fi[v01+2];
+                       sum += fi[u] + fi[u+1] + fi[u+2];
+//                     fprintf(stderr, "F (%d,%d;%d): %lg,%lg,%lg\n", i, k, u, fi[u], fi[u+1], fi[u+2]); // DEBUG
+               }
+               // rescale
+               s[i] = sum;
+               set_u(_beg, bw, i, beg); set_u(_end, bw, i, end); _end += 2;
+               for (k = _beg, sum = 1./sum; k <= _end; ++k) fi[k] *= sum;
+       }
+       { // f[l_query+1]
+               double sum;
+               for (k = 1, sum = 0.; k <= l_ref; ++k) {
+                       int u;
+                       set_u(u, bw, l_query, k);
+                       if (u < 3 || u >= bw2*3+3) continue;
+                   sum += f[l_query][u+0] * sM + f[l_query][u+1] * sI;
+               }
+               s[l_query+1] = sum; // the last scaling factor
+       }
+       /*** backward ***/
+       // b[l_query] (b[l_query+1][0]=1 and thus \tilde{b}[][]=1/s[l_query+1]; this is where s[l_query+1] comes from)
+       for (k = 1; k <= l_ref; ++k) {
+               int u;
+               double *bi = b[l_query];
+               set_u(u, bw, l_query, k);
+               if (u < 3 || u >= bw2*3+3) continue;
+               bi[u+0] = sM / s[l_query] / s[l_query+1]; bi[u+1] = sI / s[l_query] / s[l_query+1];
+       }
+       // b[l_query-1..1]
+       for (i = l_query - 1; i >= 1; --i) {
+               int beg = 1, end = l_ref, x, _beg, _end;
+               double *bi = b[i], *bi1 = b[i+1], y = (i > 1), qli1 = qual[i+1];
+               uint8_t qyi1 = query[i+1];
+               x = i - bw; beg = beg > x? beg : x;
+               x = i + bw; end = end < x? end : x;
+               for (k = end; k >= beg; --k) {
+                       int u, v11, v01, v10;
+                       double e;
+                       set_u(u, bw, i, k); set_u(v11, bw, i+1, k+1); set_u(v10, bw, i+1, k); set_u(v01, bw, i, k+1);
+                       e = (k >= l_ref? 0 : (ref[k+1] > 3 || qyi1 > 3)? 1. : ref[k+1] == qyi1? 1. - qli1 : qli1 * EM) * bi1[v11];
+                       bi[u+0] = e * m[0] + EI * m[1] * bi1[v10+1] + m[2] * bi[v01+2]; // bi1[v11] has been foled into e.
+                       bi[u+1] = e * m[3] + EI * m[4] * bi1[v10+1];
+                       bi[u+2] = (e * m[6] + m[8] * bi[v01+2]) * y;
+//                     fprintf(stderr, "B (%d,%d;%d): %lg,%lg,%lg\n", i, k, u, bi[u], bi[u+1], bi[u+2]); // DEBUG
+               }
+               // rescale
+               set_u(_beg, bw, i, beg); set_u(_end, bw, i, end); _end += 2;
+               for (k = _beg, y = 1./s[i]; k <= _end; ++k) bi[k] *= y;
+       }
+       { // b[0]
+               int beg = 1, end = l_ref < bw + 1? l_ref : bw + 1;
+               double sum = 0.;
+               for (k = end; k >= beg; --k) {
+                       int u;
+                       double e = (ref[k] > 3 || query[1] > 3)? 1. : ref[k] == query[1]? 1. - qual[1] : qual[1] * EM;
+                       set_u(u, bw, 1, k);
+                       if (u < 3 || u >= bw2*3+3) continue;
+                   sum += e * b[1][u+0] * bM + EI * b[1][u+1] * bI;
+               }
+               set_u(k, bw, 0, 0);
+               pb = b[0][k] = sum / s[0]; // if everything works as is expected, pb == 1.0
+       }
+       is_diff = fabs(pb - 1.) > 1e-7? 1 : 0;
+       /*** MAP ***/
+       for (i = 1; i <= l_query; ++i) {
+               double sum = 0., *fi = f[i], *bi = b[i], max = 0.;
+               int beg = 1, end = l_ref, x, max_k = -1;
+               x = i - bw; beg = beg > x? beg : x;
+               x = i + bw; end = end < x? end : x;
+               for (k = beg; k <= end; ++k) {
+                       int u;
+                       double z;
+                       set_u(u, bw, i, k);
+                       z = fi[u+0] * bi[u+0]; if (z > max) max = z, max_k = (k-1)<<2 | 0; sum += z;
+                       z = fi[u+1] * bi[u+1]; if (z > max) max = z, max_k = (k-1)<<2 | 1; sum += z;
+               }
+               max /= sum; sum *= s[i]; // if everything works as is expected, sum == 1.0
+               if (state) state[i-1] = max_k;
+               if (q) k = (int)(-4.343 * log(1. - max) + .499), q[i-1] = k > 100? 99 : k;
+#ifdef _MAIN
+               fprintf(stderr, "(%.10lg,%.10lg) (%d,%d:%d)~%lg\n", pb, sum, i-1, max_k>>2, max_k&3, max); // DEBUG
+#endif
+       }
+       /*** free ***/
+       for (i = 0; i <= l_query; ++i) {
+               free(f[i]); free(b[i]);
+       }
+       free(f); free(b); free(s); free(_qual);
+       return 0;
+}
+
+#ifdef _MAIN
+int main()
+{
+       int l_ref = 5, l_query = 4;
+       uint8_t *ref = (uint8_t*)"\0\1\3\3\1";
+       uint8_t *query = (uint8_t*)"\0\3\3\1";
+//     uint8_t *query = (uint8_t*)"\1\3\3\1";
+       static uint8_t qual[4] = {20, 20, 20, 20};
+       ka_prob_glocal(ref, l_ref, query, l_query, qual, &ka_probpar_def, 0, 0);
+       return 0;
+}
+#endif

diff --git a/kaln.h b/kaln.h
index b04d8cc0b001c17c20654fc125e392b08aefb343..b24597cfc2ee159529c5746916597591ecababe2 100644 (file)
--- a/kaln.h
+++ b/kaln.h
@@ -33,23 +33,33 @@
 typedef struct {
        int gap_open;
        int gap_ext;
 typedef struct {
        int gap_open;
        int gap_ext;

-       int gap_end;
+       int gap_end_open;
+       int gap_end_ext;

 
        int *matrix;
        int row;
        int band_width;
 } ka_param_t;
 
 
        int *matrix;
        int row;
        int band_width;
 } ka_param_t;
 

+typedef struct {
+       float d, e;
+       int bw;
+} ka_probpar_t;
+

 #ifdef __cplusplus
 extern "C" {
 #endif
 
 #ifdef __cplusplus
 extern "C" {
 #endif
 

-       uint32_t *ka_global_core(uint8_t *seq1, int len1, uint8_t *seq2, int len2, const ka_param_t *ap, int *_score, int *n_cigar);
+       uint32_t *ka_global_core(uint8_t *seq1, int len1, uint8_t *seq2, int len2, const ka_param_t *ap,
+                                                        int *_score, int *n_cigar);
+       int ka_prob_glocal(const uint8_t *_ref, int l_ref, const uint8_t *_query, int l_query, const uint8_t *iqual,
+                                          const ka_probpar_t *c, int *state, uint8_t *q);

 
 #ifdef __cplusplus
 }
 #endif
 
 
 #ifdef __cplusplus
 }
 #endif
 

-extern ka_param_t ka_param_blast; /* = {  5,  2,  2, aln_sm_blast, 5, 50 }; */
+extern ka_param_t ka_param_blast; /* = { 5, 2, 5, 2, aln_sm_blast, 5, 50 }; */
+extern ka_probpar_t ka_probpar_def; /* { 0.0001, 0.1, 10 } */

 
 #endif
 
 #endif

diff --git a/ksort.h b/ksort.h
index 16a03fd1aad8be1d3b4a090b52c4daf37ae7b217..fa850ab0762a9225bae5b3bcad6739a3bb6a3c27 100644 (file)
--- a/ksort.h
+++ b/ksort.h
@@ -250,6 +250,15 @@ typedef struct {
                        if (hh <= k) low = ll;                                                                          \
                        if (hh >= k) high = hh - 1;                                                                     \
                }                                                                                                                               \
                        if (hh <= k) low = ll;                                                                          \
                        if (hh >= k) high = hh - 1;                                                                     \
                }                                                                                                                               \

+       }                                                                                                                                       \
+       void ks_shuffle_##name(size_t n, type_t a[])                                            \
+       {                                                                                                                                       \
+               int i, j;                                                                                                               \
+               for (i = n; i > 1; --i) {                                                                               \
+                       type_t tmp;                                                                                                     \
+                       j = (int)(drand48() * i);                                                                       \
+                       tmp = a[j]; a[j] = a[i-1]; a[i-1] = tmp;                                        \
+               }                                                                                                                               \

        }
 
 #define ks_mergesort(name, n, a, t) ks_mergesort_##name(n, a, t)
        }
 
 #define ks_mergesort(name, n, a, t) ks_mergesort_##name(n, a, t)


@@ -259,6 +268,7 @@ typedef struct {
 #define ks_heapmake(name, n, a) ks_heapmake_##name(n, a)
 #define ks_heapadjust(name, i, n, a) ks_heapadjust_##name(i, n, a)
 #define ks_ksmall(name, n, a, k) ks_ksmall_##name(n, a, k)
 #define ks_heapmake(name, n, a) ks_heapmake_##name(n, a)
 #define ks_heapadjust(name, i, n, a) ks_heapadjust_##name(i, n, a)
 #define ks_ksmall(name, n, a, k) ks_ksmall_##name(n, a, k)

+#define ks_shuffle(name, n, a) ks_shuffle_##name(n, a)

 
 #define ks_lt_generic(a, b) ((a) < (b))
 #define ks_lt_str(a, b) (strcmp((a), (b)) < 0)
 
 #define ks_lt_generic(a, b) ((a) < (b))
 #define ks_lt_str(a, b) (strcmp((a), (b)) < 0)

diff --git a/kstring.c b/kstring.c
index e0203fa3e877604f53291b39ff6a2f273e6dff8f..43d524c92efc38fb7d417759aa4caaabf402aa05 100644 (file)
--- a/kstring.c
+++ b/kstring.c
@@ -24,6 +24,24 @@ int ksprintf(kstring_t *s, const char *fmt, ...)
        return l;
 }
 
        return l;
 }
 

+char *kstrtok(const char *str, const char *sep, ks_tokaux_t *aux)
+{
+       const char *p, *start;
+       if (sep) { // set up the table
+               if (str == 0 && (aux->tab[0]&1)) return 0; // no need to set up if we have finished
+               aux->tab[0] = aux->tab[1] = aux->tab[2] = aux->tab[3] = 0;
+               for (p = sep; *p; ++p)
+                       aux->tab[*p/64] |= 1ull<<(*p%64);
+       }
+       if (str) aux->p = str - 1, aux->tab[0] &= ~1ull;
+       else if (aux->tab[0]&1) return 0;
+       for (p = start = aux->p + 1; *p; ++p)
+               if (aux->tab[*p/64]>>(*p%64)&1) break;
+       aux->p = p; // end of token
+       if (*p == 0) aux->tab[0] |= 1; // no more tokens
+       return (char*)start;
+}
+

 // s MUST BE a null terminated string; l = strlen(s)
 int ksplit_core(char *s, int delimiter, int *_max, int **_offsets)
 {
 // s MUST BE a null terminated string; l = strlen(s)
 int ksplit_core(char *s, int delimiter, int *_max, int **_offsets)
 {


@@ -66,11 +84,13 @@ int ksplit_core(char *s, int delimiter, int *_max, int **_offsets)
  * Boyer-Moore search *
  **********************/
 
  * Boyer-Moore search *
  **********************/
 

+typedef unsigned char ubyte_t;
+

 // reference: http://www-igm.univ-mlv.fr/~lecroq/string/node14.html
 // reference: http://www-igm.univ-mlv.fr/~lecroq/string/node14.html

-int *ksBM_prep(const uint8_t *pat, int m)
+static int *ksBM_prep(const ubyte_t *pat, int m)

 {
        int i, *suff, *prep, *bmGs, *bmBc;
 {
        int i, *suff, *prep, *bmGs, *bmBc;

-       prep = calloc(m + 256, 1);
+       prep = calloc(m + 256, sizeof(int));

        bmGs = prep; bmBc = prep + m;
        { // preBmBc()
                for (i = 0; i < 256; ++i) bmBc[i] = m;
        bmGs = prep; bmBc = prep + m;
        { // preBmBc()
                for (i = 0; i < 256; ++i) bmBc[i] = m;


@@ -107,39 +127,49 @@ int *ksBM_prep(const uint8_t *pat, int m)
        return prep;
 }
 
        return prep;
 }
 

-int *ksBM_search(const uint8_t *str, int n, const uint8_t *pat, int m, int *_prep, int *n_matches)
+void *kmemmem(const void *_str, int n, const void *_pat, int m, int **_prep)

 {
 {

-       int i, j, *prep, *bmGs, *bmBc;
-       int *matches = 0, mm = 0, nm = 0;
-       prep = _prep? _prep : ksBM_prep(pat, m);
+       int i, j, *prep = 0, *bmGs, *bmBc;
+       const ubyte_t *str, *pat;
+       str = (const ubyte_t*)_str; pat = (const ubyte_t*)_pat;
+       prep = (_prep == 0 || *_prep == 0)? ksBM_prep(pat, m) : *_prep;
+       if (_prep && *_prep == 0) *_prep = prep;

        bmGs = prep; bmBc = prep + m;
        j = 0;
        while (j <= n - m) {
                for (i = m - 1; i >= 0 && pat[i] == str[i+j]; --i);
        bmGs = prep; bmBc = prep + m;
        j = 0;
        while (j <= n - m) {
                for (i = m - 1; i >= 0 && pat[i] == str[i+j]; --i);

-               if (i < 0) {
-                       if (nm == mm) {
-                               mm = mm? mm<<1 : 1;
-                               matches = realloc(matches, mm * sizeof(int));
-                       }
-                       matches[nm++] = j;
-                       j += bmGs[0];
-               } else {
+               if (i >= 0) {

                        int max = bmBc[str[i+j]] - m + 1 + i;
                        if (max < bmGs[i]) max = bmGs[i];
                        j += max;
                        int max = bmBc[str[i+j]] - m + 1 + i;
                        if (max < bmGs[i]) max = bmGs[i];
                        j += max;

-               }
+               } else return (void*)(str + j);

        }
        }

-       *n_matches = nm;

        if (_prep == 0) free(prep);
        if (_prep == 0) free(prep);

-       return matches;
+       return 0;

 }
 
 }
 

+char *kstrstr(const char *str, const char *pat, int **_prep)
+{
+       return (char*)kmemmem(str, strlen(str), pat, strlen(pat), _prep);
+}
+
+char *kstrnstr(const char *str, const char *pat, int n, int **_prep)
+{
+       return (char*)kmemmem(str, n, pat, strlen(pat), _prep);
+}
+
+/***********************
+ * The main() function *
+ ***********************/
+

 #ifdef KSTRING_MAIN
 #include <stdio.h>
 int main()
 {
        kstring_t *s;
        int *fields, n, i;
 #ifdef KSTRING_MAIN
 #include <stdio.h>
 int main()
 {
        kstring_t *s;
        int *fields, n, i;

+       ks_tokaux_t aux;
+       char *p;

        s = (kstring_t*)calloc(1, sizeof(kstring_t));
        // test ksprintf()
        ksprintf(s, " abcdefg:    %d ", 100);
        s = (kstring_t*)calloc(1, sizeof(kstring_t));
        // test ksprintf()
        ksprintf(s, " abcdefg:    %d ", 100);


@@ -148,17 +178,26 @@ int main()
        fields = ksplit(s, 0, &n);
        for (i = 0; i < n; ++i)
                printf("field[%d] = '%s'\n", i, s->s + fields[i]);
        fields = ksplit(s, 0, &n);
        for (i = 0; i < n; ++i)
                printf("field[%d] = '%s'\n", i, s->s + fields[i]);

-       free(s);
+       // test kstrtok()
+       s->l = 0;
+       for (p = kstrtok("ab:cde:fg/hij::k", ":/", &aux); p; p = kstrtok(0, 0, &aux)) {
+               kputsn(p, aux.p - p, s);
+               kputc('\n', s);
+       }
+       printf("%s", s->s);
+       // free
+       free(s->s); free(s); free(fields);

 
        {
 
        {

-               static char *str = "abcdefgcdg";
+               static char *str = "abcdefgcdgcagtcakcdcd";

                static char *pat = "cd";
                static char *pat = "cd";

-               int n, *matches;
-               matches = ksBM_search(str, strlen(str), pat, strlen(pat), 0, &n);
-               printf("%d: \n", n);
-               for (i = 0; i < n; ++i)
-                       printf("- %d\n", matches[i]);
-               free(matches);
+               char *ret, *s = str;
+               int *prep = 0;
+               while ((ret = kstrstr(s, pat, &prep)) != 0) {
+                       printf("match: %s\n", ret);
+                       s = ret + prep[0];
+               }
+               free(prep);

        }
        return 0;
 }
        }
        return 0;
 }

diff --git a/kstring.h b/kstring.h
index 925117a38646ab4c1b82db01e46091f9e073eefd..c46a62bc8bd0284826c1c7dd78ff1698d54f54f7 100644 (file)
--- a/kstring.h
+++ b/kstring.h
@@ -17,17 +17,31 @@ typedef struct __kstring_t {
 } kstring_t;
 #endif
 
 } kstring_t;
 #endif
 

-int ksprintf(kstring_t *s, const char *fmt, ...);
-int ksplit_core(char *s, int delimiter, int *_max, int **_offsets);
+typedef struct {
+       uint64_t tab[4];
+       const char *p; // end of the current token
+} ks_tokaux_t;

 
 

-// calculate the auxiliary array, allocated by calloc()
-int *ksBM_prep(const uint8_t *pat, int m);
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+       int ksprintf(kstring_t *s, const char *fmt, ...);
+       int ksplit_core(char *s, int delimiter, int *_max, int **_offsets);
+       char *kstrstr(const char *str, const char *pat, int **_prep);
+       char *kstrnstr(const char *str, const char *pat, int n, int **_prep);
+       void *kmemmem(const void *_str, int n, const void *_pat, int m, int **_prep);

 
 

-/* Search pat in str and returned the list of matches. The size of the
- * list is returned as n_matches. _prep is the array returned by
- * ksBM_prep(). If it is a NULL pointer, ksBM_prep() will be called. */
-int *ksBM_search(const uint8_t *str, int n, const uint8_t *pat, int m, int *_prep, int *n_matches);
+       /* kstrtok() is similar to strtok_r() except that str is not
+        * modified and both str and sep can be NULL. For efficiency, it is
+        * actually recommended to set both to NULL in the subsequent calls
+        * if sep is not changed. */
+       char *kstrtok(const char *str, const char *sep, ks_tokaux_t *aux);

 
 

+#ifdef __cplusplus
+}
+#endif
+       

 static inline int kputsn(const char *p, int l, kstring_t *s)
 {
        if (s->l + l + 1 >= s->m) {
 static inline int kputsn(const char *p, int l, kstring_t *s)
 {
        if (s->l + l + 1 >= s->m) {


@@ -35,7 +49,7 @@ static inline int kputsn(const char *p, int l, kstring_t *s)
                kroundup32(s->m);
                s->s = (char*)realloc(s->s, s->m);
        }
                kroundup32(s->m);
                s->s = (char*)realloc(s->s, s->m);
        }

-       strncpy(s->s + s->l, p, l);
+       memcpy(s->s + s->l, p, l);

        s->l += l;
        s->s[s->l] = 0;
        return l;
        s->l += l;
        s->s[s->l] = 0;
        return l;

diff --git a/misc/HmmGlocal.java b/misc/HmmGlocal.java
new file mode 100644 (file)
index 0000000..9e93b13
--- /dev/null
+++ b/misc/HmmGlocal.java
@@ -0,0 +1,178 @@
+import java.io.*;
+import java.lang.*;
+
+public class HmmGlocal
+{
+       private double[] qual2prob;
+       private double cd, ce; // gap open probility [1e-3], gap extension probability [0.1]
+       private int cb; // band width [7]
+
+       public HmmGlocal(final double d, final double e, final int b) {
+               cd = d; ce = e; cb = b;
+               qual2prob = new double[256];
+               for (int i = 0; i < 256; ++i)
+                       qual2prob[i] = Math.pow(10, -i/10.);
+       }
+       private static int set_u(final int b, final int i, final int k) {
+               int x = i - b;
+               x = x > 0? x : 0;
+               return (k + 1 - x) * 3;
+       }
+       public int hmm_glocal(final byte[] _ref, final byte[] _query, final byte[] _iqual, int[] state, byte[] q) {
+               int i, k;
+               /*** initialization ***/
+               // change coordinates
+               int l_ref = _ref.length;
+               byte[] ref = new byte[l_ref+1];
+               for (i = 0; i < l_ref; ++i) ref[i+1] = _ref[i]; // FIXME: this is silly...
+               int l_query = _query.length;
+               byte[] query = new byte[l_query+1];
+               double[] qual = new double[l_query+1];
+               for (i = 0; i < l_query; ++i) {
+                       query[i+1] = _query[i];
+                       qual[i+1] = qual2prob[_iqual[i]];
+               }
+               // set band width
+               int bw2, bw = l_ref > l_query? l_ref : l_query;
+               if (bw > cb) bw = cb;
+               if (bw < Math.abs(l_ref - l_query)) bw = Math.abs(l_ref - l_query);
+               bw2 = bw * 2 + 1;
+               // allocate the forward and backward matrices f[][] and b[][] and the scaling array s[]
+               double[][] f = new double[l_query+1][bw2*3 + 6];
+               double[][] b = new double[l_query+1][bw2*3 + 6];
+               double[] s = new double[l_query+2];
+               // initialize transition probabilities
+               double sM, sI, bM, bI;
+               sM = sI = 1. / (2 * l_query + 2);
+               bM = (1 - cd) / l_query; bI = cd / l_query; // (bM+bI)*l_query==1
+               double[] m = new double[9];
+               m[0*3+0] = (1 - cd - cd) * (1 - sM); m[0*3+1] = m[0*3+2] = cd * (1 - sM);
+               m[1*3+0] = (1 - ce) * (1 - sI); m[1*3+1] = ce * (1 - sI); m[1*3+2] = 0.;
+               m[2*3+0] = 1 - ce; m[2*3+1] = 0.; m[2*3+2] = ce;
+               /*** forward ***/
+               // f[0]
+               f[0][set_u(bw, 0, 0)] = s[0] = 1.;
+               { // f[1]
+                       double[] fi = f[1];
+                       double sum;
+                       int beg = 1, end = l_ref < bw + 1? l_ref : bw + 1, _beg, _end;
+                       for (k = beg, sum = 0.; k <= end; ++k) {
+                               int u;
+                               double e = (ref[k] > 3 || query[1] > 3)? 1. : ref[k] == query[1]? 1. - qual[1] : qual[1] / 3.;
+                               u = set_u(bw, 1, k);
+                               fi[u+0] = e * bM; fi[u+1] = .25 * bI;
+                               sum += fi[u] + fi[u+1];
+                       }
+                       // rescale
+                       s[1] = sum;
+                       _beg = set_u(bw, 1, beg); _end = set_u(bw, 1, end); _end += 2;
+                       for (k = _beg; k <= _end; ++k) fi[k] /= sum;
+               }
+               // f[2..l_query]
+               for (i = 2; i <= l_query; ++i) {
+                       double[] fi = f[i], fi1 = f[i-1];
+                       double sum, qli = qual[i];
+                       int beg = 1, end = l_ref, x, _beg, _end;
+                       byte qyi = query[i];
+                       x = i - bw; beg = beg > x? beg : x; // band start
+                       x = i + bw; end = end < x? end : x; // band end
+                       for (k = beg, sum = 0.; k <= end; ++k) {
+                               int u, v11, v01, v10;
+                               double e;
+                               e = (ref[k] > 3 || qyi > 3)? 1. : ref[k] == qyi? 1. - qli : qli / 3.;
+                               u = set_u(bw, i, k); v11 = set_u(bw, i-1, k-1); v10 = set_u(bw, i-1, k); v01 = set_u(bw, i, k-1);
+                               fi[u+0] = e * (m[0] * fi1[v11+0] + m[3] * fi1[v11+1] + m[6] * fi1[v11+2]);
+                               fi[u+1] = .25 * (m[1] * fi1[v10+0] + m[4] * fi1[v10+1]);
+                               fi[u+2] = m[2] * fi[v01+0] + m[8] * fi[v01+2];
+                               sum += fi[u] + fi[u+1] + fi[u+2];
+                               //System.out.println("("+i+","+k+";"+u+"): "+fi[u]+","+fi[u+1]+","+fi[u+2]);
+                       }
+                       // rescale
+                       s[i] = sum;
+                       _beg = set_u(bw, i, beg); _end = set_u(bw, i, end); _end += 2;
+                       for (k = _beg, sum = 1./sum; k <= _end; ++k) fi[k] *= sum;
+               }
+               { // f[l_query+1]
+                       double sum;
+                       for (k = 1, sum = 0.; k <= l_ref; ++k) {
+                               int u = set_u(bw, l_query, k);
+                               if (u < 3 || u >= bw2*3+3) continue;
+                               sum += f[l_query][u+0] * sM + f[l_query][u+1] * sI;
+                       }
+                       s[l_query+1] = sum; // the last scaling factor
+               }
+               /*** backward ***/
+               // b[l_query] (b[l_query+1][0]=1 and thus \tilde{b}[][]=1/s[l_query+1]; this is where s[l_query+1] comes from)
+               for (k = 1; k <= l_ref; ++k) {
+                       int u = set_u(bw, l_query, k);
+                       double[] bi = b[l_query];
+                       if (u < 3 || u >= bw2*3+3) continue;
+                       bi[u+0] = sM / s[l_query] / s[l_query+1]; bi[u+1] = sI / s[l_query] / s[l_query+1];
+               }
+               // b[l_query-1..1]
+               for (i = l_query - 1; i >= 1; --i) {
+                       int beg = 1, end = l_ref, x, _beg, _end;
+                       double[] bi = b[i], bi1 = b[i+1];
+                       double y = (i > 1)? 1. : 0., qli1 = qual[i+1];
+                       byte qyi1 = query[i+1];
+                       x = i - bw; beg = beg > x? beg : x;
+                       x = i + bw; end = end < x? end : x;
+                       for (k = end; k >= beg; --k) {
+                               int u, v11, v01, v10;
+                               double e;
+                               u = set_u(bw, i, k); v11 = set_u(bw, i+1, k+1); v10 = set_u(bw, i+1, k); v01 = set_u(bw, i, k+1);
+                               e = (k >= l_ref? 0 : (ref[k+1] > 3 || qyi1 > 3)? 1. : ref[k+1] == qyi1? 1. - qli1 : qli1 / 3.) * bi1[v11];
+                               bi[u+0] = e * m[0] + .25 * m[1] * bi1[v10+1] + m[2] * bi[v01+2]; // bi1[v11] has been foled into e.
+                               bi[u+1] = e * m[3] + .25 * m[4] * bi1[v10+1];
+                               bi[u+2] = (e * m[6] + m[8] * bi[v01+2]) * y;
+                       }
+                       // rescale
+                       _beg = set_u(bw, i, beg); _end = set_u(bw, i, end); _end += 2;
+                       for (k = _beg, y = 1./s[i]; k <= _end; ++k) bi[k] *= y;
+               }
+               double pb;
+               { // b[0]
+                       int beg = 1, end = l_ref < bw + 1? l_ref : bw + 1;
+                       double sum = 0.;
+                       for (k = end; k >= beg; --k) {
+                               int u = set_u(bw, 1, k);
+                               double e = (ref[k] > 3 || query[1] > 3)? 1. : ref[k] == query[1]? 1. - qual[1] : qual[1] / 3.;
+                               if (u < 3 || u >= bw2*3+3) continue;
+                               sum += e * b[1][u+0] * bM + .25 * b[1][u+1] * bI;
+                       }
+                       pb = b[0][set_u(bw, 0, 0)] = sum / s[0]; // if everything works as is expected, pb == 1.0
+               }
+               int is_diff = Math.abs(pb - 1.) > 1e-7? 1 : 0;
+               /*** MAP ***/
+               for (i = 1; i <= l_query; ++i) {
+                       double sum = 0., max = 0.;
+                       double[] fi = f[i], bi = b[i];
+                       int beg = 1, end = l_ref, x, max_k = -1;
+                       x = i - bw; beg = beg > x? beg : x;
+                       x = i + bw; end = end < x? end : x;
+                       for (k = beg; k <= end; ++k) {
+                               int u = set_u(bw, i, k);
+                               double z;
+                               sum += (z = fi[u+0] * bi[u+0]); if (z > max) { max = z; max_k = (k-1)<<2 | 0; }
+                               sum += (z = fi[u+1] * bi[u+1]); if (z > max) { max = z; max_k = (k-1)<<2 | 1; }
+                       }
+                       max /= sum; sum *= s[i]; // if everything works as is expected, sum == 1.0
+                       if (state != null) state[i-1] = max_k;
+                       if (q != null) {
+                               k = (int)(-4.343 * Math.log(1. - max) + .499);
+                               q[i-1] = (byte)(k > 100? 99 : k);
+                       }
+                       //System.out.println("("+pb+","+sum+")"+" ("+(i-1)+","+(max_k>>2)+","+(max_k&3)+","+max+")");
+               }
+               return 0;
+       }
+
+       public static void main(String[] args) {
+               byte[] ref = {'\0', '\1', '\3', '\3', '\1'};
+               byte[] query = {'\0', '\3', '\3', '\1'};
+               byte[] qual = new byte[4];
+               qual[0] = qual[1] = qual[2] = qual[3] = (byte)20;
+               HmmGlocal hg = new HmmGlocal(1e-3, 0.1, 7);
+               hg.hmm_glocal(ref, query, qual, null, null);
+       }
+}
\ No newline at end of file

diff --git a/misc/Makefile b/misc/Makefile
index 2d7139d4e8a5c62068afb381f47493f8408bc681..6c25c782eb4e2ad784563991bd69922e885737cb 100644 (file)
--- a/misc/Makefile
+++ b/misc/Makefile
@@ -1,6 +1,6 @@
 CC=                    gcc
 CXX=           g++
 CC=                    gcc
 CXX=           g++

-CFLAGS=                -g -Wall #-O2 #-m64 #-arch ppc
+CFLAGS=                -g -Wall -O2 #-m64 #-arch ppc

 CXXFLAGS=      $(CFLAGS)
 DFLAGS=                -D_FILE_OFFSET_BITS=64
 OBJS=          
 CXXFLAGS=      $(CFLAGS)
 DFLAGS=                -D_FILE_OFFSET_BITS=64
 OBJS=          

diff --git a/misc/samtools.pl b/misc/samtools.pl
index 9f48b8f7586899a9b9456b3fdc09fb8a28faa3d3..ce8449de622884847003ba538a2cf60c57118c99 100755 (executable)
--- a/misc/samtools.pl
+++ b/misc/samtools.pl
@@ -104,7 +104,6 @@ Options: -Q INT    minimum RMS mapping quality for SNPs [$opts{Q}]
     my $len=0;
        if ($flt == 0) {
          if ($t[2] eq '*') { # an indel
     my $len=0;
        if ($flt == 0) {
          if ($t[2] eq '*') { # an indel

-        

         # If deletion, remember the length of the deletion
         my ($a,$b) = split(m{/},$t[3]);
         my $alen = length($a) - 1;
         # If deletion, remember the length of the deletion
         my ($a,$b) = split(m{/},$t[3]);
         my $alen = length($a) - 1;

diff --git a/razip.c b/razip.c
index dff9347770f281fc90631dcc24fbe77f7fc5cc94..825e73268114736530e35415f19f47af23dac03e 100644 (file)
--- a/razip.c
+++ b/razip.c
@@ -94,7 +94,7 @@ int main(int argc, char **argv)
                buffer = malloc(WINDOW_SIZE);
                while((c = read(f_src, buffer, WINDOW_SIZE)) > 0) razf_write(rz, buffer, c);
                razf_close(rz); // f_dst will be closed here
                buffer = malloc(WINDOW_SIZE);
                while((c = read(f_src, buffer, WINDOW_SIZE)) > 0) razf_write(rz, buffer, c);
                razf_close(rz); // f_dst will be closed here

-               if (argc > optind) unlink(argv[optind]);
+               if (argc > optind && !pstdout) unlink(argv[optind]);

                free(buffer);
                close(f_src);
                return 0;
                free(buffer);
                close(f_src);
                return 0;

diff --git a/sam_view.c b/sam_view.c
index 3b10e2e5372e98a2b9ebda4bd5791dc1b837a20f..d0fdad25445508e90052970a90de92e1f0f61ae5 100644 (file)
--- a/sam_view.c
+++ b/sam_view.c
@@ -124,15 +124,18 @@ int main_samview(int argc, char *argv[])
        if (fn_list == 0 && fn_ref) fn_list = samfaipath(fn_ref);
        // open file handlers
        if ((in = samopen(argv[optind], in_mode, fn_list)) == 0) {
        if (fn_list == 0 && fn_ref) fn_list = samfaipath(fn_ref);
        // open file handlers
        if ((in = samopen(argv[optind], in_mode, fn_list)) == 0) {

-               fprintf(stderr, "[main_samview] fail to open file for reading.\n");
+               fprintf(stderr, "[main_samview] fail to open \"%s\" for reading.\n", argv[optind]);
+               ret = 1;

                goto view_end;
        }
        if (in->header == 0) {
                goto view_end;
        }
        if (in->header == 0) {

-               fprintf(stderr, "[main_samview] fail to read the header.\n");
+               fprintf(stderr, "[main_samview] fail to read the header from \"%s\".\n", argv[optind]);
+               ret = 1;

                goto view_end;
        }
        if ((out = samopen(fn_out? fn_out : "-", out_mode, in->header)) == 0) {
                goto view_end;
        }
        if ((out = samopen(fn_out? fn_out : "-", out_mode, in->header)) == 0) {

-               fprintf(stderr, "[main_samview] fail to open file for writing.\n");
+               fprintf(stderr, "[main_samview] fail to open \"%s\" for writing.\n", fn_out? fn_out : "standard output");
+               ret = 1;

                goto view_end;
        }
        if (is_header_only) goto view_end; // no need to print alignments
                goto view_end;
        }
        if (is_header_only) goto view_end; // no need to print alignments


@@ -146,7 +149,10 @@ int main_samview(int argc, char *argv[])
                                samwrite(out, b); // write the alignment to `out'
                        }
                }
                                samwrite(out, b); // write the alignment to `out'
                        }
                }

-               if (r < -1) fprintf(stderr, "[main_samview] truncated file.\n");
+               if (r < -1) {
+                       fprintf(stderr, "[main_samview] truncated file.\n");
+                       ret = 1;
+               }

                bam_destroy1(b);
        } else { // retrieve alignments in specified regions
                int i;
                bam_destroy1(b);
        } else { // retrieve alignments in specified regions
                int i;


@@ -161,10 +167,15 @@ int main_samview(int argc, char *argv[])
                        int tid, beg, end;
                        bam_parse_region(in->header, argv[i], &tid, &beg, &end); // parse a region in the format like `chr2:100-200'
                        if (tid < 0) { // reference name is not found
                        int tid, beg, end;
                        bam_parse_region(in->header, argv[i], &tid, &beg, &end); // parse a region in the format like `chr2:100-200'
                        if (tid < 0) { // reference name is not found

-                               fprintf(stderr, "[main_samview] fail to get the reference name. Continue anyway.\n");
+                               fprintf(stderr, "[main_samview] region \"%s\" specifies an unknown reference name. Continue anyway.\n", argv[i]);

                                continue;
                        }
                                continue;
                        }

-                       bam_fetch(in->x.bam, idx, tid, beg, end, out, view_func); // fetch alignments
+                       // fetch alignments
+                       if (bam_fetch(in->x.bam, idx, tid, beg, end, out, view_func) < 0) {
+                               fprintf(stderr, "[main_samview] retrieval of region \"%s\" failed due to truncated file or corrupt BAM index file\n", argv[i]);
+                               ret = 1;
+                               break;
+                       }

                }
                bam_index_destroy(idx); // destroy the BAM index
        }
                }
                bam_index_destroy(idx); // destroy the BAM index
        }

diff --git a/sample.c b/sample.c
new file mode 100644 (file)
index 0000000..b3d2642
--- /dev/null
+++ b/sample.c
@@ -0,0 +1,92 @@
+#include <stdlib.h>
+#include <string.h>
+#include "sample.h"
+#include "khash.h"
+KHASH_MAP_INIT_STR(sm, int)
+
+bam_sample_t *bam_smpl_init(void)
+{
+       bam_sample_t *s;
+       s = calloc(1, sizeof(bam_sample_t));
+       s->rg2smid = kh_init(sm);
+       s->sm2id = kh_init(sm);
+       return s;
+}
+
+void bam_smpl_destroy(bam_sample_t *sm)
+{
+       int i;
+       khint_t k;
+       khash_t(sm) *rg2smid = (khash_t(sm)*)sm->rg2smid;
+       if (sm == 0) return;
+       for (i = 0; i < sm->n; ++i) free(sm->smpl[i]);
+       free(sm->smpl);
+       for (k = kh_begin(rg2smid); k != kh_end(rg2smid); ++k)
+               if (kh_exist(rg2smid, k)) free((char*)kh_key(rg2smid, k));
+       kh_destroy(sm, sm->rg2smid);
+       kh_destroy(sm, sm->sm2id);
+       free(sm);
+}
+
+static void add_pair(bam_sample_t *sm, khash_t(sm) *sm2id, const char *key, const char *val)
+{
+       khint_t k_rg, k_sm;
+       int ret;
+       khash_t(sm) *rg2smid = (khash_t(sm)*)sm->rg2smid;
+       k_rg = kh_get(sm, rg2smid, key);
+       if (k_rg != kh_end(rg2smid)) return; // duplicated @RG-ID
+       k_rg = kh_put(sm, rg2smid, strdup(key), &ret);
+       k_sm = kh_get(sm, sm2id, val);
+       if (k_sm == kh_end(sm2id)) { // absent
+               if (sm->n == sm->m) {
+                       sm->m = sm->m? sm->m<<1 : 1;
+                       sm->smpl = realloc(sm->smpl, sizeof(void*) * sm->m);
+               }
+               sm->smpl[sm->n] = strdup(val);
+               k_sm = kh_put(sm, sm2id, sm->smpl[sm->n], &ret);
+               kh_val(sm2id, k_sm) = sm->n++;
+       }
+       kh_val(rg2smid, k_rg) = kh_val(sm2id, k_sm);
+}
+
+int bam_smpl_add(bam_sample_t *sm, const char *fn, const char *txt)
+{
+       const char *p = txt, *q, *r;
+       kstring_t buf;
+       int n = 0;
+       khash_t(sm) *sm2id = (khash_t(sm)*)sm->sm2id;
+       memset(&buf, 0, sizeof(kstring_t));
+       while ((q = strstr(p, "@RG")) != 0) {
+               p = q + 3;
+               r = q = 0;
+               if ((q = strstr(p, "\tID:")) != 0) q += 4;
+               if ((r = strstr(p, "\tSM:")) != 0) r += 4;
+               if (r && q) {
+                       char *u, *v;
+                       int oq, or;
+                       for (u = (char*)q; *u && *u != '\t' && *u != '\n'; ++u);
+                       for (v = (char*)r; *v && *v != '\t' && *v != '\n'; ++v);
+                       oq = *u; or = *v; *u = *v = '\0';
+                       buf.l = 0; kputs(fn, &buf); kputc('/', &buf); kputs(q, &buf);
+                       add_pair(sm, sm2id, buf.s, r);
+                       *u = oq; *v = or;
+               } else break;
+               p = q > r? q : r;
+               ++n;
+       }
+       if (n == 0) add_pair(sm, sm2id, fn, fn);
+       free(buf.s);
+       return 0;
+}
+
+int bam_smpl_rg2smid(const bam_sample_t *sm, const char *fn, const char *rg, kstring_t *str)
+{
+       khint_t k;
+       khash_t(sm) *rg2smid = (khash_t(sm)*)sm->rg2smid;
+       if (rg) {
+               str->l = 0;
+               kputs(fn, str); kputc('/', str); kputs(rg, str);
+               k = kh_get(sm, rg2smid, str->s);
+       } else k = kh_get(sm, rg2smid, fn);
+       return k == kh_end(rg2smid)? -1 : kh_val(rg2smid, k);
+}

diff --git a/sample.h b/sample.h
new file mode 100644 (file)
index 0000000..85fe499
--- /dev/null
+++ b/sample.h
@@ -0,0 +1,17 @@
+#ifndef BAM_SAMPLE_H
+#define BAM_SAMPLE_H
+
+#include "kstring.h"
+
+typedef struct {
+       int n, m;
+       char **smpl;
+       void *rg2smid, *sm2id;
+} bam_sample_t;
+
+bam_sample_t *bam_smpl_init(void);
+int bam_smpl_add(bam_sample_t *sm, const char *abs, const char *txt);
+int bam_smpl_rg2smid(const bam_sample_t *sm, const char *fn, const char *rg, kstring_t *str);
+void bam_smpl_destroy(bam_sample_t *sm);
+
+#endif

diff --git a/samtools.1 b/samtools.1
index d79d176468ebe53f41bd45e7447cd1d2b0704a4e..cc421d4bc34fd53efa123b4630fcbccb4ca58708 100644 (file)
--- a/samtools.1
+++ b/samtools.1
@@ -1,4 +1,4 @@
-.TH samtools 1 "11 July 2010" "samtools-0.1.8" "Bioinformatics tools"
+.TH samtools 1 "27 October 2010" "samtools-0.1.9" "Bioinformatics tools"

 .SH NAME
 .PP
 samtools - Utilities for the Sequence Alignment/Map (SAM) format
 .SH NAME
 .PP
 samtools - Utilities for the Sequence Alignment/Map (SAM) format


@@ -18,9 +18,9 @@ samtools merge out.bam in1.bam in2.bam in3.bam
 .PP
 samtools faidx ref.fasta
 .PP
 .PP
 samtools faidx ref.fasta
 .PP

-samtools pileup -f ref.fasta aln.sorted.bam
+samtools pileup -vcf ref.fasta aln.sorted.bam

 .PP
 .PP

-samtools mpileup -f ref.fasta -r chr3:1,000-2,000 in1.bam in2.bam
+samtools mpileup -C50 -agf ref.fasta -r chr3:1,000-2,000 in1.bam in2.bam

 .PP
 samtools tview aln.sorted.bam ref.fasta
 
 .PP
 samtools tview aln.sorted.bam ref.fasta
 


@@ -65,10 +65,12 @@ format: `chr2' (the whole chr2), `chr2:1000000' (region starting from
 .B -b
 Output in the BAM format.
 .TP
 .B -b
 Output in the BAM format.
 .TP

-.B -u
-Output uncompressed BAM. This option saves time spent on
-compression/decomprssion and is thus preferred when the output is piped
-to another samtools command.
+.BI -f " INT"
+Only output alignments with all bits in INT present in the FLAG
+field. INT can be in hex in the format of /^0x[0-9A-F]+/ [0]
+.TP
+.BI -F " INT"
+Skip alignments with bits present in INT [0]

 .TP
 .B -h
 Include the header in the output.
 .TP
 .B -h
 Include the header in the output.


@@ -76,12 +78,29 @@ Include the header in the output.
 .B -H
 Output the header only.
 .TP
 .B -H
 Output the header only.
 .TP

+.BI -l " STR"
+Only output reads in library STR [null]
+.TP
+.BI -o " FILE"
+Output file [stdout]
+.TP
+.BI -q " INT"
+Skip alignments with MAPQ smaller than INT [0]
+.TP
+.BI -r " STR"
+Only output reads in read group STR [null]
+.TP
+.BI -R " FILE"
+Output reads in read groups listed in
+.I FILE
+[null]
+.TP

 .B -S
 Input is in SAM. If @SQ header lines are absent, the
 .B `-t'
 option is required.
 .TP
 .B -S
 Input is in SAM. If @SQ header lines are absent, the
 .B `-t'
 option is required.
 .TP

-.B -t FILE
+.BI -t " FILE"

 This file is TAB-delimited. Each line must contain the reference name
 and the length of the reference, one line for each distinct reference;
 additional fields are ignored. This file also defines the order of the
 This file is TAB-delimited. Each line must contain the reference name
 and the length of the reference, one line for each distinct reference;
 additional fields are ignored. This file also defines the order of the


@@ -92,29 +111,10 @@ can be used as this
 .I <in.ref_list>
 file.
 .TP
 .I <in.ref_list>
 file.
 .TP

-.B -o FILE
-Output file [stdout]
-.TP
-.B -f INT
-Only output alignments with all bits in INT present in the FLAG
-field. INT can be in hex in the format of /^0x[0-9A-F]+/ [0]
-.TP
-.B -F INT
-Skip alignments with bits present in INT [0]
-.TP
-.B -q INT
-Skip alignments with MAPQ smaller than INT [0]
-.TP
-.B -l STR
-Only output reads in library STR [null]
-.TP
-.B -r STR
-Only output reads in read group STR [null]
-.TP
-.B -R FILE
-Output reads in read groups listed in
-.I FILE
-[null]
+.B -u
+Output uncompressed BAM. This option saves time spent on
+compression/decomprssion and is thus preferred when the output is piped
+to another samtools command.

 .RE
 
 .TP
 .RE
 
 .TP


@@ -128,8 +128,10 @@ viewing the same reference sequence.
 
 .TP
 .B pileup
 
 .TP
 .B pileup

-samtools pileup [-f in.ref.fasta] [-t in.ref_list] [-l in.site_list]
-[-iscgS2] [-T theta] [-N nHap] [-r pairDiffRate] <in.bam>|<in.sam>
+samtools pileup [-2sSBicv] [-f in.ref.fasta] [-t in.ref_list] [-l
+in.site_list] [-C capMapQ] [-M maxMapQ] [-T theta] [-N nHap] [-r
+pairDiffRate] [-m mask] [-d maxIndelDepth] [-G indelPrior]
+<in.bam>|<in.sam>

 
 Print the alignment in the pileup format. In the pileup format, each
 line represents a genomic position, consisting of chromosome name,
 
 Print the alignment in the pileup format. In the pileup format, each
 line represents a genomic position, consisting of chromosome name,


@@ -138,17 +140,17 @@ mapping qualities. Information on match, mismatch, indel, strand,
 mapping quality and start and end of a read are all encoded at the read
 base column. At this column, a dot stands for a match to the reference
 base on the forward strand, a comma for a match on the reverse strand,
 mapping quality and start and end of a read are all encoded at the read
 base column. At this column, a dot stands for a match to the reference
 base on the forward strand, a comma for a match on the reverse strand,

-`ACGTN' for a mismatch on the forward strand and `acgtn' for a mismatch
-on the reverse strand. A pattern `\\+[0-9]+[ACGTNacgtn]+' indicates
-there is an insertion between this reference position and the next
-reference position. The length of the insertion is given by the integer
-in the pattern, followed by the inserted sequence. Similarly, a pattern
-`-[0-9]+[ACGTNacgtn]+' represents a deletion from the reference. The
-deleted bases will be presented as `*' in the following lines. Also at
-the read base column, a symbol `^' marks the start of a read segment
-which is a contiguous subsequence on the read separated by `N/S/H' CIGAR
-operations. The ASCII of the character following `^' minus 33 gives the
-mapping quality. A symbol `$' marks the end of a read segment.
+a '>' or '<' for a reference skip, `ACGTN' for a mismatch on the forward
+strand and `acgtn' for a mismatch on the reverse strand. A pattern
+`\\+[0-9]+[ACGTNacgtn]+' indicates there is an insertion between this
+reference position and the next reference position. The length of the
+insertion is given by the integer in the pattern, followed by the
+inserted sequence. Similarly, a pattern `-[0-9]+[ACGTNacgtn]+'
+represents a deletion from the reference. The deleted bases will be
+presented as `*' in the following lines. Also at the read base column, a
+symbol `^' marks the start of a read. The ASCII of the character
+following `^' minus 33 gives the mapping quality. A symbol `$' marks the
+end of a read segment.

 
 If option
 .B -c
 
 If option
 .B -c


@@ -168,102 +170,139 @@ The position of indels is offset by -1.
 .B OPTIONS:
 .RS
 .TP 10
 .B OPTIONS:
 .RS
 .TP 10

-.B -s
-Print the mapping quality as the last column. This option makes the
-output easier to parse, although this format is not space efficient.
+.B -B
+Disable the BAQ computation. See the
+.B mpileup
+command for details.

 .TP
 .TP

-.B -S
-The input file is in SAM.
+.B -c
+Call the consensus sequence using SOAPsnp consensus model. Options
+.BR -T ", " -N ", " -I " and " -r
+are only effective when
+.BR -c " or " -g
+is in use.

 .TP
 .TP

-.B -i
-Only output pileup lines containing indels.
+.BI -C " INT"
+Coefficient for downgrading the mapping quality of poorly mapped
+reads. See the
+.B mpileup
+command for details. [0]

 .TP
 .TP

-.B -f FILE
+.BI -d " INT"
+Use the first
+.I NUM
+reads in the pileup for indel calling for speed up. Zero for unlimited. [1024]
+.TP
+.BI -f " FILE"

 The reference sequence in the FASTA format. Index file
 .I FILE.fai
 will be created if
 absent.
 .TP
 The reference sequence in the FASTA format. Index file
 .I FILE.fai
 will be created if
 absent.
 .TP

-.B -M INT
-Cap mapping quality at INT [60]
+.B -g
+Generate genotype likelihood in the binary GLFv3 format. This option
+suppresses -c, -i and -s. This option is deprecated by the
+.B mpileup
+command.

 .TP
 .TP

-.B -m INT
+.B -i
+Only output pileup lines containing indels.
+.TP
+.BI -I " INT"
+Phred probability of an indel in sequencing/prep. [40]
+.TP
+.BI -l " FILE"
+List of sites at which pileup is output. This file is space
+delimited. The first two columns are required to be chromosome and
+1-based coordinate. Additional columns are ignored. It is
+recommended to use option
+.TP
+.BI -m " INT"

 Filter reads with flag containing bits in
 Filter reads with flag containing bits in

-.I
-INT
+.I INT

 [1796]
 .TP
 [1796]
 .TP

-.B -d INT
-Use the first
-.I NUM
-reads in the pileup for indel calling for speed up. Zero for unlimited. [0]
+.BI -M " INT"
+Cap mapping quality at INT [60]
+.TP
+.BI -N " INT"
+Number of haplotypes in the sample (>=2) [2]

 .TP
 .TP

-.B -t FILE
+.BI -r " FLOAT"
+Expected fraction of differences between a pair of haplotypes [0.001]
+.TP
+.B -s
+Print the mapping quality as the last column. This option makes the
+output easier to parse, although this format is not space efficient.
+.TP
+.B -S
+The input file is in SAM.
+.TP
+.BI -t " FILE"

 List of reference names ane sequence lengths, in the format described
 for the
 .B import
 command. If this option is present, samtools assumes the input
 .I <in.alignment>
 is in SAM format; otherwise it assumes in BAM format.
 List of reference names ane sequence lengths, in the format described
 for the
 .B import
 command. If this option is present, samtools assumes the input
 .I <in.alignment>
 is in SAM format; otherwise it assumes in BAM format.

-.TP
-.B -l FILE
-List of sites at which pileup is output. This file is space
-delimited. The first two columns are required to be chromosome and
-1-based coordinate. Additional columns are ignored. It is
-recommended to use option

 .B -s
 together with
 .B -l
 as in the default format we may not know the mapping quality.
 .TP
 .B -s
 together with
 .B -l
 as in the default format we may not know the mapping quality.
 .TP

-.B -c
-Call the consensus sequence using SOAPsnp consensus model. Options
-.B -T,
-.B -N,
-.B -I
-and
-.B -r
-are only effective when
-.B -c
-or
-.B -g
-is in use.
-.TP
-.B -g
-Generate genotype likelihood in the binary GLFv3 format. This option
-suppresses -c, -i and -s.
-.TP
-.B -T FLOAT
+.BI -T " FLOAT"

 The theta parameter (error dependency coefficient) in the maq consensus
 calling model [0.85]
 The theta parameter (error dependency coefficient) in the maq consensus
 calling model [0.85]

-.TP
-.B -N INT
-Number of haplotypes in the sample (>=2) [2]
-.TP
-.B -r FLOAT
-Expected fraction of differences between a pair of haplotypes [0.001]
-.TP
-.B -I INT
-Phred probability of an indel in sequencing/prep. [40]

 .RE
 
 .TP
 .B mpileup
 .RE
 
 .TP
 .B mpileup

-samtools mpileup [-r reg] [-f in.fa] in.bam [in2.bam [...]]
+samtools mpileup [-Bug] [-C capQcoef] [-r reg] [-f in.fa] [-l list] [-M capMapQ] [-Q minBaseQ] [-q minMapQ] in.bam [in2.bam [...]]

 
 

-Generate pileup for multiple BAM files. Consensus calling is not
-implemented.
+Generate BCF or pileup for one or multiple BAM files. Alignment records
+are grouped by sample identifiers in @RG header lines. If sample
+identifiers are absent, each input file is regarded as one sample.

 
 .B OPTIONS:
 .RS
 .TP 8
 
 .B OPTIONS:
 .RS
 .TP 8

-.B -r STR
+.B -B
+Disable probabilistic realignment for the computation of base alignment
+quality (BAQ). BAQ is the Phred-scaled probability of a read base being
+misaligned. Applying this option greatly helps to reduce false SNPs
+caused by misalignments.
+.TP
+.BI -C " INT"
+Coefficient for downgrading mapping quality for reads containing
+excessive mismatches. Given a read with a phred-scaled probability q of
+being generated from the mapped position, the new mapping quality is
+about sqrt((INT-q)/INT)*INT. A zero value disables this
+functionality; if enabled, the recommended value is 50. [0]
+.TP
+.BI -f " FILE"
+The reference file [null]
+.TP
+.B -g
+Compute genotype likelihoods and output them in the binary call format (BCF).
+.TP
+.B -u
+Similar to
+.B -g
+except that the output is uncompressed BCF, which is preferred for pipeing.
+.TP
+.BI -l " FILE"
+File containing a list of sites where pileup or BCF is outputted [null]
+.TP
+.BI -q " INT"
+Minimum mapping quality for an alignment to be used [0]
+.TP
+.BI -Q " INT"
+Minimum base quality for a base to be considered [13]
+.TP
+.BI -r " STR"

 Only generate pileup in region
 .I STR
 [all sites]
 Only generate pileup in region
 .I STR
 [all sites]

-.TP
-.B -f FILE
-The reference file [null]

 .RE
 
 .TP
 .RE
 
 .TP


@@ -303,7 +342,7 @@ Approximately the maximum required memory. [500000000]
 
 .TP
 .B merge
 
 .TP
 .B merge

-samtools merge [-h inh.sam] [-nr] <out.bam> <in1.bam> <in2.bam> [...]
+samtools merge [-nur] [-h inh.sam] [-R reg] <out.bam> <in1.bam> <in2.bam> [...]

 
 Merge multiple sorted alignments.
 The header reference lists of all the input BAM files, and the @SQ headers of
 
 Merge multiple sorted alignments.
 The header reference lists of all the input BAM files, and the @SQ headers of


@@ -320,7 +359,7 @@ and the headers of other files will be ignored.
 .B OPTIONS:
 .RS
 .TP 8
 .B OPTIONS:
 .RS
 .TP 8

-.B -h FILE
+.BI -h " FILE"

 Use the lines of
 .I FILE
 as `@' headers to be copied to
 Use the lines of
 .I FILE
 as `@' headers to be copied to


@@ -331,12 +370,19 @@ replacing any header lines that would otherwise be copied from
 is actually in SAM format, though any alignment records it may contain
 are ignored.)
 .TP
 is actually in SAM format, though any alignment records it may contain
 are ignored.)
 .TP

+.BI -R " STR"
+Merge files in the specified region indicated by
+.I STR
+.TP

 .B -r
 Attach an RG tag to each alignment. The tag value is inferred from file names.
 .TP
 .B -n
 The input alignments are sorted by read names rather than by chromosomal
 coordinates
 .B -r
 Attach an RG tag to each alignment. The tag value is inferred from file names.
 .TP
 .B -n
 The input alignments are sorted by read names rather than by chromosomal
 coordinates

+.TP
+.B -u
+Uncompressed BAM output

 .RE
 
 .TP
 .RE
 
 .TP


@@ -402,7 +448,7 @@ Treat paired-end reads and single-end reads.
 
 .TP
 .B calmd
 
 .TP
 .B calmd

-samtools calmd [-eubS] <aln.bam> <ref.fasta>
+samtools calmd [-eubSr] [-C capQcoef] <aln.bam> <ref.fasta>

 
 Generate the MD tag. If the MD tag is already present, this command will
 give a warning if the MD tag generated is different from the existing
 
 Generate the MD tag. If the MD tag is already present, this command will
 give a warning if the MD tag generated is different from the existing


@@ -423,6 +469,15 @@ Output compressed BAM
 .TP
 .B -S
 The input is SAM with header lines
 .TP
 .B -S
 The input is SAM with header lines

+.TP
+.BI -C " INT"
+Coefficient to cap mapping quality of poorly mapped reads. See the
+.B pileup
+command for details. [0]
+.TP
+.B -r
+Perform probabilistic realignment to compute BAQ, which will be used to
+cap base quality.

 .RE
 
 .SH SAM FORMAT
 .RE
 
 .SH SAM FORMAT


@@ -472,6 +527,125 @@ _
 0x0400 d       the read is either a PCR or an optical duplicate
 .TE
 
 0x0400 d       the read is either a PCR or an optical duplicate
 .TE
 

+.SH EXAMPLES
+.IP o 2
+Import SAM to BAM when
+.B @SQ
+lines are present in the header:
+
+  samtools view -bS aln.sam > aln.bam
+
+If
+.B @SQ
+lines are absent:
+
+  samtools faidx ref.fa
+  samtools view -bt ref.fa.fai aln.sam > aln.bam
+
+where
+.I ref.fa.fai
+is generated automatically by the
+.B faidx
+command.
+
+.IP o 2
+Attach the
+.B RG
+tag while merging sorted alignments:
+
+  perl -e 'print "@RG\\tID:ga\\tSM:hs\\tLB:ga\\tPL:Illumina\\n@RG\\tID:454\\tSM:hs\\tLB:454\\tPL:454\\n"' > rg.txt
+  samtools merge -rh rg.txt merged.bam ga.bam 454.bam
+
+The value in a
+.B RG
+tag is determined by the file name the read is coming from. In this
+example, in the
+.IR merged.bam ,
+reads from
+.I ga.bam
+will be attached 
+.IR RG:Z:ga ,
+while reads from
+.I 454.bam
+will be attached
+.IR RG:Z:454 .
+
+.IP o 2
+Call SNPs and short indels for one diploid individual:
+
+  samtools pileup -vcf ref.fa aln.bam > var.raw.plp
+  samtools.pl varFilter -D 100 var.raw.plp > var.flt.plp
+  awk '($3=="*"&&$6>=50)||($3!="*"&&$6>=20)' var.flt.plp > var.final.plp
+
+The
+.B -D
+option of varFilter controls the maximum read depth, which should be
+adjusted to about twice the average read depth.  One may consider to add
+.B -C50
+to
+.B pileup
+if mapping quality is overestimated for reads containing excessive
+mismatches. Applying this option usually helps
+.B BWA-short
+but may not other mappers. It also potentially increases reference
+biases.
+
+.IP o 2
+Call SNPs (not short indels) for multiple diploid individuals:
+
+  samtools mpileup -augf ref.fa *.bam | bcftools view -bcv - > snp.raw.bcf
+  bcftools view snp.raw.bcf | vcfutils.pl filter4vcf -D 2000 | bgzip > snp.flt.vcf.gz
+
+Individuals are identified from the
+.B SM
+tags in the
+.B @RG
+header lines. Individuals can be pooled in one alignment file; one
+individual can also be separated into multiple files. Similarly, one may
+consider to apply
+.B -C50
+to
+.BR mpileup .
+SNP calling in this way also works for single sample and has the
+advantage of enabling more powerful filtering. The drawback is the lack
+of short indel calling, which may be implemented in future.
+
+.IP o 2
+Derive the allele frequency spectrum (AFS) on a list of sites from multiple individuals:
+
+  samtools mpileup -gf ref.fa *.bam > all.bcf
+  bcftools view -bl sites.list all.bcf > sites.bcf
+  bcftools view -cGP cond2 sites.bcf > /dev/null 2> sites.1.afs
+  bcftools view -cGP sites.1.afs sites.bcf > /dev/null 2> sites.2.afs
+  bcftools view -cGP sites.2.afs sites.bcf > /dev/null 2> sites.3.afs
+  ......
+
+where
+.I sites.list
+contains the list of sites with each line consisting of the reference
+sequence name and position. The following
+.B bcftools
+commands estimate AFS by EM.
+
+.IP o 2
+Dump BAQ applied alignment for other SNP callers:
+
+  samtools calmd -br aln.bam > aln.baq.bam
+
+It adds and corrects the
+.B NM
+and
+.B MD
+tags at the same time. The
+.B calmd
+command also comes with the
+.B -C
+option, the same as the on in
+.B pileup
+and
+.BR mpileup .
+Apply if it helps.
+

 .SH LIMITATIONS
 .PP
 .IP o 2
 .SH LIMITATIONS
 .PP
 .IP o 2

diff --git a/samtools.txt b/samtools.txt
index 20e6c15cd707d11a7d1ed41f15b9d54732cd67ec..09c1ccdedf9be5dd1964022519948a3932d384c8 100644 (file)
--- a/samtools.txt
+++ b/samtools.txt
@@ -20,60 +20,73 @@ SYNOPSIS
 
        samtools faidx ref.fasta
 
 
        samtools faidx ref.fasta
 

-       samtools pileup -f ref.fasta aln.sorted.bam
+       samtools pileup -vcf ref.fasta aln.sorted.bam

 
 

-       samtools mpileup -f ref.fasta -r chr3:1,000-2,000 in1.bam in2.bam
+       samtools  mpileup  -C50  -agf  ref.fasta  -r  chr3:1,000-2,000  in1.bam
+       in2.bam

 
        samtools tview aln.sorted.bam ref.fasta
 
 
 DESCRIPTION
 
        samtools tview aln.sorted.bam ref.fasta
 
 
 DESCRIPTION

-       Samtools  is  a  set of utilities that manipulate alignments in the BAM
+       Samtools is a set of utilities that manipulate alignments  in  the  BAM

        format. It imports from and exports to the SAM (Sequence Alignment/Map)
        format. It imports from and exports to the SAM (Sequence Alignment/Map)

-       format,  does  sorting,  merging  and  indexing, and allows to retrieve
+       format, does sorting, merging and  indexing,  and  allows  to  retrieve

        reads in any regions swiftly.
 
        reads in any regions swiftly.
 

-       Samtools is designed to work on a stream. It regards an input file  `-'
-       as  the  standard  input (stdin) and an output file `-' as the standard
+       Samtools  is designed to work on a stream. It regards an input file `-'
+       as the standard input (stdin) and an output file `-'  as  the  standard

        output (stdout). Several commands can thus be combined with Unix pipes.
        Samtools always output warning and error messages to the standard error
        output (stderr).
 
        output (stdout). Several commands can thus be combined with Unix pipes.
        Samtools always output warning and error messages to the standard error
        output (stderr).
 

-       Samtools is also able to open a BAM (not SAM) file on a remote  FTP  or
-       HTTP  server  if  the  BAM file name starts with `ftp://' or `http://'.
-       Samtools checks the current working directory for the  index  file  and
-       will  download  the  index upon absence. Samtools does not retrieve the
+       Samtools  is  also able to open a BAM (not SAM) file on a remote FTP or
+       HTTP server if the BAM file name starts  with  `ftp://'  or  `http://'.
+       Samtools  checks  the  current working directory for the index file and
+       will download the index upon absence. Samtools does  not  retrieve  the

        entire alignment file unless it is asked to do so.
 
 
 COMMANDS AND OPTIONS
        view      samtools  view  [-bhuHS]  [-t  in.refList]  [-o  output]  [-f
        entire alignment file unless it is asked to do so.
 
 
 COMMANDS AND OPTIONS
        view      samtools  view  [-bhuHS]  [-t  in.refList]  [-o  output]  [-f

-                 reqFlag]  [-F  skipFlag]  [-q minMapQ] [-l library] [-r read-
+                 reqFlag] [-F skipFlag] [-q minMapQ] [-l  library]  [-r  read-

                  Group] [-R rgFile] <in.bam>|<in.sam> [region1 [...]]
 
                  Group] [-R rgFile] <in.bam>|<in.sam> [region1 [...]]
 

-                 Extract/print all or sub alignments in SAM or BAM format.  If
-                 no  region  is specified, all the alignments will be printed;
-                 otherwise only alignments overlapping the  specified  regions
-                 will  be  output. An alignment may be given multiple times if
+                 Extract/print  all or sub alignments in SAM or BAM format. If
+                 no region is specified, all the alignments will  be  printed;
+                 otherwise  only  alignments overlapping the specified regions
+                 will be output. An alignment may be given multiple  times  if

                  it is overlapping several regions. A region can be presented,
                  it is overlapping several regions. A region can be presented,

-                 for  example,  in  the  following  format:  `chr2' (the whole
-                 chr2), `chr2:1000000' (region starting from  1,000,000bp)  or
-                 `chr2:1,000,000-2,000,000'   (region  between  1,000,000  and
-                 2,000,000bp including the  end  points).  The  coordinate  is
+                 for example, in  the  following  format:  `chr2'  (the  whole
+                 chr2),  `chr2:1000000'  (region starting from 1,000,000bp) or
+                 `chr2:1,000,000-2,000,000'  (region  between  1,000,000   and
+                 2,000,000bp  including  the  end  points).  The coordinate is

                  1-based.
 
                  OPTIONS:
 
                  -b      Output in the BAM format.
 
                  1-based.
 
                  OPTIONS:
 
                  -b      Output in the BAM format.
 

-                 -u      Output uncompressed BAM. This option saves time spent
-                         on compression/decomprssion  and  is  thus  preferred
-                         when the output is piped to another samtools command.
+                 -f INT  Only output alignments with all bits in  INT  present
+                         in the FLAG field. INT can be in hex in the format of
+                         /^0x[0-9A-F]+/ [0]
+
+                 -F INT  Skip alignments with bits present in INT [0]

 
                  -h      Include the header in the output.
 
                  -H      Output the header only.
 
 
                  -h      Include the header in the output.
 
                  -H      Output the header only.
 

+                 -l STR  Only output reads in library STR [null]
+
+                 -o FILE Output file [stdout]
+
+                 -q INT  Skip alignments with MAPQ smaller than INT [0]
+
+                 -r STR  Only output reads in read group STR [null]
+
+                 -R FILE Output reads in read groups listed in FILE [null]
+

                  -S      Input is in SAM. If @SQ header lines are absent,  the
                          `-t' option is required.
 
                  -S      Input is in SAM. If @SQ header lines are absent,  the
                          `-t' option is required.
 


@@ -85,35 +98,24 @@ COMMANDS AND OPTIONS
                          `samtools faidx <ref.fa>', the resultant  index  file
                          <ref.fa>.fai  can be used as this <in.ref_list> file.
 
                          `samtools faidx <ref.fa>', the resultant  index  file
                          <ref.fa>.fai  can be used as this <in.ref_list> file.
 

-                 -o FILE Output file [stdout]
-
-                 -f INT  Only output alignments with all bits in  INT  present
-                         in the FLAG field. INT can be in hex in the format of
-                         /^0x[0-9A-F]+/ [0]
-
-                 -F INT  Skip alignments with bits present in INT [0]
-
-                 -q INT  Skip alignments with MAPQ smaller than INT [0]
-
-                 -l STR  Only output reads in library STR [null]
-
-                 -r STR  Only output reads in read group STR [null]
-
-                 -R FILE Output reads in read groups listed in FILE [null]
+                 -u      Output uncompressed BAM. This option saves time spent
+                         on  compression/decomprssion  and  is  thus preferred
+                         when the output is piped to another samtools command.

 
 
        tview     samtools tview <in.sorted.bam> [ref.fasta]
 
 
 
        tview     samtools tview <in.sorted.bam> [ref.fasta]
 

-                 Text alignment viewer (based on the ncurses library). In  the
-                 viewer,  press `?' for help and press `g' to check the align-
-                 ment   start   from   a   region   in   the    format    like
-                 `chr10:10,000,000'  or  `=10,000,000'  when  viewing the same
+                 Text  alignment viewer (based on the ncurses library). In the
+                 viewer, press `?' for help and press `g' to check the  align-
+                 ment    start    from   a   region   in   the   format   like
+                 `chr10:10,000,000' or `=10,000,000'  when  viewing  the  same

                  reference sequence.
 
 
                  reference sequence.
 
 

-       pileup    samtools  pileup  [-f  in.ref.fasta]  [-t  in.ref_list]   [-l
-                 in.site_list]    [-iscgS2]   [-T   theta]   [-N   nHap]   [-r
-                 pairDiffRate] <in.bam>|<in.sam>
+       pileup    samtools pileup [-2sSBicv] [-f in.ref.fasta] [-t in.ref_list]
+                 [-l in.site_list] [-C capMapQ] [-M maxMapQ]  [-T  theta]  [-N
+                 nHap]  [-r  pairDiffRate]  [-m  mask]  [-d maxIndelDepth] [-G
+                 indelPrior] <in.bam>|<in.sam>

 
                  Print the alignment in the pileup format. In the pileup  for-
                  mat,  each  line represents a genomic position, consisting of
 
                  Print the alignment in the pileup format. In the pileup  for-
                  mat,  each  line represents a genomic position, consisting of


@@ -123,20 +125,18 @@ COMMANDS AND OPTIONS
                  end  of  a  read  are all encoded at the read base column. At
                  this column, a dot stands for a match to the  reference  base
                  on  the  forward  strand,  a comma for a match on the reverse
                  end  of  a  read  are all encoded at the read base column. At
                  this column, a dot stands for a match to the  reference  base
                  on  the  forward  strand,  a comma for a match on the reverse

-                 strand, `ACGTN' for a mismatch  on  the  forward  strand  and
-                 `acgtn'  for  a  mismatch  on  the  reverse strand. A pattern
-                 `\+[0-9]+[ACGTNacgtn]+'  indicates  there  is  an   insertion
-                 between  this reference position and the next reference posi-
-                 tion. The length of the insertion is given by the integer  in
-                 the  pattern, followed by the inserted sequence. Similarly, a
-                 pattern `-[0-9]+[ACGTNacgtn]+' represents a deletion from the
-                 reference.  The deleted bases will be presented as `*' in the
-                 following lines. Also at the read base column, a  symbol  `^'
-                 marks  the start of a read segment which is a contiguous sub-
-                 sequence on the read separated by `N/S/H'  CIGAR  operations.
-                 The  ASCII  of the character following `^' minus 33 gives the
-                 mapping quality. A symbol `$' marks the end of  a  read  seg-
-                 ment.
+                 strand, a '>' or '<' for a reference skip, `ACGTN' for a mis-
+                 match on the forward strand and `acgtn' for a mismatch on the
+                 reverse strand. A pattern  `\+[0-9]+[ACGTNacgtn]+'  indicates
+                 there is an insertion between this reference position and the
+                 next reference position. The length of the insertion is given
+                 by  the  integer  in  the  pattern,  followed by the inserted
+                 sequence. Similarly, a pattern `-[0-9]+[ACGTNacgtn]+'  repre-
+                 sents  a  deletion from the reference. The deleted bases will
+                 be presented as `*' in the following lines. Also at the  read
+                 base  column,  a  symbol  `^'  marks the start of a read. The
+                 ASCII of the character following `^' minus 33 gives the  map-
+                 ping quality. A symbol `$' marks the end of a read segment.

 
                  If  option  -c  is  applied, the consensus base, Phred-scaled
                  consensus quality, SNP quality (i.e. the Phred-scaled  proba-
 
                  If  option  -c  is  applied, the consensus base, Phred-scaled
                  consensus quality, SNP quality (i.e. the Phred-scaled  proba-


@@ -155,68 +155,109 @@ COMMANDS AND OPTIONS
 
                  OPTIONS:
 
 
                  OPTIONS:
 

-                 -s        Print the mapping quality as the last column.  This
-                           option  makes  the output easier to parse, although
-                           this format is not space efficient.
+                 -B        Disable the BAQ computation. See the  mpileup  com-
+                           mand for details.

 
 

-                 -S        The input file is in SAM.
+                 -c        Call the consensus sequence using SOAPsnp consensus
+                           model. Options -T, -N, -I and -r are only effective
+                           when -c or -g is in use.

 
 

-                 -i        Only output pileup lines containing indels.
+                 -C INT    Coefficient  for downgrading the mapping quality of
+                           poorly mapped reads. See the  mpileup  command  for
+                           details. [0]
+
+                 -d INT    Use  the  first  NUM  reads in the pileup for indel
+                           calling for speed up. Zero for unlimited. [1024]

 
                  -f FILE   The reference sequence in the FASTA  format.  Index
                            file FILE.fai will be created if absent.
 
 
                  -f FILE   The reference sequence in the FASTA  format.  Index
                            file FILE.fai will be created if absent.
 

-                 -M INT    Cap mapping quality at INT [60]
+                 -g        Generate  genotype  likelihood  in the binary GLFv3
+                           format. This option suppresses -c, -i and -s.  This
+                           option is deprecated by the mpileup command.

 
 

-                 -m INT    Filter  reads  with  flag  containing  bits  in INT
-                           [1796]
-
-                 -d INT    Use the first NUM reads in  the  pileup  for  indel
-                           calling for speed up. Zero for unlimited. [0]
-
-                 -t FILE   List  of  reference  names ane sequence lengths, in
-                           the format described for  the  import  command.  If
-                           this  option is present, samtools assumes the input
-                           <in.alignment>  is  in  SAM  format;  otherwise  it
-                           assumes in BAM format.
+                 -i        Only output pileup lines containing indels.

 
 

-                 -l FILE   List  of sites at which pileup is output. This file
-                           is space  delimited.  The  first  two  columns  are
-                           required  to  be chromosome and 1-based coordinate.
-                           Additional columns are ignored. It  is  recommended
-                           to use option -s together with -l as in the default
-                           format we may not know the mapping quality.
+                 -I INT    Phred  probability  of an indel in sequencing/prep.
+                           [40]

 
 

-                 -c        Call the consensus sequence using SOAPsnp consensus
-                           model. Options -T, -N, -I and -r are only effective
-                           when -c or -g is in use.
+                 -l FILE   List of sites at which pileup is output. This  file
+                           is  space  delimited.  The  first  two  columns are
+                           required to be chromosome and  1-based  coordinate.
+                           Additional  columns  are ignored. It is recommended
+                           to use option

 
 

-                 -g        Generate genotype likelihood in  the  binary  GLFv3
-                           format. This option suppresses -c, -i and -s.
+                 -m INT    Filter reads  with  flag  containing  bits  in  INT
+                           [1796]

 
 

-                 -T FLOAT  The  theta parameter (error dependency coefficient)
-                           in the maq consensus calling model [0.85]
+                 -M INT    Cap mapping quality at INT [60]

 
                  -N INT    Number of haplotypes in the sample (>=2) [2]
 
 
                  -N INT    Number of haplotypes in the sample (>=2) [2]
 

-                 -r FLOAT  Expected fraction of differences between a pair  of
+                 -r FLOAT  Expected  fraction of differences between a pair of

                            haplotypes [0.001]
 
                            haplotypes [0.001]
 

-                 -I INT    Phred  probability  of an indel in sequencing/prep.
-                           [40]
+                 -s        Print the mapping quality as the last column.  This
+                           option  makes  the output easier to parse, although
+                           this format is not space efficient.
+
+                 -S        The input file is in SAM.
+
+                 -t FILE   List of reference names ane  sequence  lengths,  in
+                           the  format  described  for  the import command. If
+                           this option is present, samtools assumes the  input
+                           <in.alignment>  is  in  SAM  format;  otherwise  it
+                           assumes in BAM format.  -s together with -l  as  in
+                           the  default  format  we  may  not know the mapping
+                           quality.
+
+                 -T FLOAT  The theta parameter (error dependency  coefficient)
+                           in the maq consensus calling model [0.85]

 
 
 
 

-       mpileup   samtools mpileup [-r reg] [-f in.fa] in.bam [in2.bam [...]]
+       mpileup   samtools mpileup [-Bug] [-C capQcoef] [-r reg] [-f in.fa] [-l
+                 list] [-M capMapQ] [-Q minBaseQ] [-q minMapQ] in.bam [in2.bam
+                 [...]]

 
 

-                 Generate pileup for multiple BAM files. Consensus calling  is
-                 not implemented.
+                 Generate  BCF or pileup for one or multiple BAM files. Align-
+                 ment records are grouped by sample identifiers in @RG  header
+                 lines.  If  sample identifiers are absent, each input file is
+                 regarded as one sample.

 
                  OPTIONS:
 
 
                  OPTIONS:
 

-                 -r STR  Only generate pileup in region STR [all sites]
+                 -B      Disable probabilistic realignment for the computation
+                         of  base  alignment  quality (BAQ). BAQ is the Phred-
+                         scaled probability of a read base  being  misaligned.
+                         Applying  this  option  greatly helps to reduce false
+                         SNPs caused by misalignments.
+
+                 -C INT  Coefficient for downgrading mapping quality for reads
+                         containing  excessive mismatches. Given a read with a
+                         phred-scaled probability q of  being  generated  from
+                         the mapped position, the new mapping quality is about
+                         sqrt((INT-q)/INT)*INT. A  zero  value  disables  this
+                         functionality;  if  enabled, the recommended value is
+                         50. [0]

 
                  -f FILE The reference file [null]
 
 
                  -f FILE The reference file [null]
 

+                 -g      Compute genotype likelihoods and output them  in  the
+                         binary call format (BCF).
+
+                 -u      Similar  to -g except that the output is uncompressed
+                         BCF, which is preferred for pipeing.
+
+                 -l FILE File containing a list of sites where pileup  or  BCF
+                         is outputted [null]
+
+                 -q INT  Minimum  mapping  quality for an alignment to be used
+                         [0]
+
+                 -Q INT  Minimum base quality for a base to be considered [13]
+
+                 -r STR  Only generate pileup in region STR [all sites]
+

 
        reheader  samtools reheader <in.header.sam> <in.bam>
 
 
        reheader  samtools reheader <in.header.sam> <in.bam>
 


@@ -243,8 +284,8 @@ COMMANDS AND OPTIONS
                          [500000000]
 
 
                          [500000000]
 
 

-       merge     samtools  merge  [-h  inh.sam]  [-nr]   <out.bam>   <in1.bam>
-                 <in2.bam> [...]
+       merge     samtools  merge  [-nur]  [-h  inh.sam]  [-R  reg]   <out.bam>
+                 <in1.bam> <in2.bam> [...]

 
                  Merge multiple sorted alignments.  The header reference lists
                  of all the input BAM files, and the @SQ headers  of  inh.sam,
 
                  Merge multiple sorted alignments.  The header reference lists
                  of all the input BAM files, and the @SQ headers  of  inh.sam,


@@ -261,12 +302,16 @@ COMMANDS AND OPTIONS
                          SAM  format, though any alignment records it may con-
                          tain are ignored.)
 
                          SAM  format, though any alignment records it may con-
                          tain are ignored.)
 

+                 -R STR  Merge files in the specified region indicated by STR
+

                  -r      Attach an RG tag to each alignment. The tag value  is
                          inferred from file names.
 
                  -n      The  input alignments are sorted by read names rather
                          than by chromosomal coordinates
 
                  -r      Attach an RG tag to each alignment. The tag value  is
                          inferred from file names.
 
                  -n      The  input alignments are sorted by read names rather
                          than by chromosomal coordinates
 

+                 -u      Uncompressed BAM output
+

 
        index     samtools index <aln.bam>
 
 
        index     samtools index <aln.bam>
 


@@ -315,7 +360,7 @@ COMMANDS AND OPTIONS
                  -S      Treat paired-end reads and single-end reads.
 
 
                  -S      Treat paired-end reads and single-end reads.
 
 

-       calmd     samtools calmd [-eubS] <aln.bam> <ref.fasta>
+       calmd     samtools calmd [-eubSr] [-C capQcoef] <aln.bam> <ref.fasta>

 
                  Generate the MD tag. If the MD tag is already  present,  this
                  command  will  give a warning if the MD tag generated is dif-
 
                  Generate the MD tag. If the MD tag is already  present,  this
                  command  will  give a warning if the MD tag generated is dif-


@@ -333,6 +378,12 @@ COMMANDS AND OPTIONS
 
                  -S      The input is SAM with header lines
 
 
                  -S      The input is SAM with header lines
 

+                 -C INT  Coefficient to cap mapping quality of  poorly  mapped
+                         reads. See the pileup command for details. [0]
+
+                 -r      Perform  probabilistic  realignment  to  compute BAQ,
+                         which will be used to cap base quality.
+

 
 SAM FORMAT
        SAM is TAB-delimited. Apart from the header lines,  which  are  started
 
 SAM FORMAT
        SAM is TAB-delimited. Apart from the header lines,  which  are  started


@@ -375,26 +426,106 @@ SAM FORMAT
           |0x0400 |  d  | the read is either a PCR or an optical duplicate |
           +-------+-----+--------------------------------------------------+
 
           |0x0400 |  d  | the read is either a PCR or an optical duplicate |
           +-------+-----+--------------------------------------------------+
 

+EXAMPLES
+       o Import SAM to BAM when @SQ lines are present in the header:
+
+           samtools view -bS aln.sam > aln.bam
+
+         If @SQ lines are absent:
+
+           samtools faidx ref.fa
+           samtools view -bt ref.fa.fai aln.sam > aln.bam
+
+         where ref.fa.fai is generated automatically by the faidx command.
+
+
+       o Attach the RG tag while merging sorted alignments:
+
+           perl       -e       'print      "@RG\tID:ga\tSM:hs\tLB:ga\tPL:Illu-
+         mina\n@RG\tID:454\tSM:hs\tLB:454\tPL:454\n"' > rg.txt
+           samtools merge -rh rg.txt merged.bam ga.bam 454.bam
+
+         The value in a RG tag is determined by the file name the read is com-
+         ing  from. In this example, in the merged.bam, reads from ga.bam will
+         be attached RG:Z:ga,  while  reads  from  454.bam  will  be  attached
+         RG:Z:454.
+
+
+       o Call SNPs and short indels for one diploid individual:
+
+           samtools pileup -vcf ref.fa aln.bam > var.raw.plp
+           samtools.pl varFilter -D 100 var.raw.plp > var.flt.plp
+           awk     '($3=="*"&&$6>=50)||($3!="*"&&$6>=20)'     var.flt.plp    >
+         var.final.plp
+
+         The -D option of varFilter controls the  maximum  read  depth,  which
+         should  be  adjusted  to about twice the average read depth.  One may
+         consider to add -C50 to pileup if mapping  quality  is  overestimated
+         for  reads containing excessive mismatches. Applying this option usu-
+         ally helps BWA-short but may not other mappers. It  also  potentially
+         increases reference biases.
+
+
+       o Call SNPs (not short indels) for multiple diploid individuals:
+
+           samtools  mpileup  -augf  ref.fa  *.bam  |  bcftools  view -bcv - >
+         snp.raw.bcf
+           bcftools view snp.raw.bcf | vcfutils.pl filter4vcf -D 2000 |  bgzip
+         > snp.flt.vcf.gz
+
+         Individuals  are identified from the SM tags in the @RG header lines.
+         Individuals can be pooled in one alignment file; one  individual  can
+         also be separated into multiple files. Similarly, one may consider to
+         apply -C50 to mpileup.  SNP calling in this way also works for single
+         sample and has the advantage of enabling more powerful filtering. The
+         drawback is the lack of short indel calling, which may be implemented
+         in future.
+
+
+       o Derive  the  allele  frequency spectrum (AFS) on a list of sites from
+         multiple individuals:
+
+           samtools mpileup -gf ref.fa *.bam > all.bcf
+           bcftools view -bl sites.list all.bcf > sites.bcf
+           bcftools view -cGP cond2 sites.bcf > /dev/null 2> sites.1.afs
+           bcftools view -cGP sites.1.afs sites.bcf > /dev/null 2> sites.2.afs
+           bcftools view -cGP sites.2.afs sites.bcf > /dev/null 2> sites.3.afs
+           ......
+
+         where sites.list contains the list of sites with each line consisting
+         of  the  reference sequence name and position. The following bcftools
+         commands estimate AFS by EM.
+
+
+       o Dump BAQ applied alignment for other SNP callers:
+
+           samtools calmd -br aln.bam > aln.baq.bam
+
+         It adds and corrects the NM and MD tags at the same time.  The  calmd
+         command  also  comes with the -C option, the same as the on in pileup
+         and mpileup.  Apply if it helps.
+
+

 LIMITATIONS
 LIMITATIONS

-       o Unaligned   words  used  in  bam_import.c,  bam_endian.h,  bam.c  and
+       o Unaligned  words  used  in  bam_import.c,  bam_endian.h,  bam.c   and

          bam_aux.c.
 
          bam_aux.c.
 

-       o In merging, the input files are required to have the same  number  of
-         reference  sequences.  The  requirement  can be relaxed. In addition,
-         merging does not reconstruct the header  dictionaries  automatically.
-         Endusers  have  to  provide  the  correct header. Picard is better at
+       o In  merging,  the input files are required to have the same number of
+         reference sequences. The requirement can  be  relaxed.  In  addition,
+         merging  does  not reconstruct the header dictionaries automatically.
+         Endusers have to provide the correct  header.  Picard  is  better  at

          merging.
 
          merging.
 

-       o Samtools paired-end rmdup does not  work  for  unpaired  reads  (e.g.
-         orphan  reads  or ends mapped to different chromosomes). If this is a
-         concern, please use Picard's MarkDuplicate  which  correctly  handles
+       o Samtools  paired-end  rmdup  does  not  work for unpaired reads (e.g.
+         orphan reads or ends mapped to different chromosomes). If this  is  a
+         concern,  please  use  Picard's MarkDuplicate which correctly handles

          these cases, although a little slower.
 
 
 AUTHOR
          these cases, although a little slower.
 
 
 AUTHOR

-       Heng  Li from the Sanger Institute wrote the C version of samtools. Bob
+       Heng Li from the Sanger Institute wrote the C version of samtools.  Bob

        Handsaker from the Broad Institute implemented the BGZF library and Jue
        Handsaker from the Broad Institute implemented the BGZF library and Jue

-       Ruan  from  Beijing  Genomics Institute wrote the RAZF library. Various
+       Ruan from Beijing Genomics Institute wrote the  RAZF  library.  Various

        people in the 1000 Genomes Project contributed to the SAM format speci-
        fication.
 
        people in the 1000 Genomes Project contributed to the SAM format speci-
        fication.
 


@@ -404,4 +535,4 @@ SEE ALSO
 
 
 
 
 
 

-samtools-0.1.8                   11 July 2010                      samtools(1)
+samtools-0.1.9                  27 October 2010                    samtools(1)