+------------------------------------------------------------------------
+r782 | lh3lh3 | 2010-10-27 19:58:54 -0400 (Wed, 27 Oct 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam_plcmd.c
+
+fixed a silly bug in pileup
+
+------------------------------------------------------------------------
+r781 | lh3lh3 | 2010-10-27 14:39:48 -0400 (Wed, 27 Oct 2010) | 5 lines
+Changed paths:
+ M /trunk/samtools/ChangeLog
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bam_sort.c
+ M /trunk/samtools/bamtk.c
+ M /trunk/samtools/samtools.1
+
+ * samtools-0.1.8-22 (r781)
+ * made BAQ the default behavior of mpileup
+ * updated manual
+ * in merge, force to exit given inconsistent header when "-R" is not in use.
+
+------------------------------------------------------------------------
+r780 | lh3lh3 | 2010-10-27 11:01:11 -0400 (Wed, 27 Oct 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam.h
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-21 (r780)
+ * minor speedup to pileup
+
+------------------------------------------------------------------------
+r779 | lh3lh3 | 2010-10-27 09:58:56 -0400 (Wed, 27 Oct 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam_pileup.c
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/examples/toy.sam
+
+improve pileup a little bit
+
+------------------------------------------------------------------------
+r778 | lh3lh3 | 2010-10-27 00:14:43 -0400 (Wed, 27 Oct 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam.h
+ M /trunk/samtools/bam_pileup.c
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bam_tview.c
+ M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-20 (r778)
+ * speed up pileup, although I do not know how much is the improvement
+
+------------------------------------------------------------------------
+r777 | lh3lh3 | 2010-10-26 17:26:04 -0400 (Tue, 26 Oct 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam_maqcns.c
+ M /trunk/samtools/bam_maqcns.h
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bamtk.c
+ M /trunk/samtools/examples/Makefile
+
+ * samtools-0.1.8-19 (r777)
+ * integrate mpileup features to pileup: min_baseQ, capQ, prob_realn, paired-only and biased prior
+
+------------------------------------------------------------------------
+r776 | lh3lh3 | 2010-10-26 15:27:46 -0400 (Tue, 26 Oct 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam_md.c
+
+remove local realignment (probabilistic realignment is still there)
+
+------------------------------------------------------------------------
+r774 | jmarshall | 2010-10-21 06:52:38 -0400 (Thu, 21 Oct 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/sam_view.c
+
+Add the relevant filename or region to error messages, and cause a failure
+exit status where appropriate. Based on a patch provided by Marcel Martin.
+
+------------------------------------------------------------------------
+r773 | lh3lh3 | 2010-10-19 19:44:31 -0400 (Tue, 19 Oct 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/examples/toy.sam
+ M /trunk/samtools/kaln.c
+
+ * Minor code changes. No real effect.
+ * change quality to 30 in toy.sam
+
+------------------------------------------------------------------------
+r772 | lh3lh3 | 2010-10-18 23:40:13 -0400 (Mon, 18 Oct 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/examples/toy.fa
+ M /trunk/samtools/examples/toy.sam
+
+added another toy example
+
+------------------------------------------------------------------------
+r771 | lh3lh3 | 2010-10-13 23:32:12 -0400 (Wed, 13 Oct 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/call1.c
+ M /trunk/samtools/bcftools/ld.c
+ M /trunk/samtools/bcftools/vcfutils.pl
+
+improve the LD statistics
+
+------------------------------------------------------------------------
+r770 | lh3lh3 | 2010-10-12 23:49:26 -0400 (Tue, 12 Oct 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bcftools/call1.c
+ M /trunk/samtools/bcftools/vcfutils.pl
+
+ * a minor fix to the -L option
+ * add ldstats to vcfutils.pl
+
+------------------------------------------------------------------------
+r769 | lh3lh3 | 2010-10-12 15:51:57 -0400 (Tue, 12 Oct 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/bcf.c
+
+a minor change
+
+------------------------------------------------------------------------
+r768 | lh3lh3 | 2010-10-12 15:49:06 -0400 (Tue, 12 Oct 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/Makefile
+ A /trunk/samtools/bcftools/ld.c
+
+forget to add the key file
+
+------------------------------------------------------------------------
+r767 | lh3lh3 | 2010-10-12 15:48:46 -0400 (Tue, 12 Oct 2010) | 4 lines
+Changed paths:
+ M /trunk/samtools/bcftools/Makefile
+ M /trunk/samtools/bcftools/bcf.c
+ M /trunk/samtools/bcftools/bcf.h
+ M /trunk/samtools/bcftools/call1.c
+ M /trunk/samtools/bcftools/prob1.c
+ M /trunk/samtools/bcftools/vcfutils.pl
+
+ * vcfutils.pl: fixed a typo in help message
+ * added APIs: bcf_append_info() and bcf_cpy()
+ * calculate adjacent LD
+
+------------------------------------------------------------------------
+r766 | lh3lh3 | 2010-10-11 11:06:40 -0400 (Mon, 11 Oct 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/vcfutils.pl
+
+added filter for samtools/bcftools genetated VCFs
+
+------------------------------------------------------------------------
+r765 | lh3lh3 | 2010-10-05 14:05:18 -0400 (Tue, 05 Oct 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bcftools/vcfutils.pl
+ M /trunk/samtools/kaln.c
+
+ * removed a comment line in kaln.c
+ * vcfutils.pl fillac works when GT is not the first field
+
+------------------------------------------------------------------------
+r764 | petulda | 2010-10-05 08:59:36 -0400 (Tue, 05 Oct 2010) | 1 line
+Changed paths:
+ A /trunk/samtools/bcftools/bcf-fix.pl
+
+Convert VCF output of "bcftools view -bgcv" to a valid VCF file
+------------------------------------------------------------------------
+r763 | lh3lh3 | 2010-10-02 22:51:03 -0400 (Sat, 02 Oct 2010) | 4 lines
+Changed paths:
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bamtk.c
+ A /trunk/samtools/bcftools/bcftools.1
+ M /trunk/samtools/bcftools/call1.c
+ M /trunk/samtools/samtools.1
+
+ * samtools-0.1.8-18 (r763)
+ * added bcftools manual page
+ * minor fix to mpileup and view command lines
+
+------------------------------------------------------------------------
+r762 | lh3lh3 | 2010-10-02 21:46:25 -0400 (Sat, 02 Oct 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bcftools/bcf.c
+ M /trunk/samtools/bcftools/call1.c
+ M /trunk/samtools/bcftools/vcfutils.pl
+
+ * vcfutils.pl qstats: calculate marginal ts/tv
+ * allow to call genotypes at variant sites
+
+------------------------------------------------------------------------
+r761 | lh3lh3 | 2010-10-01 00:29:55 -0400 (Fri, 01 Oct 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/kaln.c
+ M /trunk/samtools/misc/HmmGlocal.java
+
+I am changing the gap open probability back to 0.001. It seems that
+being conservative here is a good thing...
+
+------------------------------------------------------------------------
+r760 | lh3lh3 | 2010-10-01 00:11:27 -0400 (Fri, 01 Oct 2010) | 5 lines
+Changed paths:
+ M /trunk/samtools/bamtk.c
+ M /trunk/samtools/kaln.c
+ A /trunk/samtools/misc/HmmGlocal.java
+
+ * samtools-0.1.8-17 (r760)
+ * the default gap open penalty is too small (a typo)
+ * added comments on hmm_realn
+ * Java implementation
+
+------------------------------------------------------------------------
+r759 | lh3lh3 | 2010-09-30 10:12:54 -0400 (Thu, 30 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bamtk.c
+
+mark samtools-0.1.8-16 (r759)
+
+------------------------------------------------------------------------
+r758 | lh3lh3 | 2010-09-30 10:12:02 -0400 (Thu, 30 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/kaln.c
+
+round to the nearest integer
+
+------------------------------------------------------------------------
+r757 | lh3lh3 | 2010-09-28 17:16:43 -0400 (Tue, 28 Sep 2010) | 4 lines
+Changed paths:
+ M /trunk/samtools/kaln.c
+
+I was trying to accelerate ka_prob_glocal() as this will be the
+bottleneck. After an hour, the only gain is to change division to
+multiplication. OK. I will stop.
+
+------------------------------------------------------------------------
+r756 | lh3lh3 | 2010-09-28 16:57:49 -0400 (Tue, 28 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/kaln.c
+
+this is interesting. multiplication is much faster than division, at least on my Mac
+
+------------------------------------------------------------------------
+r755 | lh3lh3 | 2010-09-28 16:19:13 -0400 (Tue, 28 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/kaln.c
+
+minor changes
+
+------------------------------------------------------------------------
+r754 | lh3lh3 | 2010-09-28 15:44:16 -0400 (Tue, 28 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam_md.c
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/kaln.c
+
+prob_realn() seems working!
+
+------------------------------------------------------------------------
+r753 | lh3lh3 | 2010-09-28 12:48:23 -0400 (Tue, 28 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/kaln.c
+
+minor
+
+------------------------------------------------------------------------
+r752 | lh3lh3 | 2010-09-28 12:47:41 -0400 (Tue, 28 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/kaln.c
+ M /trunk/samtools/kaln.h
+
+Convert phredQ to probabilities
+
+------------------------------------------------------------------------
+r751 | lh3lh3 | 2010-09-28 12:32:08 -0400 (Tue, 28 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/kaln.c
+ M /trunk/samtools/kaln.h
+
+Implement the glocal HMM; discard the extention HMM
+
+------------------------------------------------------------------------
+r750 | lh3lh3 | 2010-09-28 00:06:11 -0400 (Tue, 28 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/kaln.c
+
+improve numerical stability
+
+------------------------------------------------------------------------
+r749 | lh3lh3 | 2010-09-27 23:27:54 -0400 (Mon, 27 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/kaln.c
+
+more comments
+
+------------------------------------------------------------------------
+r748 | lh3lh3 | 2010-09-27 23:17:16 -0400 (Mon, 27 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/kaln.c
+
+fixed a bug in banded DP
+
+------------------------------------------------------------------------
+r747 | lh3lh3 | 2010-09-27 23:05:12 -0400 (Mon, 27 Sep 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/kaln.c
+
+ * fixed that weird issue.
+ * the banded version is NOT working
+
+------------------------------------------------------------------------
+r746 | lh3lh3 | 2010-09-27 22:57:05 -0400 (Mon, 27 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/kaln.c
+
+More comments. This version seems working, but something is a little weird...
+
+------------------------------------------------------------------------
+r745 | lh3lh3 | 2010-09-27 17:21:40 -0400 (Mon, 27 Sep 2010) | 6 lines
+Changed paths:
+ M /trunk/samtools/kaln.c
+
+A little code cleanup. Now the forward and backback algorithms give
+nearly identical P(x), which means both are close to the correct
+forms. However, I have only tested on toy examples. Minor errors in
+the implementation may not be obvious.
+
+
+------------------------------------------------------------------------
+r744 | lh3lh3 | 2010-09-27 16:55:15 -0400 (Mon, 27 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bam_sort.c
+ M /trunk/samtools/kaln.c
+ M /trunk/samtools/kaln.h
+
+...
+
+------------------------------------------------------------------------
+r743 | jmarshall | 2010-09-27 08:19:06 -0400 (Mon, 27 Sep 2010) | 6 lines
+Changed paths:
+ M /trunk/samtools/bam_sort.c
+
+Abort if merge -h's INH.SAM cannot be opened, just as we abort
+if any of the IN#.BAM input files cannot be opened.
+
+Also propagate any error indication returned by bam_merge_core()
+to samtools merge's exit status.
+
+------------------------------------------------------------------------
+r741 | jmarshall | 2010-09-24 11:08:24 -0400 (Fri, 24 Sep 2010) | 5 lines
+Changed paths:
+ M /trunk/samtools/bam_index.c
+
+Use bam_validate1() to detect garbage records in the event of a corrupt
+BAI index file that causes a bam_seek() to an invalid position. At most
+one record (namely, the bam_iter_read terminator) is tested per bam_fetch()
+call, so the cost is insignificant in the normal case.
+
+------------------------------------------------------------------------
+r740 | jmarshall | 2010-09-24 11:00:19 -0400 (Fri, 24 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam.c
+ M /trunk/samtools/bam.h
+
+Add bam_validate1().
+
+------------------------------------------------------------------------
+r739 | lh3lh3 | 2010-09-22 12:07:50 -0400 (Wed, 22 Sep 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam_md.c
+ M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-15 (r379)
+ * allow to change capQ parameter in calmd
+
+------------------------------------------------------------------------
+r738 | jmarshall | 2010-09-22 11:15:33 -0400 (Wed, 22 Sep 2010) | 13 lines
+Changed paths:
+ M /trunk/samtools/bam_index.c
+ M /trunk/samtools/sam_view.c
+
+When bam_read1() returns an error (return value <= -2), propagate that error
+to bam_iter_read()'s own return value. Similarly, also propagate it up to
+bam_fetch()'s return value. Previously bam_fetch() always returned 0, and
+callers ignored its return value anyway. With this change, 0 continues to
+indicate success, while <= -2 (which can be written as < 0, as -1 is never
+returned) indicates corrupted input.
+
+bam_iter_read() ought also to propagate errors returned by bam_seek().
+
+main_samview() can now print an error message and fail when bam_fetch()
+detects that a .bai index file is corrupted or otherwise does not correspond
+to the .bam file it is being used with.
+
+------------------------------------------------------------------------
+r737 | jmarshall | 2010-09-22 10:47:42 -0400 (Wed, 22 Sep 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam_index.c
+
+0 is a successful return value from bam_read1(). (In practice, it never
+returns 0 anyway; but all the other callers treat 0 as successful.)
+
+------------------------------------------------------------------------
+r736 | lh3lh3 | 2010-09-20 17:43:08 -0400 (Mon, 20 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam.h
+ M /trunk/samtools/bam_index.c
+ M /trunk/samtools/bam_sort.c
+
+ * merge files region-by-region. work on small examples but more tests are needed.
+
+------------------------------------------------------------------------
+r735 | lh3lh3 | 2010-09-20 16:56:24 -0400 (Mon, 20 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/vcfutils.pl
+
+improve qstats by checking the alleles as well
+
+------------------------------------------------------------------------
+r734 | lh3lh3 | 2010-09-17 18:12:13 -0400 (Fri, 17 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/vcfutils.pl
+
+convert UCSC SNP SQL dump to VCF
+
+------------------------------------------------------------------------
+r733 | lh3lh3 | 2010-09-17 13:02:11 -0400 (Fri, 17 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/vcfutils.pl
+
+hapmap2vcf convertor
+
+------------------------------------------------------------------------
+r732 | lh3lh3 | 2010-09-17 10:11:37 -0400 (Fri, 17 Sep 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bcftools/Makefile
+ M /trunk/samtools/bcftools/bcf.c
+ M /trunk/samtools/bcftools/bcf.h
+ M /trunk/samtools/bcftools/vcf.c
+
+ * added comments
+ * VCF->BCF is not possible without knowing the sequence dictionary before hand...
+
+------------------------------------------------------------------------
+r731 | lh3lh3 | 2010-09-17 09:15:53 -0400 (Fri, 17 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam2bcf.c
+ M /trunk/samtools/bcftools/bcf.c
+ M /trunk/samtools/bcftools/bcf.h
+ M /trunk/samtools/bcftools/bcfutils.c
+ M /trunk/samtools/bcftools/call1.c
+ M /trunk/samtools/bcftools/vcf.c
+
+ * put n_smpl to "bcf1_t" to simplify API a little
+
+------------------------------------------------------------------------
+r730 | lh3lh3 | 2010-09-16 21:36:01 -0400 (Thu, 16 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/bcf.h
+ M /trunk/samtools/bcftools/call1.c
+ M /trunk/samtools/bcftools/index.c
+
+fixed a bug in indexing
+
+------------------------------------------------------------------------
+r729 | lh3lh3 | 2010-09-16 16:54:48 -0400 (Thu, 16 Sep 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam.c
+ M /trunk/samtools/bam_md.c
+ M /trunk/samtools/bam_pileup.c
+
+ * fixed a bug in capQ
+ * valgrind identifies a use of uninitialised value, but I have not fixed it.
+
+------------------------------------------------------------------------
+r728 | lh3lh3 | 2010-09-16 15:03:59 -0400 (Thu, 16 Sep 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bgzip.c
+ M /trunk/samtools/razip.c
+
+ * fixed a bug in razip: -c will delete the input file
+ * copy tabix/bgzip to here
+
+------------------------------------------------------------------------
+r727 | lh3lh3 | 2010-09-16 13:45:49 -0400 (Thu, 16 Sep 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam_md.c
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-14 (r727)
+ * allow to change the capQ parameter at the command line
+
+------------------------------------------------------------------------
+r726 | lh3lh3 | 2010-09-16 13:38:43 -0400 (Thu, 16 Sep 2010) | 4 lines
+Changed paths:
+ M /trunk/samtools/bam_md.c
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bcftools/vcfutils.pl
+ M /trunk/samtools/misc/samtools.pl
+
+ * added varFilter to vcfutils.pl
+ * reimplement realn(). now it performs a local alignment
+ * added cap_mapQ() to cap mapping quality when there are many substitutions
+
+------------------------------------------------------------------------
+r724 | lh3lh3 | 2010-09-15 00:18:31 -0400 (Wed, 15 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/Makefile
+ A /trunk/samtools/bcftools/bcf2qcall.c
+ M /trunk/samtools/bcftools/call1.c
+
+ * convert BCF to QCALL input
+
+------------------------------------------------------------------------
+r723 | lh3lh3 | 2010-09-14 22:41:50 -0400 (Tue, 14 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam_md.c
+
+dynamic band width in realignment
+
+------------------------------------------------------------------------
+r722 | lh3lh3 | 2010-09-14 22:05:32 -0400 (Tue, 14 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam_md.c
+ M /trunk/samtools/bam_plcmd.c
+
+fixed a bug in realignment
+
+------------------------------------------------------------------------
+r721 | lh3lh3 | 2010-09-14 20:54:09 -0400 (Tue, 14 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/prob1.c
+
+fixed a minor issue
+
+------------------------------------------------------------------------
+r720 | lh3lh3 | 2010-09-14 19:25:10 -0400 (Tue, 14 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/Makefile
+ M /trunk/samtools/bam_maqcns.c
+ M /trunk/samtools/bam_md.c
+
+fixed a bug in realignment
+
+------------------------------------------------------------------------
+r719 | lh3lh3 | 2010-09-14 19:18:24 -0400 (Tue, 14 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam_plcmd.c
+
+minor changes. It is BUGGY now!
+
+------------------------------------------------------------------------
+r718 | lh3lh3 | 2010-09-14 16:32:33 -0400 (Tue, 14 Sep 2010) | 4 lines
+Changed paths:
+ M /trunk/samtools/bam_md.c
+ M /trunk/samtools/bam_pileup.c
+ M /trunk/samtools/kaln.c
+ M /trunk/samtools/kaln.h
+
+ * aggressive gapped aligner is implemented in calmd.
+ * distinguish gap_open and gap_end_open in banded alignment
+ * make tview accepts alignment with heading and tailing D
+
+------------------------------------------------------------------------
+r717 | jmarshall | 2010-09-14 09:04:28 -0400 (Tue, 14 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools
+
+Add svn:ignore properties for generated files that don't appear in "make all".
+
+------------------------------------------------------------------------
+r716 | jmarshall | 2010-09-13 08:37:53 -0400 (Mon, 13 Sep 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools
+ M /trunk/samtools/bcftools
+ M /trunk/samtools/misc
+
+Add svn:ignore properties listing the generated files.
+(Except for *.o, which we'll assume is in global-ignores.)
+
+------------------------------------------------------------------------
+r715 | lh3lh3 | 2010-09-08 12:53:55 -0400 (Wed, 08 Sep 2010) | 5 lines
+Changed paths:
+ M /trunk/samtools/bamtk.c
+ M /trunk/samtools/bcftools/call1.c
+ M /trunk/samtools/bcftools/prob1.c
+ M /trunk/samtools/sample.c
+ M /trunk/samtools/sample.h
+
+ * samtools-0.1.8-13 (r715)
+ * fixed a bug in identifying SM across files
+ * bcftools: estimate heterozygosity
+ * bcftools: allow to skip sites without reference bases
+
+------------------------------------------------------------------------
+r713 | lh3lh3 | 2010-09-03 17:19:12 -0400 (Fri, 03 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/Makefile
+ M /trunk/samtools/bcftools/call1.c
+ M /trunk/samtools/bcftools/prob1.c
+ M /trunk/samtools/bcftools/prob1.h
+
+quite a lot changes to the contrast caller, but I still feel something is missing...
+
+------------------------------------------------------------------------
+r711 | lh3lh3 | 2010-09-03 00:30:48 -0400 (Fri, 03 Sep 2010) | 4 lines
+Changed paths:
+ M /trunk/samtools/bcftools/Makefile
+ M /trunk/samtools/bcftools/call1.c
+ M /trunk/samtools/bcftools/prob1.c
+ M /trunk/samtools/bcftools/vcfutils.pl
+
+ * changed 3.434 to 4.343 (typo!)
+ * fixed a bug in the contrast caller
+ * calculate heterozygosity
+
+------------------------------------------------------------------------
+r710 | lh3lh3 | 2010-09-01 23:24:47 -0400 (Wed, 01 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/bcf.h
+ M /trunk/samtools/bcftools/bcfutils.c
+ M /trunk/samtools/bcftools/call1.c
+
+SNP calling from the GL field
+
+------------------------------------------------------------------------
+r709 | lh3lh3 | 2010-09-01 18:52:30 -0400 (Wed, 01 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/vcf.c
+
+fixed another problem
+
+------------------------------------------------------------------------
+r708 | lh3lh3 | 2010-09-01 18:31:17 -0400 (Wed, 01 Sep 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bcftools/bcf.c
+ M /trunk/samtools/bcftools/vcf.c
+
+ * fixed bugs in parsing VCF
+ * parser now works with GT/GQ/DP/PL/GL
+
+------------------------------------------------------------------------
+r707 | lh3lh3 | 2010-09-01 15:28:29 -0400 (Wed, 01 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/Makefile
+ M /trunk/samtools/bcftools/prob1.c
+
+Do not compile _BCF_QUAD by default
+
+------------------------------------------------------------------------
+r706 | lh3lh3 | 2010-09-01 15:21:41 -0400 (Wed, 01 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/bcf.c
+ M /trunk/samtools/bcftools/bcf.h
+ M /trunk/samtools/bcftools/bcfutils.c
+ M /trunk/samtools/bcftools/call1.c
+
+Write the correct ALT and PL in the SNP calling mode.
+
+------------------------------------------------------------------------
+r705 | lh3lh3 | 2010-09-01 12:50:33 -0400 (Wed, 01 Sep 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/vcfutils.pl
+
+more commands for my own uses
+
+------------------------------------------------------------------------
+r704 | lh3lh3 | 2010-09-01 09:26:10 -0400 (Wed, 01 Sep 2010) | 2 lines
+Changed paths:
+ A /trunk/samtools/bcftools/vcfutils.pl
+
+Utilities for processing VCF
+
+------------------------------------------------------------------------
+r703 | lh3lh3 | 2010-08-31 16:44:57 -0400 (Tue, 31 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/Makefile
+ M /trunk/samtools/bcftools/call1.c
+ M /trunk/samtools/bcftools/prob1.c
+ M /trunk/samtools/bcftools/prob1.h
+
+preliminary contrast variant caller
+
+------------------------------------------------------------------------
+r702 | lh3lh3 | 2010-08-31 12:28:39 -0400 (Tue, 31 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/call1.c
+ M /trunk/samtools/bcftools/prob1.c
+ M /trunk/samtools/bcftools/prob1.h
+
+z' and z'' can be calculated
+
+------------------------------------------------------------------------
+r701 | lh3lh3 | 2010-08-31 10:20:57 -0400 (Tue, 31 Aug 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bcftools/Makefile
+ A /trunk/samtools/bcftools/call1.c (from /trunk/samtools/bcftools/vcfout.c:699)
+ M /trunk/samtools/bcftools/prob1.c
+ D /trunk/samtools/bcftools/vcfout.c
+
+ * rename vcfout.c as call1.c
+ * prepare to add two-sample comparison
+
+------------------------------------------------------------------------
+r699 | lh3lh3 | 2010-08-24 15:28:16 -0400 (Tue, 24 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/vcfout.c
+
+fixed a bug in calculating the t statistics
+
+------------------------------------------------------------------------
+r698 | lh3lh3 | 2010-08-24 14:05:50 -0400 (Tue, 24 Aug 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam2bcf.c
+ M /trunk/samtools/bam2bcf.h
+ M /trunk/samtools/bamtk.c
+ M /trunk/samtools/bcftools/kfunc.c
+ M /trunk/samtools/bcftools/vcfout.c
+
+ * samtools-0.1.8-13 (r698)
+ * perform one-tailed t-test for baseQ, mapQ and endDist
+
+------------------------------------------------------------------------
+r697 | lh3lh3 | 2010-08-24 12:30:13 -0400 (Tue, 24 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/kfunc.c
+
+added regularized incomplete beta function
+
+------------------------------------------------------------------------
+r695 | lh3lh3 | 2010-08-23 17:36:17 -0400 (Mon, 23 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam_maqcns.c
+ M /trunk/samtools/bam_plcmd.c
+
+change the default correlation coefficient
+
+------------------------------------------------------------------------
+r694 | lh3lh3 | 2010-08-23 14:46:52 -0400 (Mon, 23 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/bcf.c
+ M /trunk/samtools/bcftools/vcfout.c
+
+print QUAL as floating numbers
+
+------------------------------------------------------------------------
+r693 | lh3lh3 | 2010-08-23 14:06:07 -0400 (Mon, 23 Aug 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/Makefile
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bamtk.c
+ M /trunk/samtools/examples/Makefile
+ A /trunk/samtools/sample.c
+ A /trunk/samtools/sample.h
+
+ * samtools-0.1.8-12 (r692)
+ * group data by samples in "mpileup -g"
+
+------------------------------------------------------------------------
+r692 | lh3lh3 | 2010-08-23 10:58:53 -0400 (Mon, 23 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/Makefile
+ D /trunk/samtools/bam_mcns.c
+ D /trunk/samtools/bam_mcns.h
+ M /trunk/samtools/bam_plcmd.c
+
+remove VCF output in mpileup
+
+------------------------------------------------------------------------
+r691 | lh3lh3 | 2010-08-23 10:48:20 -0400 (Mon, 23 Aug 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam2bcf.c
+ M /trunk/samtools/bam2bcf.h
+
+ * use the revised MAQ error model for mpileup
+ * prepare to remove the independent model from mpileup
+
+------------------------------------------------------------------------
+r690 | lh3lh3 | 2010-08-20 15:46:40 -0400 (Fri, 20 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/Makefile
+ M /trunk/samtools/bam_maqcns.c
+ M /trunk/samtools/bam_maqcns.h
+ M /trunk/samtools/bam_plcmd.c
+ A /trunk/samtools/errmod.c
+ A /trunk/samtools/errmod.h
+ M /trunk/samtools/ksort.h
+
+added revised MAQ error model
+
+------------------------------------------------------------------------
+r689 | lh3lh3 | 2010-08-18 09:55:20 -0400 (Wed, 18 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/prob1.c
+ M /trunk/samtools/bcftools/prob1.h
+ M /trunk/samtools/bcftools/vcfout.c
+
+allow to read the prior from the error output. EM iteration is working.
+
+------------------------------------------------------------------------
+r688 | lh3lh3 | 2010-08-17 12:12:20 -0400 (Tue, 17 Aug 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bcftools/main.c
+ M /trunk/samtools/bcftools/vcf.c
+
+ * write a little more VCF header
+ * concatenate BCFs
+
+------------------------------------------------------------------------
+r687 | lh3lh3 | 2010-08-16 20:53:16 -0400 (Mon, 16 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/bcf.c
+ M /trunk/samtools/bcftools/bcf.h
+ M /trunk/samtools/bcftools/bcf.tex
+
+use float for QUAL
+
+------------------------------------------------------------------------
+r686 | lh3lh3 | 2010-08-14 00:11:13 -0400 (Sat, 14 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/bcf.c
+ M /trunk/samtools/bcftools/prob1.c
+
+faster for large sample size (in principle)
+
+------------------------------------------------------------------------
+r685 | lh3lh3 | 2010-08-13 23:28:31 -0400 (Fri, 13 Aug 2010) | 4 lines
+Changed paths:
+ M /trunk/samtools/bcftools/prob1.c
+
+ * a numerically stable method to calculate z_{jk}
+ * currently slower than the old method but will be important for large sample size
+ * in principle, we can speed up for large n, but have not tried
+
+------------------------------------------------------------------------
+r684 | lh3lh3 | 2010-08-11 21:58:31 -0400 (Wed, 11 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/vcfout.c
+
+fixed an issue in parsing integer
+
+------------------------------------------------------------------------
+r683 | lh3lh3 | 2010-08-09 13:05:07 -0400 (Mon, 09 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/bcf.c
+
+do not print refname if file is converted from VCF
+
+------------------------------------------------------------------------
+r682 | lh3lh3 | 2010-08-09 12:59:47 -0400 (Mon, 09 Aug 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bcftools/vcf.c
+
+ * parse PL
+ * fixed a bug in parsing VCF
+
+------------------------------------------------------------------------
+r681 | lh3lh3 | 2010-08-09 12:49:23 -0400 (Mon, 09 Aug 2010) | 4 lines
+Changed paths:
+ M /trunk/samtools/bcftools/bcf.c
+ M /trunk/samtools/bcftools/bcf.h
+ M /trunk/samtools/bcftools/bcfutils.c
+ M /trunk/samtools/bcftools/main.c
+ M /trunk/samtools/bcftools/vcf.c
+ M /trunk/samtools/bcftools/vcfout.c
+ M /trunk/samtools/bgzf.c
+ M /trunk/samtools/kstring.c
+
+ * fixed a bug in kstrtok@kstring.c
+ * preliminary VCF parser (not parse everything for now)
+ * improved view interface
+
+------------------------------------------------------------------------
+r680 | lh3lh3 | 2010-08-09 10:43:13 -0400 (Mon, 09 Aug 2010) | 4 lines
+Changed paths:
+ M /trunk/samtools/bcftools/bcf.c
+ M /trunk/samtools/bcftools/bcf.h
+ M /trunk/samtools/bcftools/vcfout.c
+ M /trunk/samtools/kstring.c
+ M /trunk/samtools/kstring.h
+
+ * improved kstring (added kstrtok)
+ * removed the limit on the format string length in bcftools
+ * use kstrtok to parse format which fixed a bug in the old code
+
+------------------------------------------------------------------------
+r679 | lh3lh3 | 2010-08-09 01:12:05 -0400 (Mon, 09 Aug 2010) | 2 lines
+Changed paths:
+ A /trunk/samtools/bcftools/README
+ M /trunk/samtools/bcftools/vcfout.c
+
+help messages
+
+------------------------------------------------------------------------
+r678 | lh3lh3 | 2010-08-09 00:01:52 -0400 (Mon, 09 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/vcfout.c
+
+perform single-tail test for ED4
+
+------------------------------------------------------------------------
+r677 | lh3lh3 | 2010-08-08 23:48:35 -0400 (Sun, 08 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/Makefile
+ M /trunk/samtools/bcftools/kfunc.c
+ M /trunk/samtools/bcftools/vcfout.c
+
+ * test depth, end distance and HWE
+
+------------------------------------------------------------------------
+r676 | lh3lh3 | 2010-08-08 02:04:15 -0400 (Sun, 08 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/kfunc.c
+
+reimplement incomplete gamma functions. no copy-paste
+
+------------------------------------------------------------------------
+r675 | lh3lh3 | 2010-08-06 22:42:54 -0400 (Fri, 06 Aug 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam2bcf.c
+ M /trunk/samtools/bam2bcf.h
+ M /trunk/samtools/bcftools/fet.c
+ M /trunk/samtools/bcftools/prob1.c
+ M /trunk/samtools/bcftools/prob1.h
+ M /trunk/samtools/bcftools/vcfout.c
+
+ * bcftools: add HWE (no testing for now)
+ * record end dist in a 2x2 table, not avg, std any more
+
+------------------------------------------------------------------------
+r674 | lh3lh3 | 2010-08-06 17:30:16 -0400 (Fri, 06 Aug 2010) | 3 lines
+Changed paths:
+ A /trunk/samtools/bcftools/kfunc.c
+
+ * Special functions: log(gamma()), erfc(), P(a,x) (incomplete gamma)
+ * Not using Numerical Recipe due to licensing issues
+
+------------------------------------------------------------------------
+r673 | lh3lh3 | 2010-08-05 23:46:53 -0400 (Thu, 05 Aug 2010) | 2 lines
+Changed paths:
+ A /trunk/samtools/bcftools/fet.c
+
+Fisher's exact test
+
+------------------------------------------------------------------------
+r672 | lh3lh3 | 2010-08-05 21:48:33 -0400 (Thu, 05 Aug 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam2bcf.c
+ M /trunk/samtools/bam2bcf.h
+ M /trunk/samtools/bamtk.c
+ M /trunk/samtools/examples/Makefile
+
+ * samtools-0.1.8-11 (r672)
+ * collect more stats for allele balance test in bcftools (not yet)
+
+------------------------------------------------------------------------
+r671 | lh3lh3 | 2010-08-05 16:17:58 -0400 (Thu, 05 Aug 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bcftools/bcf.c
+ M /trunk/samtools/bcftools/main.c
+
+ * the code base is stablized again.
+ * I will delay the vcf parser, which is quite complicated but with little value for now
+
+------------------------------------------------------------------------
+r670 | lh3lh3 | 2010-08-05 16:03:23 -0400 (Thu, 05 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/examples/Makefile
+
+minor
+
+------------------------------------------------------------------------
+r669 | lh3lh3 | 2010-08-05 16:03:08 -0400 (Thu, 05 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcftools/vcf.c
+
+unfinished vcf parser
+
+------------------------------------------------------------------------
+r668 | lh3lh3 | 2010-08-05 15:46:40 -0400 (Thu, 05 Aug 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bcftools/Makefile
+ M /trunk/samtools/bcftools/bcf.c
+ M /trunk/samtools/bcftools/bcf.h
+ M /trunk/samtools/bcftools/bcfutils.c
+ M /trunk/samtools/bcftools/index.c
+ M /trunk/samtools/bcftools/main.c
+ A /trunk/samtools/bcftools/vcf.c
+ M /trunk/samtools/bcftools/vcfout.c
+
+ * added prelimiary VCF parser (not finished)
+ * change struct a bit
+
+------------------------------------------------------------------------
+r667 | lh3lh3 | 2010-08-03 22:35:27 -0400 (Tue, 03 Aug 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam2bcf.c
+ M /trunk/samtools/bam2bcf.h
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bcftools/bcf.c
+
+ * allow to set min base q
+ * fixed a bug in mpileup -u
+
+------------------------------------------------------------------------
+r666 | lh3lh3 | 2010-08-03 22:08:44 -0400 (Tue, 03 Aug 2010) | 2 lines
+Changed paths:
+ A /trunk/samtools/bcftools/bcf.tex
+
+spec
+
+------------------------------------------------------------------------
+r665 | lh3lh3 | 2010-08-03 21:18:57 -0400 (Tue, 03 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/examples/Makefile
+
+added more examples
+
+------------------------------------------------------------------------
+r664 | lh3lh3 | 2010-08-03 21:13:00 -0400 (Tue, 03 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/Makefile
+ M /trunk/samtools/bam2bcf.c
+ M /trunk/samtools/bam2bcf.h
+ M /trunk/samtools/bcftools/Makefile
+
+fixed compilation error
+
+------------------------------------------------------------------------
+r662 | lh3lh3 | 2010-08-03 21:04:00 -0400 (Tue, 03 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/Makefile
+ D /trunk/samtools/bcf.c
+ D /trunk/samtools/bcf.h
+ A /trunk/samtools/bcftools
+ A /trunk/samtools/bcftools/Makefile
+ A /trunk/samtools/bcftools/bcf.c
+ A /trunk/samtools/bcftools/bcf.h
+ A /trunk/samtools/bcftools/bcfutils.c
+ A /trunk/samtools/bcftools/index.c
+ A /trunk/samtools/bcftools/main.c
+ A /trunk/samtools/bcftools/prob1.c
+ A /trunk/samtools/bcftools/prob1.h
+ A /trunk/samtools/bcftools/vcfout.c
+
+move bcftools to samtools
+
+------------------------------------------------------------------------
+r660 | lh3lh3 | 2010-08-03 15:58:32 -0400 (Tue, 03 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam2bcf.c
+
+fixed another minor bug
+
+------------------------------------------------------------------------
+r658 | lh3lh3 | 2010-08-03 15:06:45 -0400 (Tue, 03 Aug 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bamtk.c
+ M /trunk/samtools/bcf.c
+
+ * samtools-0.1.8-10 (r658)
+ * fixed a bug in bam2bcf when the reference is N
+
+------------------------------------------------------------------------
+r657 | lh3lh3 | 2010-08-03 14:50:23 -0400 (Tue, 03 Aug 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam2bcf.c
+ M /trunk/samtools/bam2bcf.h
+
+ * fixed a bug
+ * treat ambiguous ref base as the fifth base
+
+------------------------------------------------------------------------
+r654 | lh3lh3 | 2010-08-02 17:38:27 -0400 (Mon, 02 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/bcftools/bcf.c
+ M /trunk/samtools/bcf.c
+
+missing a column in VCF output...
+
+------------------------------------------------------------------------
+r653 | lh3lh3 | 2010-08-02 17:31:33 -0400 (Mon, 02 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcf.c
+
+fixed a memory leak
+
+------------------------------------------------------------------------
+r651 | lh3lh3 | 2010-08-02 17:27:31 -0400 (Mon, 02 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bcf.c
+
+fixed a bug in bcf reader
+
+------------------------------------------------------------------------
+r650 | lh3lh3 | 2010-08-02 17:00:41 -0400 (Mon, 02 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam2bcf.c
+
+fixed a bug
+
+------------------------------------------------------------------------
+r649 | lh3lh3 | 2010-08-02 16:49:35 -0400 (Mon, 02 Aug 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/Makefile
+ M /trunk/samtools/bam2bcf.c
+ M /trunk/samtools/bam2bcf.h
+ M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-9 (r649)
+ * lossless representation of PL in BCF output
+
+------------------------------------------------------------------------
+r648 | lh3lh3 | 2010-08-02 16:07:25 -0400 (Mon, 02 Aug 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/Makefile
+ A /trunk/samtools/bam2bcf.c
+ A /trunk/samtools/bam2bcf.h
+ M /trunk/samtools/bam_plcmd.c
+ A /trunk/samtools/bcf.c
+ A /trunk/samtools/bcf.h
+
+Generate binary VCF
+
+------------------------------------------------------------------------
+r644 | lh3lh3 | 2010-07-28 11:59:19 -0400 (Wed, 28 Jul 2010) | 5 lines
+Changed paths:
+ M /trunk/samtools/bam_mcns.c
+ M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-8 (r644)
+ * mpileup becomes a little stable again
+ * the method is slightly different, but is more theoretically correct
+ * snp calling is O(n^2) instead of O(n^3)
+
+------------------------------------------------------------------------
+r643 | lh3lh3 | 2010-07-28 11:54:15 -0400 (Wed, 28 Jul 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam_mcns.c
+
+ * fixed a STUPID bug, which cost me a lot of time.
+ * I am going to clean up mcns a little bit
+
+------------------------------------------------------------------------
+r642 | lh3lh3 | 2010-07-27 23:23:07 -0400 (Tue, 27 Jul 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam_mcns.c
+ M /trunk/samtools/bam_mcns.h
+ M /trunk/samtools/bam_plcmd.c
+
+supposedly this is THE correct implementation, but more testing is needed
+
+------------------------------------------------------------------------
+r641 | lh3lh3 | 2010-07-27 22:43:39 -0400 (Tue, 27 Jul 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam_mcns.c
+
+NOT ready yet. Going to make further changes...
+
+------------------------------------------------------------------------
+r639 | lh3lh3 | 2010-07-25 22:18:38 -0400 (Sun, 25 Jul 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam_mcns.c
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-7 (r639)
+ * fixed the reference allele assignment
+
+------------------------------------------------------------------------
+r638 | lh3lh3 | 2010-07-25 12:01:26 -0400 (Sun, 25 Jul 2010) | 5 lines
+Changed paths:
+ M /trunk/samtools/bam_mcns.c
+ M /trunk/samtools/bam_mcns.h
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-6 (r638)
+ * skip isnan/isinf in case of float underflow
+ * added the flat prior
+ * fixed an issue where there are no reads supporting the reference
+
+------------------------------------------------------------------------
+r637 | lh3lh3 | 2010-07-24 14:16:27 -0400 (Sat, 24 Jul 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam_plcmd.c
+
+minor changes
+
+------------------------------------------------------------------------
+r636 | lh3lh3 | 2010-07-24 14:07:27 -0400 (Sat, 24 Jul 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam_mcns.c
+ M /trunk/samtools/bam_mcns.h
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bamtk.c
+
+minor tweaks
+
+------------------------------------------------------------------------
+r635 | lh3lh3 | 2010-07-24 01:49:49 -0400 (Sat, 24 Jul 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam_mcns.c
+ M /trunk/samtools/bam_mcns.h
+ M /trunk/samtools/bam_plcmd.c
+
+posterior expectation FINALLY working. I am so tired...
+
+------------------------------------------------------------------------
+r633 | lh3lh3 | 2010-07-23 13:50:48 -0400 (Fri, 23 Jul 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam_plcmd.c
+
+another minor fix to mpileup
+
+------------------------------------------------------------------------
+r632 | lh3lh3 | 2010-07-23 13:43:31 -0400 (Fri, 23 Jul 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam_plcmd.c
+
+added the format column
+
+------------------------------------------------------------------------
+r631 | lh3lh3 | 2010-07-23 13:25:44 -0400 (Fri, 23 Jul 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam_mcns.c
+ M /trunk/samtools/bam_mcns.h
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bamtk.c
+
+added an alternative prior
+
+------------------------------------------------------------------------
+r628 | lh3lh3 | 2010-07-23 11:48:51 -0400 (Fri, 23 Jul 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/bam_mcns.c
+ M /trunk/samtools/bam_mcns.h
+ M /trunk/samtools/bam_plcmd.c
+
+calculate posterior allele frequency
+
+------------------------------------------------------------------------
+r627 | lh3lh3 | 2010-07-22 21:39:13 -0400 (Thu, 22 Jul 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam_mcns.c
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-3 (r627)
+ * multi-sample snp calling appears to work. More tests needed.
+
+------------------------------------------------------------------------
+r626 | lh3lh3 | 2010-07-22 16:37:56 -0400 (Thu, 22 Jul 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam_mcns.c
+ M /trunk/samtools/bam_mcns.h
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bam_tview.c
+
+ * preliminary multisample SNP caller.
+ * something looks not so right, but it largely works
+
+------------------------------------------------------------------------
+r617 | lh3lh3 | 2010-07-14 16:26:27 -0400 (Wed, 14 Jul 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam_mcns.c
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bamtk.c
+
+ * samtools-0.1.8-2 (r617)
+ * allele frequency calculation apparently works...
+
+------------------------------------------------------------------------
+r616 | lh3lh3 | 2010-07-14 13:33:51 -0400 (Wed, 14 Jul 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/Makefile
+ A /trunk/samtools/bam_mcns.c
+ A /trunk/samtools/bam_mcns.h
+ M /trunk/samtools/bam_plcmd.c
+
+ * added mutli-sample framework. It is not working, yet.
+ * improved the mpileup interface
+
+------------------------------------------------------------------------
+r615 | lh3lh3 | 2010-07-13 14:50:12 -0400 (Tue, 13 Jul 2010) | 3 lines
+Changed paths:
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bamtk.c
+ M /trunk/samtools/misc/Makefile
+
+ * samtools-0.1.8-1 (r615)
+ * allow to get mpileup at required sites
+
+------------------------------------------------------------------------
+r613 | lh3lh3 | 2010-07-11 22:40:56 -0400 (Sun, 11 Jul 2010) | 2 lines
+Changed paths:
+ M /trunk/samtools/ChangeLog
+ M /trunk/samtools/NEWS
+ M /trunk/samtools/bam_plcmd.c
+ M /trunk/samtools/bamtk.c
+ M /trunk/samtools/samtools.1
+
+Release samtools-0.1.8
+
------------------------------------------------------------------------
r612 | lh3lh3 | 2010-07-11 21:08:56 -0400 (Sun, 11 Jul 2010) | 2 lines
Changed paths:
$(KNETFILE_O) bam_sort.o sam_header.o bam_reheader.o
AOBJS= bam_tview.o bam_maqcns.o bam_plcmd.o sam_view.o \
bam_rmdup.o bam_rmdupse.o bam_mate.o bam_stat.o bam_color.o \
- bamtk.o kaln.o
+ bamtk.o kaln.o bam2bcf.o errmod.o sample.o
PROG= samtools
-INCLUDES=
-SUBDIRS= . misc
+INCLUDES= -I.
+SUBDIRS= . bcftools misc
LIBPATH=
LIBCURSES= -lcurses # -lXCurses
libbam.a:$(LOBJS)
$(AR) -cru $@ $(LOBJS)
-samtools:$(AOBJS) libbam.a
- $(CC) $(CFLAGS) -o $@ $(AOBJS) libbam.a -lm $(LIBPATH) $(LIBCURSES) -lz
+samtools:lib-recur $(AOBJS)
+ $(CC) $(CFLAGS) -o $@ $(AOBJS) libbam.a -lm $(LIBPATH) $(LIBCURSES) -lz -Lbcftools -lbcf
razip:razip.o razf.o $(KNETFILE_O)
$(CC) $(CFLAGS) -o $@ razf.o razip.o $(KNETFILE_O) -lz
sam.o:sam.h bam.h
bam_import.o:bam.h kseq.h khash.h razf.h
bam_pileup.o:bam.h razf.h ksort.h
-bam_plcmd.o:bam.h faidx.h bam_maqcns.h glf.h
+bam_plcmd.o:bam.h faidx.h bam_maqcns.h glf.h bcftools/bcf.h bam2bcf.h
bam_index.o:bam.h khash.h ksort.h razf.h bam_endian.h
bam_lpileup.o:bam.h ksort.h
bam_tview.o:bam.h faidx.h bam_maqcns.h
-bam_maqcns.o:bam.h ksort.h bam_maqcns.h
+bam_maqcns.o:bam.h ksort.h bam_maqcns.h kaln.h
bam_sort.o:bam.h ksort.h razf.h
bam_md.o:bam.h faidx.h
glf.o:glf.h
sam_header.o:sam_header.h khash.h
+bcf.o:bcftools/bcf.h
+bam2bcf.o:bam2bcf.h errmod.h bcftools/bcf.h
+errmod.o:errmod.h
faidx.o:faidx.h razf.h khash.h
faidx_main.o:faidx.h razf.h
+Beta Release 0.1.9 (27 October, 2010)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This release is featured as the first major improvement to the samtools'
+SNP caller. It comes with a revised MAQ error model, the support of
+multi-sample SNP calling and the computation of base alignment quality
+(BAQ).
+
+The revised MAQ error model is based on the original model. It solves an
+issue of miscalling SNPs in repetitive regions. Althought such SNPs can
+usually be filtered at a later step, they mess up unfiltered calls. This
+is a theoretical flaw in the original model. The revised MAQ model
+deprecates the orginal MAQ model and the simplified SOAPsnp model.
+
+Multi-sample SNP calling is separated in two steps. The first is done by
+samtools mpileup and the second by a new program, bcftools, which is
+included in the samtools source code tree. Multi-sample SNP calling also
+works for single sample and has the advantage of enabling more powerful
+filtration. It is likely to deprecate pileup in future once a proper
+indel calling method is implemented.
+
+BAQ is the Phred-scaled probability of a read base being wrongly
+aligned. Capping base quality by BAQ has been shown to be very effective
+in suppressing false SNPs caused by misalignments around indels or in
+low-complexity regions with acceptable compromise on computation
+time. This strategy is highly recommended and can be used with other SNP
+callers as well.
+
+In addition to the three major improvements, other notable changes are:
+
+ * Changes to the pileup format. A reference skip (the N CIGAR operator)
+ is shown as '<' or '>' depending on the strand. Tview is also changed
+ accordingly.
+
+ * Accelerated pileup. The plain pileup is about 50% faster.
+
+ * Regional merge. The merge command now accepts a new option to merge
+ files in a specified region.
+
+ * Fixed a bug in bgzip and razip which causes source files to be
+ deleted even if option -c is applied.
+
+ * In APIs, propogate errors to downstream callers and make samtools
+ return non-zero values once errors occur.
+
+(0.1.9: 27 October 2010, r783)
+
+
+
Beta Release 0.1.8 (11 July, 2010)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
kputc('\t', &str);
}
if (c->tid < 0) kputsn("*\t", 2, &str);
- else { kputs(header->target_name[c->tid], &str); kputc('\t', &str); }
+ else {
+ if (header) kputs(header->target_name[c->tid] , &str);
+ else kputw(c->tid, &str);
+ kputc('\t', &str);
+ }
kputw(c->pos + 1, &str); kputc('\t', &str); kputw(c->qual, &str); kputc('\t', &str);
if (c->n_cigar == 0) kputc('*', &str);
else {
kputc('\t', &str);
if (c->mtid < 0) kputsn("*\t", 2, &str);
else if (c->mtid == c->tid) kputsn("=\t", 2, &str);
- else { kputs(header->target_name[c->mtid], &str); kputc('\t', &str); }
+ else {
+ if (header) kputs(header->target_name[c->mtid], &str);
+ else kputw(c->mtid, &str);
+ kputc('\t', &str);
+ }
kputw(c->mpos + 1, &str); kputc('\t', &str); kputw(c->isize, &str); kputc('\t', &str);
if (c->l_qseq) {
for (i = 0; i < c->l_qseq; ++i) kputc(bam_nt16_rev_table[bam1_seqi(s, i)], &str);
free(s);
}
+int bam_validate1(const bam_header_t *header, const bam1_t *b)
+{
+ char *s;
+
+ if (b->core.tid < -1 || b->core.mtid < -1) return 0;
+ if (header && (b->core.tid >= header->n_targets || b->core.mtid >= header->n_targets)) return 0;
+
+ if (b->data_len < b->core.l_qname) return 0;
+ s = memchr(bam1_qname(b), '\0', b->core.l_qname);
+ if (s != &bam1_qname(b)[b->core.l_qname-1]) return 0;
+
+ // FIXME: Other fields could also be checked, especially the auxiliary data
+
+ return 1;
+}
+
// FIXME: we should also check the LB tag associated with each alignment
const char *bam_get_library(bam_header_t *h, const bam1_t *b)
{
/* The MIT License
- Copyright (c) 2008 Genome Research Ltd (GRL).
+ Copyright (c) 2008-2010 Genome Research Ltd (GRL).
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
@param b pointer to an alignment
@return pointer to quality string
*/
-#define bam1_qual(b) ((b)->data + (b)->core.n_cigar*4 + (b)->core.l_qname + ((b)->core.l_qseq + 1)/2)
+#define bam1_qual(b) ((b)->data + (b)->core.n_cigar*4 + (b)->core.l_qname + (((b)->core.l_qseq + 1)>>1))
/*! @function
@abstract Get a base on read
char *bam_format1_core(const bam_header_t *header, const bam1_t *b, int of);
+ /*!
+ @abstract Check whether a BAM record is plausibly valid
+ @param header associated header structure, or NULL if unavailable
+ @param b alignment to validate
+ @return 0 if the alignment is invalid; non-zero otherwise
+
+ @discussion Simple consistency check of some of the fields of the
+ alignment record. If the header is provided, several additional checks
+ are made. Not all fields are checked, so a non-zero result is not a
+ guarantee that the record is valid. However it is usually good enough
+ to detect when bam_seek() has been called with a virtual file offset
+ that is not the offset of an alignment record.
+ */
+ int bam_validate1(const bam_header_t *header, const bam1_t *b);
+
const char *bam_get_library(bam_header_t *header, const bam1_t *b);
bam1_t *b;
int32_t qpos;
int indel, level;
- uint32_t is_del:1, is_head:1, is_tail:1;
+ uint32_t is_del:1, is_head:1, is_tail:1, is_refskip:1;
} bam_pileup1_t;
typedef int (*bam_plp_auto_f)(void *data, bam1_t *b);
--- /dev/null
+#include <math.h>
+#include <stdint.h>
+#include "bam.h"
+#include "kstring.h"
+#include "bam2bcf.h"
+#include "errmod.h"
+#include "bcftools/bcf.h"
+
+extern void ks_introsort_uint32_t(size_t n, uint32_t a[]);
+
+#define CALL_ETA 0.03f
+#define CALL_MAX 256
+#define CALL_DEFTHETA 0.83f
+
+#define CAP_DIST 25
+
+struct __bcf_callaux_t {
+ int max_bases, capQ, min_baseQ;
+ uint16_t *bases;
+ errmod_t *e;
+};
+
+bcf_callaux_t *bcf_call_init(double theta, int min_baseQ)
+{
+ bcf_callaux_t *bca;
+ if (theta <= 0.) theta = CALL_DEFTHETA;
+ bca = calloc(1, sizeof(bcf_callaux_t));
+ bca->capQ = 60;
+ bca->min_baseQ = min_baseQ;
+ bca->e = errmod_init(1. - theta);
+ return bca;
+}
+
+void bcf_call_destroy(bcf_callaux_t *bca)
+{
+ if (bca == 0) return;
+ errmod_destroy(bca->e);
+ free(bca->bases); free(bca);
+}
+
+int bcf_call_glfgen(int _n, const bam_pileup1_t *pl, int ref_base /*4-bit*/, bcf_callaux_t *bca, bcf_callret1_t *r)
+{
+ int i, n, ref4;
+ memset(r, 0, sizeof(bcf_callret1_t));
+ ref4 = bam_nt16_nt4_table[ref_base];
+ if (_n == 0) return -1;
+
+ // enlarge the bases array if necessary
+ if (bca->max_bases < _n) {
+ bca->max_bases = _n;
+ kroundup32(bca->max_bases);
+ bca->bases = (uint16_t*)realloc(bca->bases, 2 * bca->max_bases);
+ }
+ // fill the bases array
+ memset(r, 0, sizeof(bcf_callret1_t));
+ for (i = n = 0; i < _n; ++i) {
+ const bam_pileup1_t *p = pl + i;
+ int q, b, mapQ, baseQ, is_diff, min_dist;
+ // set base
+ if (p->is_del || (p->b->core.flag&BAM_FUNMAP)) continue; // skip unmapped reads and deleted bases
+ baseQ = q = (int)bam1_qual(p->b)[p->qpos]; // base quality
+ if (q < bca->min_baseQ) continue;
+ mapQ = p->b->core.qual < bca->capQ? p->b->core.qual : bca->capQ;
+ if (q > mapQ) q = mapQ;
+ if (q > 63) q = 63;
+ if (q < 4) q = 4;
+ b = bam1_seqi(bam1_seq(p->b), p->qpos); // base
+ b = bam_nt16_nt4_table[b? b : ref_base]; // b is the 2-bit base
+ bca->bases[n++] = q<<5 | (int)bam1_strand(p->b)<<4 | b;
+ // collect annotations
+ r->qsum[b] += q;
+ is_diff = (ref4 < 4 && b == ref4)? 0 : 1;
+ ++r->anno[0<<2|is_diff<<1|bam1_strand(p->b)];
+ min_dist = p->b->core.l_qseq - 1 - p->qpos;
+ if (min_dist > p->qpos) min_dist = p->qpos;
+ if (min_dist > CAP_DIST) min_dist = CAP_DIST;
+ r->anno[1<<2|is_diff<<1|0] += baseQ;
+ r->anno[1<<2|is_diff<<1|1] += baseQ * baseQ;
+ r->anno[2<<2|is_diff<<1|0] += mapQ;
+ r->anno[2<<2|is_diff<<1|1] += mapQ * mapQ;
+ r->anno[3<<2|is_diff<<1|0] += min_dist;
+ r->anno[3<<2|is_diff<<1|1] += min_dist * min_dist;
+ }
+ r->depth = n;
+ // glfgen
+ errmod_cal(bca->e, n, 5, bca->bases, r->p);
+ return r->depth;
+}
+
+int bcf_call_combine(int n, const bcf_callret1_t *calls, int ref_base /*4-bit*/, bcf_call_t *call)
+{
+ int ref4, i, j, qsum[4];
+ int64_t tmp;
+ call->ori_ref = ref4 = bam_nt16_nt4_table[ref_base];
+ if (ref4 > 4) ref4 = 4;
+ // calculate qsum
+ memset(qsum, 0, 4 * sizeof(int));
+ for (i = 0; i < n; ++i)
+ for (j = 0; j < 4; ++j)
+ qsum[j] += calls[i].qsum[j];
+ for (j = 0; j < 4; ++j) qsum[j] = qsum[j] << 2 | j;
+ // find the top 2 alleles
+ for (i = 1; i < 4; ++i) // insertion sort
+ for (j = i; j > 0 && qsum[j] < qsum[j-1]; --j)
+ tmp = qsum[j], qsum[j] = qsum[j-1], qsum[j-1] = tmp;
+ // set the reference allele and alternative allele(s)
+ for (i = 0; i < 5; ++i) call->a[i] = -1;
+ call->unseen = -1;
+ call->a[0] = ref4;
+ for (i = 3, j = 1; i >= 0; --i) {
+ if ((qsum[i]&3) != ref4) {
+ if (qsum[i]>>2 != 0) call->a[j++] = qsum[i]&3;
+ else break;
+ }
+ }
+ if (((ref4 < 4 && j < 4) || (ref4 == 4 && j < 5)) && i >= 0)
+ call->unseen = j, call->a[j++] = qsum[i]&3;
+ call->n_alleles = j;
+ // set the PL array
+ if (call->n < n) {
+ call->n = n;
+ call->PL = realloc(call->PL, 15 * n);
+ }
+ {
+ int x, g[15], z;
+ double sum_min = 0.;
+ x = call->n_alleles * (call->n_alleles + 1) / 2;
+ // get the possible genotypes
+ for (i = z = 0; i < call->n_alleles; ++i)
+ for (j = i; j < call->n_alleles; ++j)
+ g[z++] = call->a[i] * 5 + call->a[j];
+ for (i = 0; i < n; ++i) {
+ uint8_t *PL = call->PL + x * i;
+ const bcf_callret1_t *r = calls + i;
+ float min = 1e37;
+ for (j = 0; j < x; ++j)
+ if (min > r->p[g[j]]) min = r->p[g[j]];
+ sum_min += min;
+ for (j = 0; j < x; ++j) {
+ int y;
+ y = (int)(r->p[g[j]] - min + .499);
+ if (y > 255) y = 255;
+ PL[j] = y;
+ }
+ }
+ call->shift = (int)(sum_min + .499);
+ }
+ // combine annotations
+ memset(call->anno, 0, 16 * sizeof(int));
+ for (i = call->depth = 0, tmp = 0; i < n; ++i) {
+ call->depth += calls[i].depth;
+ for (j = 0; j < 16; ++j) call->anno[j] += calls[i].anno[j];
+ }
+ return 0;
+}
+
+int bcf_call2bcf(int tid, int pos, bcf_call_t *bc, bcf1_t *b)
+{
+ kstring_t s;
+ int i;
+ b->n_smpl = bc->n;
+ b->tid = tid; b->pos = pos; b->qual = 0;
+ s.s = b->str; s.m = b->m_str; s.l = 0;
+ kputc('\0', &s);
+ kputc("ACGTN"[bc->ori_ref], &s); kputc('\0', &s);
+ for (i = 1; i < 5; ++i) {
+ if (bc->a[i] < 0) break;
+ if (i > 1) kputc(',', &s);
+ kputc(bc->unseen == i? 'X' : "ACGT"[bc->a[i]], &s);
+ }
+ kputc('\0', &s);
+ kputc('\0', &s);
+ // INFO
+ kputs("I16=", &s);
+ for (i = 0; i < 16; ++i) {
+ if (i) kputc(',', &s);
+ kputw(bc->anno[i], &s);
+ }
+ kputc('\0', &s);
+ // FMT
+ kputs("PL", &s); kputc('\0', &s);
+ b->m_str = s.m; b->str = s.s; b->l_str = s.l;
+ bcf_sync(b);
+ memcpy(b->gi[0].data, bc->PL, b->gi[0].len * bc->n);
+ return 0;
+}
--- /dev/null
+#ifndef BAM2BCF_H
+#define BAM2BCF_H
+
+#include <stdint.h>
+#include "bcftools/bcf.h"
+
+struct __bcf_callaux_t;
+typedef struct __bcf_callaux_t bcf_callaux_t;
+
+typedef struct {
+ int depth, qsum[4];
+ int anno[16];
+ float p[25];
+} bcf_callret1_t;
+
+typedef struct {
+ int a[5]; // alleles: ref, alt, alt2, alt3
+ int n, n_alleles, shift, ori_ref, unseen;
+ int anno[16], depth;
+ uint8_t *PL;
+} bcf_call_t;
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+ bcf_callaux_t *bcf_call_init(double theta, int min_baseQ);
+ void bcf_call_destroy(bcf_callaux_t *bca);
+ int bcf_call_glfgen(int _n, const bam_pileup1_t *pl, int ref_base /*4-bit*/, bcf_callaux_t *bca, bcf_callret1_t *r);
+ int bcf_call_combine(int n, const bcf_callret1_t *calls, int ref_base /*4-bit*/, bcf_call_t *call);
+ int bcf_call2bcf(int tid, int pos, bcf_call_t *bc, bcf1_t *b);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
int bam_iter_read(bamFile fp, bam_iter_t iter, bam1_t *b)
{
- if (iter->finished) return -1;
- if (iter->from_first) {
- int ret = bam_read1(fp, b);
- if (ret < 0) iter->finished = 1;
+ int ret;
+ if (iter && iter->finished) return -1;
+ if (iter == 0 || iter->from_first) {
+ ret = bam_read1(fp, b);
+ if (ret < 0 && iter) iter->finished = 1;
return ret;
}
if (iter->off == 0) return -1;
for (;;) {
- int ret;
if (iter->curr_off == 0 || iter->curr_off >= iter->off[iter->i].v) { // then jump to the next chunk
- if (iter->i == iter->n_off - 1) break; // no more chunks
+ if (iter->i == iter->n_off - 1) { ret = -1; break; } // no more chunks
if (iter->i >= 0) assert(iter->curr_off == iter->off[iter->i].v); // otherwise bug
if (iter->i < 0 || iter->off[iter->i].v != iter->off[iter->i+1].u) { // not adjacent chunks; then seek
bam_seek(fp, iter->off[iter->i+1].u, SEEK_SET);
}
++iter->i;
}
- if ((ret = bam_read1(fp, b)) > 0) {
+ if ((ret = bam_read1(fp, b)) >= 0) {
iter->curr_off = bam_tell(fp);
- if (b->core.tid != iter->tid || b->core.pos >= iter->end) break; // no need to proceed
+ if (b->core.tid != iter->tid || b->core.pos >= iter->end) { // no need to proceed
+ ret = bam_validate1(NULL, b)? -1 : -5; // determine whether end of region or error
+ break;
+ }
else if (is_overlap(iter->beg, iter->end, b)) return ret;
- } else break; // end of file
+ } else break; // end of file or error
}
iter->finished = 1;
- return -1;
+ return ret;
}
int bam_fetch(bamFile fp, const bam_index_t *idx, int tid, int beg, int end, void *data, bam_fetch_f func)
{
+ int ret;
bam_iter_t iter;
bam1_t *b;
b = bam_init1();
iter = bam_iter_query(idx, tid, beg, end);
- while (bam_iter_read(fp, iter, b) >= 0) func(b, data);
+ while ((ret = bam_iter_read(fp, iter, b)) >= 0) func(b, data);
bam_destroy1(b);
- return 0;
+ return (ret == -1)? 0 : ret;
}
#include "bam.h"
#include "bam_maqcns.h"
#include "ksort.h"
+#include "errmod.h"
#include "kaln.h"
KSORT_INIT_GENERIC(uint32_t)
typedef struct __bmc_aux_t {
int max;
uint32_t *info;
+ uint16_t *info16;
+ errmod_t *em;
} bmc_aux_t;
typedef struct {
float esum[4], fsum[4];
uint32_t c[4];
- uint32_t rms_mapQ;
} glf_call_aux_t;
char bam_nt16_nt4_table[] = { 4, 0, 1, 4, 2, 4, 4, 4, 3, 4, 4, 4, 4, 4, 4, 4 };
for (n1 = 0; n1 < 256; ++n1) {
for (n2 = 0; n2 < 256; ++n2) {
long double sum = 0.0;
- double lC = aa->is_soap? 0 : lgamma(n1+n2+1) - lgamma(n1+1) - lgamma(n2+1); // \binom{n1+n2}{n1}
+ double lC = aa->errmod == BAM_ERRMOD_SOAP? 0 : lgamma(n1+n2+1) - lgamma(n1+1) - lgamma(n2+1);
for (k = 1; k <= aa->n_hap - 1; ++k) {
double pk = 1.0 / k / sum_harmo;
double log1 = log((double)k/aa->n_hap);
long double sum_a[257], b[256], q_c[256], tmp[256], fk2[256];
double *lC;
+ if (aa->errmod == BAM_ERRMOD_MAQ2) return; // no need to do the following
// aa->lhet will be allocated and initialized
free(aa->fk); free(aa->coef);
aa->coef = 0;
aa->fk[n] = pow(aa->theta, n) * (1.0 - aa->eta) + aa->eta;
fk2[n] = aa->fk[n>>1]; // this is an approximation, assuming reads equally likely come from both strands
}
- if (aa->is_soap) return;
+ if (aa->errmod == BAM_ERRMOD_SOAP) return;
aa->coef = (double*)calloc(256*256*64, sizeof(double));
lC = (double*)calloc(256 * 256, sizeof(double));
for (n = 1; n != 256; ++n)
bm = (bam_maqcns_t*)calloc(1, sizeof(bam_maqcns_t));
bm->aux = (bmc_aux_t*)calloc(1, sizeof(bmc_aux_t));
bm->het_rate = 0.001;
- bm->theta = 0.85;
+ bm->theta = 0.83f;
bm->n_hap = 2;
bm->eta = 0.03;
bm->cap_mapQ = 60;
+ bm->min_baseQ = 13;
return bm;
}
void bam_maqcns_prepare(bam_maqcns_t *bm)
{
+ if (bm->errmod == BAM_ERRMOD_MAQ2) bm->aux->em = errmod_init(1. - bm->theta);
cal_coef(bm); cal_het(bm);
}
void bam_maqcns_destroy(bam_maqcns_t *bm)
{
if (bm == 0) return;
- free(bm->lhet); free(bm->fk); free(bm->coef); free(bm->aux->info);
+ free(bm->lhet); free(bm->fk); free(bm->coef); free(bm->aux->info); free(bm->aux->info16);
+ if (bm->aux->em) errmod_destroy(bm->aux->em);
free(bm->aux); free(bm);
}
glf1_t *bam_maqcns_glfgen(int _n, const bam_pileup1_t *pl, uint8_t ref_base, bam_maqcns_t *bm)
{
- glf_call_aux_t *b;
+ glf_call_aux_t *b = 0;
int i, j, k, w[8], c, n;
glf1_t *g = (glf1_t*)calloc(1, sizeof(glf1_t));
float p[16], min_p = 1e30;
bm->aux->max = _n;
kroundup32(bm->aux->max);
bm->aux->info = (uint32_t*)realloc(bm->aux->info, 4 * bm->aux->max);
+ bm->aux->info16 = (uint16_t*)realloc(bm->aux->info16, 2 * bm->aux->max);
}
- for (i = n = 0; i < _n; ++i) {
+ for (i = n = 0, rms = 0; i < _n; ++i) {
const bam_pileup1_t *p = pl + i;
uint32_t q, x = 0, qq;
+ uint16_t y = 0;
if (p->is_del || (p->b->core.flag&BAM_FUNMAP)) continue;
q = (uint32_t)bam1_qual(p->b)[p->qpos];
+ if (q < bm->min_baseQ) continue;
x |= (uint32_t)bam1_strand(p->b) << 18 | q << 8 | p->b->core.qual;
+ y |= bam1_strand(p->b)<<4;
if (p->b->core.qual < q) q = p->b->core.qual;
+ c = p->b->core.qual < bm->cap_mapQ? p->b->core.qual : bm->cap_mapQ;
+ rms += c * c;
x |= q << 24;
+ y |= q << 5;
qq = bam1_seqi(bam1_seq(p->b), p->qpos);
q = bam_nt16_nt4_table[qq? qq : ref_base];
- if (!p->is_del && q < 4) x |= 1 << 21 | q << 16;
+ if (!p->is_del && q < 4) x |= 1 << 21 | q << 16, y |= q;
+ bm->aux->info16[n] = y;
bm->aux->info[n++] = x;
}
+ rms = (uint8_t)(sqrt((double)rms / n) + .499);
+ if (bm->errmod == BAM_ERRMOD_MAQ2) {
+ errmod_cal(bm->aux->em, n, 4, bm->aux->info16, p);
+ goto goto_glf;
+ }
ks_introsort(uint32_t, n, bm->aux->info);
// generate esum and fsum
b = (glf_call_aux_t*)calloc(1, sizeof(glf_call_aux_t));
for (k = 0; k != 8; ++k) w[k] = 0;
- rms = 0;
for (j = n - 1; j >= 0; --j) { // calculate esum and fsum
uint32_t info = bm->aux->info[j];
- int tmp;
if (info>>24 < 4 && (info>>8&0x3f) != 0) info = 4<<24 | (info&0xffffff);
k = info>>16&7;
if (info>>24 > 0) {
if (w[k] < 0xff) ++w[k];
++b->c[k&3];
}
- tmp = (int)(info&0xff) < bm->cap_mapQ? (int)(info&0xff) : bm->cap_mapQ;
- rms += tmp * tmp;
}
- b->rms_mapQ = (uint8_t)(sqrt((double)rms / n) + .499);
// rescale ->c[]
for (j = c = 0; j != 4; ++j) c += b->c[j];
if (c > 255) {
for (j = 0; j != 4; ++j) b->c[j] = (int)(254.0 * b->c[j] / c + 0.5);
for (j = c = 0; j != 4; ++j) c += b->c[j];
}
- if (!bm->is_soap) {
+ if (bm->errmod == BAM_ERRMOD_MAQ) {
// generate likelihood
for (j = 0; j != 4; ++j) {
// homozygous
if (max1 > max2 && (min_k != max_k || min1 + 1.0 > min2))
p[max_k<<2|max_k] = min1 > 1.0? min1 - 1.0 : 0.0;
}
- } else { // apply the SOAP model
+ } else if (bm->errmod == BAM_ERRMOD_SOAP) { // apply the SOAP model
// generate likelihood
for (j = 0; j != 4; ++j) {
float tmp;
}
}
+goto_glf:
// convert necessary information to glf1_t
- g->ref_base = ref_base; g->max_mapQ = b->rms_mapQ;
+ g->ref_base = ref_base; g->max_mapQ = rms;
g->depth = n > 16777215? 16777215 : n;
for (j = 0; j != 4; ++j)
for (k = j; k < 4; ++k)
uint32_t glf2cns(const glf1_t *g, int q_r)
{
- int i, j, k, tmp[16], min = 10000, min2 = 10000, min3 = 10000, min_g = -1, min_g2 = -1;
+ int i, j, k, p[10], ref4;
uint32_t x = 0;
+ ref4 = bam_nt16_nt4_table[g->ref_base];
for (i = k = 0; i < 4; ++i)
for (j = i; j < 4; ++j) {
- tmp[j<<2|i] = -1;
- tmp[i<<2|j] = g->lk[k++] + (i == j? 0 : q_r);
+ int prior = (i == ref4 && j == ref4? 0 : i == ref4 || j == ref4? q_r : q_r + 3);
+ p[k] = (g->lk[k] + prior)<<4 | i<<2 | j;
+ ++k;
}
- for (i = 0; i < 16; ++i) {
- if (tmp[i] < 0) continue;
- if (tmp[i] < min) {
- min3 = min2; min2 = min; min = tmp[i]; min_g2 = min_g; min_g = i;
- } else if (tmp[i] < min2) {
- min3 = min2; min2 = tmp[i]; min_g2 = i;
- } else if (tmp[i] < min3) min3 = tmp[i];
- }
- x = min_g >= 0? (1U<<(min_g>>2&3) | 1U<<(min_g&3)) << 28 : 0xf << 28;
- x |= min_g2 >= 0? (1U<<(min_g2>>2&3) | 1U<<(min_g2&3)) << 24 : 0xf << 24;
- x |= (uint32_t)g->max_mapQ << 16;
- x |= min2 < 10000? (min2 - min < 256? min2 - min : 255) << 8 : 0xff << 8;
- x |= min2 < 10000 && min3 < 10000? (min3 - min2 < 256? min3 - min2 : 255) : 0xff;
+ for (i = 1; i < 10; ++i) // insertion sort
+ for (j = i; j > 0 && p[j] < p[j-1]; --j)
+ k = p[j], p[j] = p[j-1], p[j-1] = k;
+ x = (1u<<(p[0]&3) | 1u<<(p[0]>>2&3)) << 28; // the best genotype
+ x |= (uint32_t)g->max_mapQ << 16; // rms mapQ
+ x |= ((p[1]>>4) - (p[0]>>4) < 256? (p[1]>>4) - (p[0]>>4) : 255) << 8; // consensus Q
+ for (k = 0; k < 10; ++k)
+ if ((p[k]&0xf) == (ref4<<2|ref4)) break;
+ if (k == 10) k = 9;
+ x |= (p[k]>>4) - (p[0]>>4) < 256? (p[k]>>4) - (p[0]>>4) : 255; // snp Q
return x;
}
uint32_t x;
if (n) {
g = bam_maqcns_glfgen(n, pl, 0xf, bm);
- x = glf2cns(g, (int)(bm->q_r + 0.5));
+ x = g->depth == 0? (0xfU<<28 | 0xfU<<24) : glf2cns(g, (int)(bm->q_r + 0.5));
free(g);
} else x = 0xfU<<28 | 0xfU<<24;
return x;
for (i = 0; i < n_types; ++i) {
ka_param_t ap = ka_param_blast;
ap.band_width = 2 * types[n_types - 1] + 2;
- ap.gap_end = 0;
+ ap.gap_end_ext = 0;
// write ref2
for (k = 0, j = left; j <= pos; ++j)
ref2[k++] = bam_nt16_nt4_table[bam_nt16_table[(int)ref[j]]];
#include "glf.h"
+#define BAM_ERRMOD_MAQ2 0
+#define BAM_ERRMOD_MAQ 1
+#define BAM_ERRMOD_SOAP 2
+
struct __bmc_aux_t;
typedef struct {
float het_rate, theta;
- int n_hap, cap_mapQ, is_soap;
+ int n_hap, cap_mapQ, errmod, min_baseQ;
float eta, q_r;
double *fk, *coef;
#include <assert.h>
#include <string.h>
#include <ctype.h>
+#include <math.h>
#include "faidx.h"
#include "sam.h"
#include "kstring.h"
+#include "kaln.h"
void bam_fillmd1_core(bam1_t *b, char *ref, int is_equal, int max_nm)
{
bam_fillmd1_core(b, ref, is_equal, 0);
}
+int bam_cap_mapQ(bam1_t *b, char *ref, int thres)
+{
+ uint8_t *seq = bam1_seq(b), *qual = bam1_qual(b);
+ uint32_t *cigar = bam1_cigar(b);
+ bam1_core_t *c = &b->core;
+ int i, x, y, mm, q, len, clip_l, clip_q;
+ double t;
+ if (thres < 0) thres = 40; // set the default
+ mm = q = len = clip_l = clip_q = 0;
+ for (i = y = 0, x = c->pos; i < c->n_cigar; ++i) {
+ int j, l = cigar[i]>>4, op = cigar[i]&0xf;
+ if (op == BAM_CMATCH) {
+ for (j = 0; j < l; ++j) {
+ int z = y + j;
+ int c1 = bam1_seqi(seq, z), c2 = bam_nt16_table[(int)ref[x+j]];
+ if (ref[x+j] == 0) break; // out of boundary
+ if (c2 != 15 && c1 != 15 && qual[z] >= 13) { // not ambiguous
+ ++len;
+ if (c1 && c1 != c2 && qual[z] >= 13) { // mismatch
+ ++mm;
+ q += qual[z] > 33? 33 : qual[z];
+ }
+ }
+ }
+ if (j < l) break;
+ x += l; y += l; len += l;
+ } else if (op == BAM_CDEL) {
+ for (j = 0; j < l; ++j)
+ if (ref[x+j] == 0) break;
+ if (j < l) break;
+ x += l;
+ } else if (op == BAM_CSOFT_CLIP) {
+ for (j = 0; j < l; ++j) clip_q += qual[y+j];
+ clip_l += l;
+ y += l;
+ } else if (op == BAM_CHARD_CLIP) {
+ clip_q += 13 * l;
+ clip_l += l;
+ } else if (op == BAM_CINS) y += l;
+ else if (op == BAM_CREF_SKIP) x += l;
+ }
+ for (i = 0, t = 1; i < mm; ++i)
+ t *= (double)len / (i+1);
+ t = q - 4.343 * log(t) + clip_q / 5.;
+ if (t > thres) return -1;
+ if (t < 0) t = 0;
+ t = sqrt((thres - t) / thres) * thres;
+// fprintf(stderr, "%s %lf %d\n", bam1_qname(b), t, q);
+ return (int)(t + .499);
+}
+
+int bam_prob_realn(bam1_t *b, const char *ref)
+{
+ int k, i, bw, x, y, yb, ye, xb, xe;
+ uint32_t *cigar = bam1_cigar(b);
+ bam1_core_t *c = &b->core;
+ ka_probpar_t conf = ka_probpar_def;
+ // find the start and end of the alignment
+ if (c->flag & BAM_FUNMAP) return -1;
+ x = c->pos, y = 0, yb = ye = xb = xe = -1;
+ for (k = 0; k < c->n_cigar; ++k) {
+ int op, l;
+ op = cigar[k]&0xf; l = cigar[k]>>4;
+ if (op == BAM_CMATCH) {
+ if (yb < 0) yb = y;
+ if (xb < 0) xb = x;
+ ye = y + l; xe = x + l;
+ x += l; y += l;
+ } else if (op == BAM_CSOFT_CLIP || op == BAM_CINS) y += l;
+ else if (op == BAM_CDEL) x += l;
+ else if (op == BAM_CREF_SKIP) return -1;
+ }
+ // set bandwidth and the start and the end
+ bw = 7;
+ if (abs((xe - xb) - (ye - yb)) > bw)
+ bw = abs((xe - xb) - (ye - yb)) + 3;
+ conf.bw = bw;
+ xb -= yb + bw/2; if (xb < 0) xb = 0;
+ xe += c->l_qseq - ye + bw/2;
+ if (xe - xb - c->l_qseq > bw)
+ xb += (xe - xb - c->l_qseq - bw) / 2, xe -= (xe - xb - c->l_qseq - bw) / 2;
+ { // glocal
+ uint8_t *s, *r, *q, *seq = bam1_seq(b), *qual = bam1_qual(b);
+ int *state;
+ s = calloc(c->l_qseq, 1);
+ for (i = 0; i < c->l_qseq; ++i) s[i] = bam_nt16_nt4_table[bam1_seqi(seq, i)];
+ r = calloc(xe - xb, 1);
+ for (i = xb; i < xe; ++i)
+ r[i-xb] = bam_nt16_nt4_table[bam_nt16_table[(int)ref[i]]];
+ state = calloc(c->l_qseq, sizeof(int));
+ q = calloc(c->l_qseq, 1);
+ ka_prob_glocal(r, xe-xb, s, c->l_qseq, qual, &conf, state, q);
+ for (k = 0, x = c->pos, y = 0; k < c->n_cigar; ++k) {
+ int op = cigar[k]&0xf, l = cigar[k]>>4;
+ if (op == BAM_CMATCH) {
+ for (i = y; i < y + l; ++i) {
+ if ((state[i]&3) != 0 || state[i]>>2 != x - xb + (i - y)) qual[i] = 0;
+ else qual[i] = qual[i] < q[i]? qual[i] : q[i];
+ }
+ x += l; y += l;
+ } else if (op == BAM_CSOFT_CLIP || op == BAM_CINS) y += l;
+ else if (op == BAM_CDEL) x += l;
+ }
+ free(s); free(r); free(q); free(state);
+ }
+ return 0;
+}
+
int bam_fillmd(int argc, char *argv[])
{
- int c, is_equal = 0, tid = -2, ret, len, is_bam_out, is_sam_in, is_uncompressed, max_nm = 0;
+ int c, is_equal = 0, tid = -2, ret, len, is_bam_out, is_sam_in, is_uncompressed, max_nm = 0, is_realn, capQ = 0;
samfile_t *fp, *fpout = 0;
faidx_t *fai;
char *ref = 0, mode_w[8], mode_r[8];
bam1_t *b;
- is_bam_out = is_sam_in = is_uncompressed = 0;
+ is_bam_out = is_sam_in = is_uncompressed = is_realn = 0;
mode_w[0] = mode_r[0] = 0;
strcpy(mode_r, "r"); strcpy(mode_w, "w");
- while ((c = getopt(argc, argv, "eubSn:")) >= 0) {
+ while ((c = getopt(argc, argv, "reubSC:n:")) >= 0) {
switch (c) {
+ case 'r': is_realn = 1; break;
case 'e': is_equal = 1; break;
case 'b': is_bam_out = 1; break;
case 'u': is_uncompressed = is_bam_out = 1; break;
case 'S': is_sam_in = 1; break;
case 'n': max_nm = atoi(optarg); break;
+ case 'C': capQ = atoi(optarg); break;
default: fprintf(stderr, "[bam_fillmd] unrecognized option '-%c'\n", c); return 1;
}
}
if (is_uncompressed) strcat(mode_w, "u");
if (optind + 1 >= argc) {
fprintf(stderr, "\n");
- fprintf(stderr, "Usage: samtools fillmd [-eubS] <aln.bam> <ref.fasta>\n\n");
+ fprintf(stderr, "Usage: samtools fillmd [-eub
rS] <aln.bam> <ref.fasta>\n\n");
fprintf(stderr, "Options: -e change identical bases to '='\n");
fprintf(stderr, " -u uncompressed BAM output (for piping)\n");
fprintf(stderr, " -b compressed BAM output\n");
- fprintf(stderr, " -S the input is SAM with header\n\n");
+ fprintf(stderr, " -S the input is SAM with header\n");
+ fprintf(stderr, " -r read-independent local realignment\n\n");
return 1;
}
fp = samopen(argv[optind], mode_r, 0);
fprintf(stderr, "[bam_fillmd] fail to find sequence '%s' in the reference.\n",
fp->header->target_name[tid]);
}
+// if (is_realn) bam_realn(b, ref);
+ if (is_realn) bam_prob_realn(b, ref);
+ if (capQ > 10) {
+ int q = bam_cap_mapQ(b, ref, capQ);
+ if (b->core.qual > q) b->core.qual = q;
+ }
if (ref) bam_fillmd1_core(b, ref, is_equal, max_nm);
}
samwrite(fpout, b);
#include <assert.h>
#include "sam.h"
+typedef struct {
+ int k, x, y, end;
+} cstate_t;
+
+static cstate_t g_cstate_null = { -1, 0, 0, 0 };
+
typedef struct __linkbuf_t {
bam1_t b;
uint32_t beg, end;
+ cstate_t s;
struct __linkbuf_t *next;
} lbnode_t;
/* --- BEGIN: Auxiliary functions */
-static inline int resolve_cigar(bam_pileup1_t *p, uint32_t pos)
+/* s->k: the index of the CIGAR operator that has just been processed.
+ s->x: the reference coordinate of the start of s->k
+ s->y: the query coordiante of the start of s->k
+ */
+static inline int resolve_cigar2(bam_pileup1_t *p, uint32_t pos, cstate_t *s)
{
- unsigned k;
+#define _cop(c) ((c)&BAM_CIGAR_MASK)
+#define _cln(c) ((c)>>BAM_CIGAR_SHIFT)
+
bam1_t *b = p->b;
bam1_core_t *c = &b->core;
- uint32_t x = c->pos, y = 0;
- int ret = 1, is_restart = 1;
-
- if (c->flag&BAM_FUNMAP) return 0; // unmapped read
- assert(x <= pos); // otherwise a bug
- p->qpos = -1; p->indel = 0; p->is_del = p->is_head = p->is_tail = 0;
- for (k = 0; k < c->n_cigar; ++k) {
- int op = bam1_cigar(b)[k] & BAM_CIGAR_MASK; // operation
- int l = bam1_cigar(b)[k] >> BAM_CIGAR_SHIFT; // length
- if (op == BAM_CMATCH) { // NOTE: this assumes the first and the last operation MUST BE a match or a clip
- if (x + l > pos) { // overlap with pos
- p->indel = p->is_del = 0;
- p->qpos = y + (pos - x);
- if (x == pos && is_restart) p->is_head = 1;
- if (x + l - 1 == pos) { // come to the end of a match
- int has_next_match = 0;
- unsigned i;
- for (i = k + 1; i < c->n_cigar; ++i) {
- uint32_t cigar = bam1_cigar(b)[i];
- int opi = cigar&BAM_CIGAR_MASK;
- if (opi == BAM_CMATCH) {
- has_next_match = 1;
- break;
- } else if (opi == BAM_CSOFT_CLIP || opi == BAM_CREF_SKIP || opi == BAM_CHARD_CLIP) break;
- }
- if (!has_next_match) p->is_tail = 1;
- if (k < c->n_cigar - 1 && has_next_match) { // there are additional operation(s)
- uint32_t cigar = bam1_cigar(b)[k+1]; // next CIGAR
- int op_next = cigar&BAM_CIGAR_MASK; // next CIGAR operation
- if (op_next == BAM_CDEL) p->indel = -(int32_t)(cigar>>BAM_CIGAR_SHIFT); // del
- else if (op_next == BAM_CINS) p->indel = cigar>>BAM_CIGAR_SHIFT; // ins
- else if (op_next == BAM_CPAD && k + 2 < c->n_cigar) { // no working for adjacent padding
- cigar = bam1_cigar(b)[k+2]; op_next = cigar&BAM_CIGAR_MASK;
- if (op_next == BAM_CDEL) p->indel = -(int32_t)(cigar>>BAM_CIGAR_SHIFT); // del
- else if (op_next == BAM_CINS) p->indel = cigar>>BAM_CIGAR_SHIFT; // ins
- }
- }
+ uint32_t *cigar = bam1_cigar(b);
+ int k, is_head = 0;
+ // determine the current CIGAR operation
+// fprintf(stderr, "%s\tpos=%d\tend=%d\t(%d,%d,%d)\n", bam1_qname(b), pos, s->end, s->k, s->x, s->y);
+ if (s->k == -1) { // never processed
+ is_head = 1;
+ if (c->n_cigar == 1) { // just one operation, save a loop
+ if (_cop(cigar[0]) == BAM_CMATCH) s->k = 0, s->x = c->pos, s->y = 0;
+ } else { // find the first match
+ for (k = 0, s->x = c->pos, s->y = 0; k < c->n_cigar; ++k) {
+ int op = _cop(cigar[k]);
+ int l = _cln(cigar[k]);
+ if (op == BAM_CMATCH) break;
+ else if (op == BAM_CDEL || op == BAM_CREF_SKIP) s->x += l;
+ else if (op == BAM_CINS || op == BAM_CSOFT_CLIP) s->y += l;
+ }
+ assert(k < c->n_cigar);
+ s->k = k;
+ }
+ } else { // the read has been processed before
+ int op, l = _cln(cigar[s->k]);
+ if (pos - s->x >= l) { // jump to the next operation
+ assert(s->k < c->n_cigar); // otherwise a bug: this function should not be called in this case
+ op = _cop(cigar[s->k+1]);
+ if (op == BAM_CMATCH || op == BAM_CDEL || op == BAM_CREF_SKIP) { // jump to the next without a loop
+ if (_cop(cigar[s->k]) == BAM_CMATCH) s->y += l;
+ s->x += l;
+ ++s->k;
+ } else { // find the next M/D/N
+ if (_cop(cigar[s->k]) == BAM_CMATCH) s->y += l;
+ s->x += l;
+ for (k = s->k + 1; k < c->n_cigar; ++k) {
+ op = _cop(cigar[k]), l = _cln(cigar[k]);
+ if (op == BAM_CMATCH || op == BAM_CDEL || op == BAM_CREF_SKIP) break;
+ else if (op == BAM_CINS || op == BAM_CSOFT_CLIP) s->y += l;
}
+ s->k = k;
}
- x += l; y += l;
- } else if (op == BAM_CDEL) { // then set ->is_del
- if (x + l > pos) {
- p->indel = 0; p->is_del = 1;
- p->qpos = y + (pos - x);
+ assert(s->k < c->n_cigar); // otherwise a bug
+ } // else, do nothing
+ }
+ { // collect pileup information
+ int op, l;
+ op = _cop(cigar[s->k]); l = _cln(cigar[s->k]);
+ p->is_del = p->indel = p->is_refskip = 0;
+ if (s->x + l - 1 == pos && s->k + 1 < c->n_cigar) { // peek the next operation
+ int op2 = _cop(cigar[s->k+1]);
+ int l2 = _cln(cigar[s->k+1]);
+ if (op2 == BAM_CDEL) p->indel = -(int)l2;
+ else if (op2 == BAM_CINS) p->indel = l2;
+ else if (op2 == BAM_CPAD && s->k + 2 < c->n_cigar) { // no working for adjacent padding
+ int l3 = 0;
+ for (k = s->k + 2; k < c->n_cigar; ++k) {
+ op2 = _cop(cigar[k]); l2 = _cln(cigar[k]);
+ if (op2 == BAM_CINS) l3 += l2;
+ else if (op2 == BAM_CDEL || op2 == BAM_CMATCH || op2 == BAM_CREF_SKIP) break;
+ }
+ if (l3 > 0) p->indel = l3;
}
- x += l;
- } else if (op == BAM_CREF_SKIP) x += l;
- else if (op == BAM_CINS || op == BAM_CSOFT_CLIP) y += l;
- if (is_restart) is_restart ^= (op == BAM_CMATCH);
- else is_restart ^= (op == BAM_CREF_SKIP || op == BAM_CSOFT_CLIP || op == BAM_CHARD_CLIP);
- if (x > pos) {
- if (op == BAM_CREF_SKIP) ret = 0; // then do not put it into pileup at all
- break;
}
+ if (op == BAM_CMATCH) {
+ p->qpos = s->y + (pos - s->x);
+ } else if (op == BAM_CDEL || op == BAM_CREF_SKIP) {
+ p->is_del = 1; p->qpos = s->y; // FIXME: distinguish D and N!!!!!
+ p->is_refskip = (op == BAM_CREF_SKIP);
+ } // cannot be other operations; otherwise a bug
+ p->is_head = (pos == c->pos); p->is_tail = (pos == s->end);
}
- assert(x > pos); // otherwise a bug
- return ret;
+ return 1;
}
/* --- END: Auxiliary functions */
iter->plp = (bam_pileup1_t*)realloc(iter->plp, sizeof(bam_pileup1_t) * iter->max_plp);
}
iter->plp[n_plp].b = &p->b;
- if (resolve_cigar(iter->plp + n_plp, iter->pos)) ++n_plp; // skip the read if we are looking at ref-skip
+ if (resolve_cigar2(iter->plp + n_plp, iter->pos, &p->s)) ++n_plp; // actually always true...
}
}
iter->head = iter->dummy->next; // dummy->next may be changed
if (b->core.flag & iter->flag_mask) return 0;
bam_copy1(&iter->tail->b, b);
iter->tail->beg = b->core.pos; iter->tail->end = bam_calend(&b->core, bam1_cigar(b));
+ iter->tail->s = g_cstate_null; iter->tail->s.end = iter->tail->end - 1; // initialize cstate_t
if (b->core.tid < iter->max_tid) {
fprintf(stderr, "[bam_pileup_core] the input is not sorted (chromosomes out of order)\n");
iter->error = 1;
if (new_min == (uint64_t)-1) return 0;
*_tid = new_min>>32; *_pos = (uint32_t)new_min;
for (i = 0; i < iter->n; ++i) {
- if (iter->pos[i] == iter->min) {
+ if (iter->pos[i] == iter->min) { // FIXME: valgrind reports "uninitialised value(s) at this line"
n_plp[i] = iter->n_plp[i], plp[i] = iter->plp[i];
++ret;
} else n_plp[i] = 0, plp[i] = 0;
#define BAM_PLF_1STBASE 0x80
#define BAM_PLF_ALLBASE 0x100
#define BAM_PLF_READPOS 0x200
+#define BAM_PLF_NOBAQ 0x400
typedef struct {
bam_header_t *h;
uint32_t format;
int tid, len, last_pos;
int mask;
+ int capQ_thres, min_baseQ;
int max_depth; // for indel calling, ignore reads with the depth too high. 0 for unlimited
char *ref;
glfFile fp_glf; // for glf output only
return hash;
}
+static inline int printw(int c, FILE *fp)
+{
+ char buf[16];
+ int l, x;
+ if (c == 0) return fputc('0', fp);
+ for (l = 0, x = c < 0? -c : c; x > 0; x /= 10) buf[l++] = x%10 + '0';
+ if (c < 0) buf[l++] = '-';
+ buf[l] = 0;
+ for (x = 0; x < l/2; ++x) {
+ int y = buf[x]; buf[x] = buf[l-1-x]; buf[l-1-x] = y;
+ }
+ fputs(buf, fp);
+ return 0;
+}
+
// an analogy to pileup_func() below
static int glt3_func(uint32_t tid, uint32_t pos, int n, const bam_pileup1_t *pu, void *data)
{
return 0;
}
-static void pileup_seq(const bam_pileup1_t *p, int pos, int ref_len, const char *ref)
+static inline void pileup_seq(const bam_pileup1_t *p, int pos, int ref_len, const char *ref)
{
- if (p->is_head) printf("^%c", p->b->core.qual > 93? 126 : p->b->core.qual + 33);
+ int j;
+ if (p->is_head) {
+ putchar('^');
+ putchar(p->b->core.qual > 93? 126 : p->b->core.qual + 33);
+ }
if (!p->is_del) {
- int j, rb, c = bam_nt16_rev_table[bam1_seqi(bam1_seq(p->b), p->qpos)];
- rb = (ref && pos < ref_len)? ref[pos] : 'N';
- if (c == '=' || toupper(c) == toupper(rb)) c = bam1_strand(p->b)? ',' : '.';
- else c = bam1_strand(p->b)? tolower(c) : toupper(c);
+ int c = bam_nt16_rev_table[bam1_seqi(bam1_seq(p->b), p->qpos)];
+ if (ref) {
+ int rb = pos < ref_len? ref[pos] : 'N';
+ if (c == '=' || bam_nt16_table[c] == bam_nt16_table[rb]) c = bam1_strand(p->b)? ',' : '.';
+ else c = bam1_strand(p->b)? tolower(c) : toupper(c);
+ } else {
+ if (c == '=') c = bam1_strand(p->b)? ',' : '.';
+ else c = bam1_strand(p->b)? tolower(c) : toupper(c);
+ }
putchar(c);
- if (p->indel > 0) {
- printf("+%d", p->indel);
- for (j = 1; j <= p->indel; ++j) {
- c = bam_nt16_rev_table[bam1_seqi(bam1_seq(p->b), p->qpos + j)];
- putchar(bam1_strand(p->b)? tolower(c) : toupper(c));
- }
- } else if (p->indel < 0) {
- printf("%d", p->indel);
- for (j = 1; j <= -p->indel; ++j) {
- c = (ref && (int)pos+j < ref_len)? ref[pos+j] : 'N';
- putchar(bam1_strand(p->b)? tolower(c) : toupper(c));
- }
+ } else putchar(p->is_refskip? (bam1_strand(p->b)? '<' : '>') : '*');
+ if (p->indel > 0) {
+ putchar('+'); printw(p->indel, stdout);
+ for (j = 1; j <= p->indel; ++j) {
+ int c = bam_nt16_rev_table[bam1_seqi(bam1_seq(p->b), p->qpos + j)];
+ putchar(bam1_strand(p->b)? tolower(c) : toupper(c));
+ }
+ } else if (p->indel < 0) {
+ printw(p->indel, stdout);
+ for (j = 1; j <= -p->indel; ++j) {
+ int c = (ref && (int)pos+j < ref_len)? ref[pos+j] : 'N';
+ putchar(bam1_strand(p->b)? tolower(c) : toupper(c));
}
- } else putchar('*');
+ }
if (p->is_tail) putchar('$');
}
}
rmsQ = (uint64_t)(sqrt((double)rmsQ / n) + .499);
cns = b<<28 | 0xf<<24 | rmsQ<<16 | 60<<8;
- } else cns = bam_maqcns_call(n, pu, d->c);
+ } else {
+ glf1_t *g = bam_maqcns_glfgen(n, pu, bam_nt16_table[rb], d->c);
+ cns = g->depth == 0? (0xfu<<28 | 0xf<<24) : glf2cns(g, (int)(d->c->q_r + .499));
+ free(g);
+ }
}
if ((d->format & (BAM_PLF_CNS|BAM_PLF_INDEL_ONLY)) && d->ref && pos < d->len) { // call indels
int m = (!d->max_depth || d->max_depth>n) ? n : d->max_depth;
}
// when only variant sites are asked for, test if the site is a variant
if ((d->format & BAM_PLF_CNS) && (d->format & BAM_PLF_VAR_ONLY)) {
- if (!(bam_nt16_table[rb] != 15 && cns>>28 != bam_nt16_table[rb])) { // not a SNP
+ if (!(bam_nt16_table[rb] != 15 && cns>>28 != 15 && cns>>28 != bam_nt16_table[rb])) { // not a SNP
if (!(r && (r->gt == 2 || strcmp(r->s[r->gt], "*")))) { // not an indel
if (r) bam_maqindel_ret_destroy(r);
return 0;
}
}
// print the first 3 columns
- printf("%s\t%d\t%c\t", d->h->target_name[tid], pos + 1, rb);
+ fputs(d->h->target_name[tid], stdout); putchar('\t');
+ printw(pos+1, stdout); putchar('\t'); putchar(rb); putchar('\t');
// print consensus information if required
if (d->format & BAM_PLF_CNS) {
- int ref_q, rb4 = bam_nt16_table[rb];
- ref_q = 0;
- if (rb4 != 15 && cns>>28 != 15 && cns>>28 != rb4) { // a SNP
- ref_q = ((cns>>24&0xf) == rb4)? cns>>8&0xff : (cns>>8&0xff) + (cns&0xff);
- if (ref_q > 255) ref_q = 255;
- }
- rms_mapq = cns>>16&0xff;
- printf("%c\t%d\t%d\t%d\t", bam_nt16_rev_table[cns>>28], cns>>8&0xff, ref_q, rms_mapq);
+ putchar(bam_nt16_rev_table[cns>>28]); putchar('\t');
+ printw(cns>>8&0xff, stdout); putchar('\t');
+ printw(cns&0xff, stdout); putchar('\t');
+ printw(cns>>16&0xff, stdout); putchar('\t');
}
// print pileup sequences
- printf("%d\t", n);
- rms_aux = 0; // we need to recalculate rms_mapq when -c is not flagged on the command line
- for (i = 0; i < n; ++i) {
- const bam_pileup1_t *p = pu + i;
- int tmp = p->b->core.qual < d->c->cap_mapQ? p->b->core.qual : d->c->cap_mapQ;
- rms_aux += tmp * tmp;
- pileup_seq(p, pos, d->len, d->ref);
- }
+ printw(n, stdout); putchar('\t');
+ for (i = 0; i < n; ++i)
+ pileup_seq(pu + i, pos, d->len, d->ref);
// finalize rms_mapq
- rms_aux = (uint64_t)(sqrt((double)rms_aux / n) + .499);
- if (rms_mapq < 0) rms_mapq = rms_aux;
+ if (d->format & BAM_PLF_CNS) {
+ for (i = rms_aux = 0; i < n; ++i) {
+ const bam_pileup1_t *p = pu + i;
+ int tmp = p->b->core.qual < d->c->cap_mapQ? p->b->core.qual : d->c->cap_mapQ;
+ rms_aux += tmp * tmp;
+ }
+ rms_aux = (uint64_t)(sqrt((double)rms_aux / n) + .499);
+ if (rms_mapq < 0) rms_mapq = rms_aux;
+ }
putchar('\t');
// print quality
for (i = 0; i < n; ++i) {
for (i = 0; i < n; ++i) {
int x = pu[i].qpos;
int l = pu[i].b->core.l_qseq;
- printf("%d,", x < l/2? x+1 : -((l-1)-x+1));
+ printw(x < l/2? x+1 : -((l-1)-x+1), stdout); putchar(',');
}
}
putchar('\n');
int c, is_SAM = 0;
char *fn_list = 0, *fn_fa = 0, *fn_pos = 0;
pu_data_t *d = (pu_data_t*)calloc(1, sizeof(pu_data_t));
- d->max_depth = 0;
- d->tid = -1; d->mask = BAM_DEF_MASK;
+ d->max_depth = 1024; d->tid = -1; d->mask = BAM_DEF_MASK; d->min_baseQ = 13;
d->c = bam_maqcns_init();
- d->c->is_soap = 1; // change the default model
+ d->c->errmod = BAM_ERRMOD_MAQ2; // change the default model
d->ido = bam_maqindel_opt_init();
- while ((c = getopt(argc, argv, "st:f:cT:N:r:l:d:im:gI:G:vM:S2aR:PA")) >= 0) {
+ while ((c = getopt(argc, argv, "st:f:cT:N:r:l:d:im:gI:G:vM:S2aR:PAQ:C:B")) >= 0) {
switch (c) {
- case 'a': d->c->is_soap = 1; break;
- case 'A': d->c->is_soap = 0; break;
+ case 'Q': d->c->min_baseQ = atoi(optarg); break;
+ case 'C': d->capQ_thres = atoi(optarg); break;
+ case 'B': d->format |= BAM_PLF_NOBAQ; break;
+ case 'a': d->c->errmod = BAM_ERRMOD_SOAP; break;
+ case 'A': d->c->errmod = BAM_ERRMOD_MAQ; break;
case 's': d->format |= BAM_PLF_SIMPLE; break;
case 't': fn_list = strdup(optarg); break;
case 'l': fn_pos = strdup(optarg); break;
default: fprintf(stderr, "Unrecognizd option '-%c'.\n", c); return 1;
}
}
+ if (d->c->errmod != BAM_ERRMOD_MAQ2) d->c->theta += 0.02;
+ if (d->c->theta > 1.0) d->c->theta = 1.0;
if (fn_list) is_SAM = 1;
if (optind == argc) {
fprintf(stderr, "\n");
fprintf(stderr, "Usage: samtools pileup [options] <in.bam>|<in.sam>\n\n");
fprintf(stderr, "Option: -s simple (yet incomplete) pileup format\n");
fprintf(stderr, " -S the input is in SAM\n");
- fprintf(stderr, " -A use the MAQ model for SNP calling\n");
+ fprintf(stderr, " -B disable BAQ computation\n");
+ fprintf(stderr, " -A use the original MAQ model for SNP calling (DEPRECATED)\n");
fprintf(stderr, " -2 output the 2nd best call and quality\n");
fprintf(stderr, " -i only show lines/consensus with indels\n");
- fprintf(stderr, " -m INT filtering reads with bits in INT [%d]\n", d->mask);
+ fprintf(stderr, " -Q INT min base quality (possibly capped by BAQ) [%d]\n", d->c->min_baseQ);
+ fprintf(stderr, " -C INT coefficient for adjusting mapQ of poor mappings [%d]\n", d->capQ_thres);
+ fprintf(stderr, " -m INT filtering reads with bits in INT [0x%x]\n", d->mask);
fprintf(stderr, " -M INT cap mapping quality at INT [%d]\n", d->c->cap_mapQ);
- fprintf(stderr, " -d INT limit maximum depth for indels [unlimited]\n");
+ fprintf(stderr, " -d INT limit maximum depth for indels [%d]\n", d->max_depth);
fprintf(stderr, " -t FILE list of reference sequences (force -S)\n");
fprintf(stderr, " -l FILE list of sites at which pileup is output\n");
fprintf(stderr, " -f FILE reference sequence in the FASTA format\n\n");
- fprintf(stderr, " -c output the SOAPsnp consensus sequence\n");
+ fprintf(stderr, " -c compute the consensus sequence\n");
fprintf(stderr, " -v print variants only (for -c)\n");
- fprintf(stderr, " -g output in the GLFv3 format (suppressing -c/-i/-s)\n");
- fprintf(stderr, " -T FLOAT theta in maq consensus calling model (for -c/-g) [%f]\n", d->c->theta);
- fprintf(stderr, " -N INT number of haplotypes in the sample (for -c/-g) [%d]\n", d->c->n_hap);
- fprintf(stderr, " -r FLOAT prior of a difference between two haplotypes (for -c/-g) [%f]\n", d->c->het_rate);
- fprintf(stderr, " -G FLOAT prior of an indel between two haplotypes (for -c/-g) [%f]\n", d->ido->r_indel);
- fprintf(stderr, " -I INT phred prob. of an indel in sequencing/prep. (for -c/-g) [%d]\n", d->ido->q_indel);
+ fprintf(stderr, " -g output in the GLFv3 format (DEPRECATED)\n");
+ fprintf(stderr, " -T FLOAT theta in maq consensus calling model (for -c) [%.4g]\n", d->c->theta);
+ fprintf(stderr, " -N INT number of haplotypes in the sample (for -c) [%d]\n", d->c->n_hap);
+ fprintf(stderr, " -r FLOAT prior of a difference between two haplotypes (for -c) [%.4g]\n", d->c->het_rate);
+ fprintf(stderr, " -G FLOAT prior of an indel between two haplotypes (for -c) [%.4g]\n", d->ido->r_indel);
+ fprintf(stderr, " -I INT phred prob. of an indel in sequencing/prep. (for -c) [%d]\n", d->ido->q_indel);
fprintf(stderr, "\n");
free(fn_list); free(fn_fa); free(d);
return 1;
}
d->h = fp->header;
if (fn_pos) d->hash = load_pos(fn_pos, d->h);
- sampileup(fp, d->mask, pileup_func, d);
+ { // run pileup
+ extern int bam_prob_realn(bam1_t *b, const char *ref);
+ extern int bam_cap_mapQ(bam1_t *b, char *ref, int thres);
+ bam1_t *b;
+ int ret, tid, pos, n_plp;
+ bam_plp_t iter;
+ const bam_pileup1_t *plp;
+ b = bam_init1();
+ iter = bam_plp_init(0, 0);
+ bam_plp_set_mask(iter, d->mask);
+ while ((ret = samread(fp, b)) >= 0) {
+ int skip = 0;
+ if ((int)b->core.tid < 0) break;
+ // update d->ref if necessary
+ if (d->fai && (int)b->core.tid != d->tid) {
+ free(d->ref);
+ d->ref = faidx_fetch_seq(d->fai, d->h->target_name[b->core.tid], 0, 0x7fffffff, &d->len);
+ d->tid = b->core.tid;
+ }
+ if (d->ref && (d->format&BAM_PLF_CNS) && !(d->format&BAM_PLF_NOBAQ)) bam_prob_realn(b, d->ref);
+ if (d->ref && (d->format&BAM_PLF_CNS) && d->capQ_thres > 10) {
+ int q = bam_cap_mapQ(b, d->ref, d->capQ_thres);
+ if (q < 0) skip = 1;
+ else if (b->core.qual > q) b->core.qual = q;
+ } else if (b->core.flag&BAM_FUNMAP) skip = 1;
+ else if ((d->format&BAM_PLF_CNS) && (b->core.flag&1) && !(b->core.flag&2)) skip = 1;
+ if (skip) continue;
+ bam_plp_push(iter, b);
+ while ((plp = bam_plp_next(iter, &tid, &pos, &n_plp)) != 0)
+ pileup_func(tid, pos, n_plp, plp, d);
+ }
+ bam_plp_push(iter, 0);
+ while ((plp = bam_plp_next(iter, &tid, &pos, &n_plp)) != 0)
+ pileup_func(tid, pos, n_plp, plp, d);
+ bam_plp_destroy(iter);
+ bam_destroy1(b);
+ }
samclose(fp); // d->h will be destroyed here
}
* mpileup *
***********/
+#include <assert.h>
+#include "bam2bcf.h"
+#include "sample.h"
+
+#define MPLP_GLF 0x10
+#define MPLP_NO_COMP 0x20
+#define MPLP_NO_ORPHAN 0x40
+#define MPLP_REALN 0x80
+
typedef struct {
- char *reg;
+ int max_mq, min_mq, flag, min_baseQ, capQ_thres;
+ char *reg, *fn_pos;
faidx_t *fai;
+ kh_64_t *hash;
} mplp_conf_t;
typedef struct {
bamFile fp;
bam_iter_t iter;
+ int min_mq, flag, ref_id, capQ_thres;
+ char *ref;
} mplp_aux_t;
+typedef struct {
+ int n;
+ int *n_plp, *m_plp;
+ bam_pileup1_t **plp;
+} mplp_pileup_t;
+
static int mplp_func(void *data, bam1_t *b)
{
+ extern int bam_realn(bam1_t *b, const char *ref);
+ extern int bam_prob_realn(bam1_t *b, const char *ref);
+ extern int bam_cap_mapQ(bam1_t *b, char *ref, int thres);
mplp_aux_t *ma = (mplp_aux_t*)data;
- if (ma->iter) return bam_iter_read(ma->fp, ma->iter, b);
- return bam_read1(ma->fp, b);
+ int ret, skip = 0;
+ do {
+ int has_ref = (ma->ref && ma->ref_id == b->core.tid)? 1 : 0;
+ ret = ma->iter? bam_iter_read(ma->fp, ma->iter, b) : bam_read1(ma->fp, b);
+ if (ret < 0) break;
+ skip = 0;
+ if (has_ref && (ma->flag&MPLP_REALN)) bam_prob_realn(b, ma->ref);
+ if (has_ref && ma->capQ_thres > 10) {
+ int q = bam_cap_mapQ(b, ma->ref, ma->capQ_thres);
+ if (q < 0) skip = 1;
+ else if (b->core.qual > q) b->core.qual = q;
+ } else if (b->core.flag&BAM_FUNMAP) skip = 1;
+ else if (b->core.qual < ma->min_mq) skip = 1;
+ else if ((ma->flag&MPLP_NO_ORPHAN) && (b->core.flag&1) && !(b->core.flag&2)) skip = 1;
+ } while (skip);
+ return ret;
+}
+
+static void group_smpl(mplp_pileup_t *m, bam_sample_t *sm, kstring_t *buf,
+ int n, char *const*fn, int *n_plp, const bam_pileup1_t **plp)
+{
+ int i, j;
+ memset(m->n_plp, 0, m->n * sizeof(int));
+ for (i = 0; i < n; ++i) {
+ for (j = 0; j < n_plp[i]; ++j) {
+ const bam_pileup1_t *p = plp[i] + j;
+ uint8_t *q;
+ int id = -1;
+ q = bam_aux_get(p->b, "RG");
+ if (q) id = bam_smpl_rg2smid(sm, fn[i], (char*)q+1, buf);
+ if (id < 0) id = bam_smpl_rg2smid(sm, fn[i], 0, buf);
+ assert(id >= 0 && id < m->n);
+ if (m->n_plp[id] == m->m_plp[id]) {
+ m->m_plp[id] = m->m_plp[id]? m->m_plp[id]<<1 : 8;
+ m->plp[id] = realloc(m->plp[id], sizeof(bam_pileup1_t) * m->m_plp[id]);
+ }
+ m->plp[id][m->n_plp[id]++] = *p;
+ }
+ }
}
static int mpileup(mplp_conf_t *conf, int n, char **fn)
bam_mplp_t iter;
bam_header_t *h = 0;
char *ref;
- // allocate
+ khash_t(64) *hash = 0;
+
+ bcf_callaux_t *bca = 0;
+ bcf_callret1_t *bcr = 0;
+ bcf_call_t bc;
+ bcf_t *bp = 0;
+ bcf_hdr_t *bh = 0;
+
+ bam_sample_t *sm = 0;
+ kstring_t buf;
+ mplp_pileup_t gplp;
+
+ memset(&gplp, 0, sizeof(mplp_pileup_t));
+ memset(&buf, 0, sizeof(kstring_t));
+ memset(&bc, 0, sizeof(bcf_call_t));
data = calloc(n, sizeof(void*));
plp = calloc(n, sizeof(void*));
n_plp = calloc(n, sizeof(int*));
+ sm = bam_smpl_init();
+
// read the header and initialize data
for (i = 0; i < n; ++i) {
bam_header_t *h_tmp;
data[i] = calloc(1, sizeof(mplp_aux_t));
- data[i]->fp = bam_open(fn[i], "r");
+ data[i]->min_mq = conf->min_mq;
+ data[i]->flag = conf->flag;
+ data[i]->capQ_thres = conf->capQ_thres;
+ data[i]->fp = strcmp(fn[i], "-") == 0? bam_dopen(fileno(stdin), "r") : bam_open(fn[i], "r");
h_tmp = bam_header_read(data[i]->fp);
+ bam_smpl_add(sm, fn[i], h_tmp->text);
if (conf->reg) {
int beg, end;
bam_index_t *idx;
bam_header_destroy(h_tmp);
}
}
- // mpileup
+ gplp.n = sm->n;
+ gplp.n_plp = calloc(sm->n, sizeof(int));
+ gplp.m_plp = calloc(sm->n, sizeof(int));
+ gplp.plp = calloc(sm->n, sizeof(void*));
+
+ fprintf(stderr, "[%s] %d samples in %d input files\n", __func__, sm->n, n);
+ if (conf->fn_pos) hash = load_pos(conf->fn_pos, h);
+ // write the VCF header
+ if (conf->flag & MPLP_GLF) {
+ kstring_t s;
+ bh = calloc(1, sizeof(bcf_hdr_t));
+ s.l = s.m = 0; s.s = 0;
+ bp = bcf_open("-", (conf->flag&MPLP_NO_COMP)? "wu" : "w");
+ for (i = 0; i < h->n_targets; ++i) {
+ kputs(h->target_name[i], &s);
+ kputc('\0', &s);
+ }
+ bh->l_nm = s.l;
+ bh->name = malloc(s.l);
+ memcpy(bh->name, s.s, s.l);
+ s.l = 0;
+ for (i = 0; i < sm->n; ++i) {
+ kputs(sm->smpl[i], &s); kputc('\0', &s);
+ }
+ bh->l_smpl = s.l;
+ bh->sname = malloc(s.l);
+ memcpy(bh->sname, s.s, s.l);
+ bh->l_txt = 0;
+ free(s.s);
+ bcf_hdr_sync(bh);
+ bcf_hdr_write(bp, bh);
+ bca = bcf_call_init(-1., conf->min_baseQ);
+ bcr = calloc(sm->n, sizeof(bcf_callret1_t));
+ }
ref_tid = -1; ref = 0;
iter = bam_mplp_init(n, mplp_func, (void**)data);
while (bam_mplp_auto(iter, &tid, &pos, n_plp, plp) > 0) {
if (conf->reg && (pos < beg0 || pos >= end0)) continue; // out of the region requested
+ if (hash) {
+ khint_t k;
+ k = kh_get(64, hash, (uint64_t)tid<<32 | pos);
+ if (k == kh_end(hash)) continue;
+ }
if (tid != ref_tid) {
- free(ref);
+ free(ref); ref = 0;
if (conf->fai) ref = fai_fetch(conf->fai, h->target_name[tid], &ref_len);
+ for (i = 0; i < n; ++i) data[i]->ref = ref, data[i]->ref_id = tid;
ref_tid = tid;
}
- printf("%s\t%d\t%c", h->target_name[tid], pos + 1, (ref && pos < ref_len)? ref[pos] : 'N');
- for (i = 0; i < n; ++i) {
- int j;
- printf("\t%d\t", n_plp[i]);
- if (n_plp[i] == 0) printf("*\t*");
- else {
- for (j = 0; j < n_plp[i]; ++j)
- pileup_seq(plp[i] + j, pos, ref_len, ref);
- putchar('\t');
- for (j = 0; j < n_plp[i]; ++j) {
- const bam_pileup1_t *p = plp[i] + j;
- int c = bam1_qual(p->b)[p->qpos] + 33;
- if (c > 126) c = 126;
- putchar(c);
+ if (conf->flag & MPLP_GLF) {
+ int _ref0, ref16;
+ bcf1_t *b = calloc(1, sizeof(bcf1_t));
+ group_smpl(&gplp, sm, &buf, n, fn, n_plp, plp);
+ _ref0 = (ref && pos < ref_len)? ref[pos] : 'N';
+ ref16 = bam_nt16_table[_ref0];
+ for (i = 0; i < gplp.n; ++i)
+ bcf_call_glfgen(gplp.n_plp[i], gplp.plp[i], ref16, bca, bcr + i);
+ bcf_call_combine(gplp.n, bcr, ref16, &bc);
+ bcf_call2bcf(tid, pos, &bc, b);
+ bcf_write(bp, bh, b);
+ bcf_destroy(b);
+ } else {
+ printf("%s\t%d\t%c", h->target_name[tid], pos + 1, (ref && pos < ref_len)? ref[pos] : 'N');
+ for (i = 0; i < n; ++i) {
+ int j;
+ printf("\t%d\t", n_plp[i]);
+ if (n_plp[i] == 0) printf("*\t*");
+ else {
+ for (j = 0; j < n_plp[i]; ++j)
+ pileup_seq(plp[i] + j, pos, ref_len, ref);
+ putchar('\t');
+ for (j = 0; j < n_plp[i]; ++j) {
+ const bam_pileup1_t *p = plp[i] + j;
+ int c = bam1_qual(p->b)[p->qpos] + 33;
+ if (c > 126) c = 126;
+ putchar(c);
+ }
}
}
+ putchar('\n');
}
- putchar('\n');
}
+
+ bcf_close(bp);
+ bam_smpl_destroy(sm); free(buf.s);
+ for (i = 0; i < gplp.n; ++i) free(gplp.plp[i]);
+ free(gplp.plp); free(gplp.n_plp); free(gplp.m_plp);
+ if (hash) { // free the hash table
+ khint_t k;
+ for (k = kh_begin(hash); k < kh_end(hash); ++k)
+ if (kh_exist(hash, k)) free(kh_val(hash, k));
+ kh_destroy(64, hash);
+ }
+ bcf_hdr_destroy(bh); bcf_call_destroy(bca); free(bc.PL); free(bcr);
bam_mplp_destroy(iter);
bam_header_destroy(h);
for (i = 0; i < n; ++i) {
int c;
mplp_conf_t mplp;
memset(&mplp, 0, sizeof(mplp_conf_t));
- while ((c = getopt(argc, argv, "f:r:")) >= 0) {
+ mplp.max_mq = 60;
+ mplp.min_baseQ = 13;
+ mplp.capQ_thres = 0;
+ mplp.flag = MPLP_NO_ORPHAN | MPLP_REALN;
+ while ((c = getopt(argc, argv, "gf:r:l:M:q:Q:uaORC:B")) >= 0) {
switch (c) {
case 'f':
mplp.fai = fai_load(optarg);
if (mplp.fai == 0) return 1;
break;
- case 'r': mplp.reg = strdup(optarg);
+ case 'r': mplp.reg = strdup(optarg); break;
+ case 'l': mplp.fn_pos = strdup(optarg); break;
+ case 'g': mplp.flag |= MPLP_GLF; break;
+ case 'u': mplp.flag |= MPLP_NO_COMP | MPLP_GLF; break;
+ case 'a': mplp.flag |= MPLP_NO_ORPHAN | MPLP_REALN; break;
+ case 'B': mplp.flag &= ~MPLP_REALN & ~MPLP_NO_ORPHAN; break;
+ case 'O': mplp.flag |= MPLP_NO_ORPHAN; break;
+ case 'R': mplp.flag |= MPLP_REALN; break;
+ case 'C': mplp.capQ_thres = atoi(optarg); break;
+ case 'M': mplp.max_mq = atoi(optarg); break;
+ case 'q': mplp.min_mq = atoi(optarg); break;
+ case 'Q': mplp.min_baseQ = atoi(optarg); break;
}
}
if (argc == 1) {
- fprintf(stderr, "Usage: samtools mpileup [-r reg] [-f in.fa] in1.bam [in2.bam [...]]\n");
+ fprintf(stderr, "\n");
+ fprintf(stderr, "Usage: samtools mpileup [options] in1.bam [in2.bam [...]]\n\n");
+ fprintf(stderr, "Options: -f FILE reference sequence file [null]\n");
+ fprintf(stderr, " -r STR region in which pileup is generated [null]\n");
+ fprintf(stderr, " -l FILE list of positions (format: chr pos) [null]\n");
+ fprintf(stderr, " -M INT cap mapping quality at INT [%d]\n", mplp.max_mq);
+ fprintf(stderr, " -Q INT min base quality [%d]\n", mplp.min_baseQ);
+ fprintf(stderr, " -q INT filter out alignment with MQ smaller than INT [%d]\n", mplp.min_mq);
+ fprintf(stderr, " -g generate BCF output\n");
+ fprintf(stderr, " -u do not compress BCF output\n");
+ fprintf(stderr, " -B disable BAQ computation\n");
+ fprintf(stderr, "\n");
+ fprintf(stderr, "Notes: Assuming diploid individuals.\n\n");
return 1;
}
mpileup(&mplp, argc - optind, argv + optind);
#include <stdlib.h>
#include <ctype.h>
#include <assert.h>
+#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
temps = h1->text, h1->text = h2->text, h2->text = temps;
}
+#define MERGE_RG 1
+#define MERGE_UNCOMP 2
+
/*!
@abstract Merge multiple sorted BAM.
@param is_by_qname whether to sort by query name
@discussion Padding information may NOT correctly maintained. This
function is NOT thread safe.
*/
-void bam_merge_core(int by_qname, const char *out, const char *headers, int n, char * const *fn, int add_RG)
+int bam_merge_core(int by_qname, const char *out, const char *headers, int n, char * const *fn,
+ int flag, const char *reg)
{
bamFile fpout, *fp;
heap1_t *heap;
int i, j, *RG_len = 0;
uint64_t idx = 0;
char **RG = 0;
+ bam_iter_t *iter = 0;
if (headers) {
tamFile fpheaders = sam_open(headers);
if (fpheaders == 0) {
- fprintf(stderr, "[bam_merge_core] Cannot open file `%s'. Continue anyway.\n", headers);
- } else {
- hheaders = sam_header_read(fpheaders);
- sam_close(fpheaders);
+ const char *message = strerror(errno);
+ fprintf(stderr, "[bam_merge_core] cannot open '%s': %s\n", headers, message);
+ return -1;
}
+ hheaders = sam_header_read(fpheaders);
+ sam_close(fpheaders);
}
g_is_by_qname = by_qname;
fp = (bamFile*)calloc(n, sizeof(bamFile));
heap = (heap1_t*)calloc(n, sizeof(heap1_t));
+ iter = (bam_iter_t*)calloc(n, sizeof(bam_iter_t));
// prepare RG tag
- if (add_RG) {
+ if (flag & MERGE_RG) {
RG = (char**)calloc(n, sizeof(void*));
RG_len = (int*)calloc(n, sizeof(int));
for (i = 0; i != n; ++i) {
}
// read the first
for (i = 0; i != n; ++i) {
- heap1_t *h;
bam_header_t *hin;
fp[i] = bam_open(fn[i], "r");
if (fp[i] == 0) {
for (j = 0; j < i; ++j) bam_close(fp[j]);
free(fp); free(heap);
// FIXME: possible memory leak
- return;
+ return -1;
}
hin = bam_header_read(fp[i]);
if (i == 0) { // the first SAM
// check that they are consistent with the existing binary list
// of reference information.
if (hheaders->n_targets > 0) {
- if (hout->n_targets != hheaders->n_targets)
+ if (hout->n_targets != hheaders->n_targets) {
fprintf(stderr, "[bam_merge_core] number of @SQ headers in `%s' differs from number of target sequences", headers);
+ if (!reg) return -1;
+ }
for (j = 0; j < hout->n_targets; ++j)
- if (strcmp(hout->target_name[j], hheaders->target_name[j]) != 0)
+ if (strcmp(hout->target_name[j], hheaders->target_name[j]) != 0) {
fprintf(stderr, "[bam_merge_core] @SQ header '%s' in '%s' differs from target sequence", hheaders->target_name[j], headers);
+ if (!reg) return -1;
+ }
}
swap_header_text(hout, hheaders);
bam_header_destroy(hheaders);
}
} else { // validate multiple baf
if (hout->n_targets != hin->n_targets) {
- fprintf(stderr, "[bam_merge_core] file '%s' has different number of target sequences. Abort!\n", fn[i]);
- exit(1);
- }
- for (j = 0; j < hout->n_targets; ++j) {
- if (strcmp(hout->target_name[j], hin->target_name[j])) {
- fprintf(stderr, "[bam_merge_core] different target sequence name: '%s' != '%s' in file '%s'. Abort!\n",
- hout->target_name[j], hin->target_name[j], fn[i]);
- exit(1);
+ fprintf(stderr, "[bam_merge_core] file '%s' has different number of target sequences. Continue anyway!\n", fn[i]);
+ } else {
+ for (j = 0; j < hout->n_targets; ++j) {
+ if (strcmp(hout->target_name[j], hin->target_name[j])) {
+ fprintf(stderr, "[bam_merge_core] different target sequence name: '%s' != '%s' in file '%s'. Continue anyway!\n",
+ hout->target_name[j], hin->target_name[j], fn[i]);
+ }
}
}
bam_header_destroy(hin);
}
- h = heap + i;
+ }
+
+ if (reg) {
+ int tid, beg, end;
+ if (bam_parse_region(hout, reg, &tid, &beg, &end) < 0) {
+ fprintf(stderr, "[%s] Malformated region string or undefined reference name\n", __func__);
+ return -1;
+ }
+ for (i = 0; i < n; ++i) {
+ bam_index_t *idx;
+ idx = bam_index_load(fn[i]);
+ iter[i] = bam_iter_query(idx, tid, beg, end);
+ bam_index_destroy(idx);
+ }
+ }
+
+ for (i = 0; i < n; ++i) {
+ heap1_t *h = heap + i;
h->i = i;
h->b = (bam1_t*)calloc(1, sizeof(bam1_t));
- if (bam_read1(fp[i], h->b) >= 0) {
+ if (bam_iter_read(fp[i], iter[i], h->b) >= 0) {
h->pos = ((uint64_t)h->b->core.tid<<32) | (uint32_t)h->b->core.pos<<1 | bam1_strand(h->b);
h->idx = idx++;
}
else h->pos = HEAP_EMPTY;
}
- fpout = strcmp(out, "-")? bam_open(out, "w") : bam_dopen(fileno(stdout), "w");
- assert(fpout);
+ if (flag & MERGE_UNCOMP) {
+ fpout = strcmp(out, "-")? bam_open(out, "wu") : bam_dopen(fileno(stdout), "wu");
+ } else {
+ fpout = strcmp(out, "-")? bam_open(out, "w") : bam_dopen(fileno(stdout), "w");
+ }
+ if (fpout == 0) {
+ fprintf(stderr, "[%s] fail to create the output file.\n", __func__);
+ return -1;
+ }
bam_header_write(fpout, hout);
bam_header_destroy(hout);
ks_heapmake(heap, n, heap);
while (heap->pos != HEAP_EMPTY) {
bam1_t *b = heap->b;
- if (add_RG && bam_aux_get(b, "RG") == 0)
+ if ((flag & MERGE_RG) && bam_aux_get(b, "RG") == 0)
bam_aux_append(b, "RG", 'Z', RG_len[heap->i] + 1, (uint8_t*)RG[heap->i]);
bam_write1_core(fpout, &b->core, b->data_len, b->data);
- if ((j = bam_read1(fp[heap->i], b)) >= 0) {
+ if ((j = bam_iter_read(fp[heap->i], iter[heap->i], b)) >= 0) {
heap->pos = ((uint64_t)b->core.tid<<32) | (uint32_t)b->core.pos<<1 | bam1_strand(b);
heap->idx = idx++;
} else if (j == -1) {
ks_heapadjust(heap, 0, n, heap);
}
- if (add_RG) {
+ if (flag & MERGE_RG) {
for (i = 0; i != n; ++i) free(RG[i]);
free(RG); free(RG_len);
}
- for (i = 0; i != n; ++i) bam_close(fp[i]);
+ for (i = 0; i != n; ++i) {
+ bam_iter_destroy(iter[i]);
+ bam_close(fp[i]);
+ }
bam_close(fpout);
- free(fp); free(heap);
+ free(fp); free(heap); free(iter);
+ return 0;
}
+
int bam_merge(int argc, char *argv[])
{
- int c, is_by_qname = 0, add_RG = 0;
- char *fn_headers = NULL;
+ int c, is_by_qname = 0, flag = 0, ret = 0;
+ char *fn_headers = NULL, *reg = 0;
- while ((c = getopt(argc, argv, "h:nr")) >= 0) {
+ while ((c = getopt(argc, argv, "h:nruR:")) >= 0) {
switch (c) {
- case 'r': add_RG = 1; break;
+ case 'r': flag |= MERGE_RG; break;
case 'h': fn_headers = strdup(optarg); break;
case 'n': is_by_qname = 1; break;
+ case 'u': flag |= MERGE_UNCOMP; break;
+ case 'R': reg = strdup(optarg); break;
}
}
if (optind + 2 >= argc) {
fprintf(stderr, "Usage: samtools merge [-nr] [-h inh.sam] <out.bam> <in1.bam> <in2.bam> [...]\n\n");
fprintf(stderr, "Options: -n sort by read names\n");
fprintf(stderr, " -r attach RG tag (inferred from file names)\n");
+ fprintf(stderr, " -u uncompressed BAM output\n");
+ fprintf(stderr, " -R STR merge file in the specified region STR [all]\n");
fprintf(stderr, " -h FILE copy the header in FILE to <out.bam> [in1.bam]\n\n");
fprintf(stderr, "Note: Samtools' merge does not reconstruct the @RG dictionary in the header. Users\n");
fprintf(stderr, " must provide the correct header with -h, or uses Picard which properly maintains\n");
fprintf(stderr, " the header dictionary in merging.\n\n");
return 1;
}
- bam_merge_core(is_by_qname, argv[optind], fn_headers, argc - optind - 1, argv + optind + 1, add_RG);
+ if (bam_merge_core(is_by_qname, argv[optind], fn_headers, argc - optind - 1, argv + optind + 1, flag, reg) < 0) ret = 1;
+ free(reg);
free(fn_headers);
- return 0;
+ return ret;
}
typedef bam1_t *bam1_p;
fns[i] = (char*)calloc(strlen(prefix) + 20, 1);
sprintf(fns[i], "%s.%.4d.bam", prefix, i);
}
- bam_merge_core(is_by_qname, fnout, 0, n, fns, 0);
+ bam_merge_core(is_by_qname, fnout, 0, n, fns, 0, 0);
free(fnout);
for (i = 0; i < n; ++i) {
unlink(fns[i]);
if (tv->is_dot && toupper(c) == toupper(rb)) c = bam1_strand(p->b)? ',' : '.';
}
}
- } else c = '*';
+ } else c = p->is_refskip? (bam1_strand(p->b)? '<' : '>') : '*';
} else { // padding
if (j > p->indel) c = '*';
else { // insertion
} else if (c == KEY_ENTER || c == '\012' || c == '\015') {
int _tid = -1, _beg, _end;
if (str[0] == '=') {
- _beg = strtol(str+1, &p, 10);
+ _beg = strtol(str+1, &p, 10) - 1;
if (_beg > 0) {
*pos = _beg;
return;
#endif
#ifndef PACKAGE_VERSION
-#define PACKAGE_VERSION "0.1.8 (r613)"
+#define PACKAGE_VERSION "0.1.9 (r783)"
#endif
int bam_taf2baf(int argc, char *argv[]);
--- /dev/null
+CC= gcc
+CFLAGS= -g -Wall -O2 #-m64 #-arch ppc
+DFLAGS= -D_FILE_OFFSET_BITS=64 -D_USE_KNETFILE
+LOBJS= bcf.o vcf.o bcfutils.o prob1.o ld.o kfunc.o index.o fet.o bcf2qcall.o
+OMISC= ..
+AOBJS= call1.o main.o $(OMISC)/kstring.o $(OMISC)/bgzf.o $(OMISC)/knetfile.o
+PROG= bcftools
+INCLUDES=
+SUBDIRS= .
+
+.SUFFIXES:.c .o
+
+.c.o:
+ $(CC) -c $(CFLAGS) $(DFLAGS) -I.. $(INCLUDES) $< -o $@
+
+all-recur lib-recur clean-recur cleanlocal-recur install-recur:
+ @target=`echo $@ | sed s/-recur//`; \
+ wdir=`pwd`; \
+ list='$(SUBDIRS)'; for subdir in $$list; do \
+ cd $$subdir; \
+ $(MAKE) CC="$(CC)" DFLAGS="$(DFLAGS)" CFLAGS="$(CFLAGS)" \
+ INCLUDES="$(INCLUDES)" LIBPATH="$(LIBPATH)" $$target || exit 1; \
+ cd $$wdir; \
+ done;
+
+all:$(PROG)
+
+lib:libbcf.a
+
+libbcf.a:$(LOBJS)
+ $(AR) -cru $@ $(LOBJS)
+
+bcftools:lib $(AOBJS)
+ $(CC) $(CFLAGS) -o $@ $(AOBJS) -lm $(LIBPATH) -lz -L. -lbcf
+
+bcf.o:bcf.h
+vcf.o:bcf.h
+index.o:bcf.h
+bcfutils.o:bcf.h
+prob1.o:prob1.h bcf.h
+call1.o:prob1.h bcf.h
+bcf2qcall.o:bcf.h
+main.o:bcf.h
+
+bcf.pdf:bcf.tex
+ pdflatex bcf
+
+cleanlocal:
+ rm -fr gmon.out *.o a.out *.dSYM $(PROG) *~ *.a bcf.aux bcf.log bcf.pdf *.class libbcf.*.dylib libbcf.so*
+
+clean:cleanlocal-recur
--- /dev/null
+The view command of bcftools calls variants, tests Hardy-Weinberg
+equilibrium (HWE), tests allele balances and estimates allele frequency.
+
+This command calls a site as a potential variant if P(ref|D,F) is below
+0.9 (controlled by the -p option), where D is data and F is the prior
+allele frequency spectrum (AFS).
+
+The view command performs two types of allele balance tests, both based
+on Fisher's exact test for 2x2 contingency tables with the row variable
+being reference allele or not. In the first table, the column variable
+is strand. Two-tail P-value is taken. We test if variant bases tend to
+come from one strand. In the second table, the column variable is
+whether a base appears in the first or the last 11bp of the read.
+One-tail P-value is taken. We test if variant bases tend to occur
+towards the end of reads, which is usually an indication of
+misalignment.
+
+Site allele frequency is estimated in two ways. In the first way, the
+frequency is esimated as \argmax_f P(D|f) under the assumption of
+HWE. Prior AFS is not used. In the second way, the frequency is
+estimated as the posterior expectation of allele counts \sum_k
+kP(k|D,F), dividied by the total number of haplotypes. HWE is not
+assumed, but the estimate depends on the prior AFS. The two estimates
+largely agree when the signal is strong, but may differ greatly on weak
+sites as in this case, the prior plays an important role.
+
+To test HWE, we calculate the posterior distribution of genotypes
+(ref-hom, het and alt-hom). Chi-square test is performed. It is worth
+noting that the model used here is prior dependent and assumes HWE,
+which is different from both models for allele frequency estimate. The
+new model actually yields a third estimate of site allele frequency.
+
+The estimate allele frequency spectrum is printed to stderr per 64k
+sites. The estimate is in fact only the first round of a EM
+procedure. The second model (not the model for HWE testing) is used to
+estimate the AFS.
\ No newline at end of file
--- /dev/null
+#!/usr/bin/perl -w
+
+use strict;
+use warnings;
+use Carp;
+
+my $opts = parse_params();
+bcf_fix();
+
+exit;
+
+#--------------------------------
+
+sub error
+{
+ my (@msg) = @_;
+ if ( scalar @msg ) { confess @msg; }
+ die
+ "Usage: bcftools view test.bcf | bcf-fix.pl > test.vcf\n",
+ "Options:\n",
+ " -h, -?, --help This help message.\n",
+ "\n";
+}
+
+
+sub parse_params
+{
+ my $opts = {};
+ while (my $arg=shift(@ARGV))
+ {
+ if ( $arg eq '-?' || $arg eq '-h' || $arg eq '--help' ) { error(); }
+ error("Unknown parameter \"$arg\". Run -h for help.\n");
+ }
+ return $opts;
+}
+
+sub bcf_fix
+{
+ while (my $line=<STDIN>)
+ {
+ if ( $line=~/^#CHROM/ )
+ {
+ print
+qq[##INFO=<ID=DP4,Number=4,Type=Integer,Description="Read depth for 1) forward reference bases; 2) reverse ref; 3) forward non-ref; 4) reverse non-ref">
+##INFO=<ID=PV4,Number=4,Type=Float,Description="P-values for 1) strand bias (exact test); 2) baseQ bias (t-test); 3) mapQ bias (t); 4) tail distance bias (t)">
+##INFO=<ID=AF1,Number=1,Type=Float,Description="EM estimate of site allele frequency without prior">
+##INFO=<ID=AFE,Number=1,Type=Float,Description="Posterior expectation of site allele frequency (with prior)">
+##INFO=<ID=HWE,Number=1,Type=Float,Description="P-value for Hardy-Weinberg equillibrium (chi-square test)">
+##INFO=<ID=MQ,Number=1,Type=Integer,Descriptin="RMS mapping quality">
+##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
+##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype Quality">
+##FORMAT=<ID=GL,Number=3,Type=Float,Description="Likelihoods for RR,RA,AA genotypes (R=ref,A=alt)">
+];
+ print $line;
+ }
+ elsif ( $line=~/^#/ )
+ {
+ print $line;
+ }
+ else
+ {
+ my @items = split(/\t/,$line);
+ my @tags = split(/:/,$items[8]); # FORMAT tags
+
+ my $nidx=2;
+ my @idxs; # Mapping which defines new ordering: $idxs[$inew]=$iold; GT comes first, PL second
+ for (my $i=0; $i<@tags; $i++)
+ {
+ if ( $tags[$i] eq 'GT' ) { $idxs[0]=$i; }
+ elsif ( $tags[$i] eq 'PL' ) { $idxs[1]=$i; }
+ else { $idxs[$nidx++]=$i; }
+ }
+ if ( !exists($tags[0]) or !exists($tags[1]) ) { error("FIXME: expected GT and PL in the format field.\n"); }
+
+ # First fix the FORMAT column
+ $items[8] = 'GT:GL';
+ for (my $i=2; $i<@tags; $i++)
+ {
+ $items[8] .= ':'.$tags[$idxs[$i]];
+ }
+
+ # Now all the genotype columns
+ for (my $iitem=9; $iitem<@items; $iitem++)
+ {
+ @tags = split(/:/,$items[$iitem]);
+ $items[$iitem] = $tags[$idxs[0]] .':';
+
+ # GL=-PL/10
+ my ($a,$b,$c) = split(/,/,$tags[$idxs[1]]);
+ $items[$iitem] .= sprintf "%.2f,%.2f,%.2f",-$a/10.,-$b/10.,-$c/10.;
+
+ for (my $itag=2; $itag<@tags; $itag++)
+ {
+ $items[$iitem] .= ':'.$tags[$idxs[$itag]];
+ }
+ }
+ print join("\t",@items);
+ }
+ }
+}
+
--- /dev/null
+#include <string.h>
+#include <ctype.h>
+#include <stdio.h>
+#include "kstring.h"
+#include "bcf.h"
+
+bcf_t *bcf_open(const char *fn, const char *mode)
+{
+ bcf_t *b;
+ b = calloc(1, sizeof(bcf_t));
+ if (strchr(mode, 'w')) {
+ b->fp = strcmp(fn, "-")? bgzf_open(fn, mode) : bgzf_fdopen(fileno(stdout), mode);
+ } else {
+ b->fp = strcmp(fn, "-")? bgzf_open(fn, mode) : bgzf_fdopen(fileno(stdin), mode);
+ }
+ b->fp->owned_file = 1;
+ return b;
+}
+
+int bcf_close(bcf_t *b)
+{
+ int ret;
+ if (b == 0) return 0;
+ ret = bgzf_close(b->fp);
+ free(b);
+ return ret;
+}
+
+int bcf_hdr_write(bcf_t *b, const bcf_hdr_t *h)
+{
+ if (b == 0 || h == 0) return -1;
+ bgzf_write(b->fp, "BCF\4", 4);
+ bgzf_write(b->fp, &h->l_nm, 4);
+ bgzf_write(b->fp, h->name, h->l_nm);
+ bgzf_write(b->fp, &h->l_smpl, 4);
+ bgzf_write(b->fp, h->sname, h->l_smpl);
+ bgzf_write(b->fp, &h->l_txt, 4);
+ bgzf_write(b->fp, h->txt, h->l_txt);
+ bgzf_flush(b->fp);
+ return 16 + h->l_nm + h->l_smpl + h->l_txt;
+}
+
+bcf_hdr_t *bcf_hdr_read(bcf_t *b)
+{
+ uint8_t magic[4];
+ bcf_hdr_t *h;
+ if (b == 0) return 0;
+ h = calloc(1, sizeof(bcf_hdr_t));
+ bgzf_read(b->fp, magic, 4);
+ bgzf_read(b->fp, &h->l_nm, 4);
+ h->name = malloc(h->l_nm);
+ bgzf_read(b->fp, h->name, h->l_nm);
+ bgzf_read(b->fp, &h->l_smpl, 4);
+ h->sname = malloc(h->l_smpl);
+ bgzf_read(b->fp, h->sname, h->l_smpl);
+ bgzf_read(b->fp, &h->l_txt, 4);
+ h->txt = malloc(h->l_txt);
+ bgzf_read(b->fp, h->txt, h->l_txt);
+ bcf_hdr_sync(h);
+ return h;
+}
+
+void bcf_hdr_destroy(bcf_hdr_t *h)
+{
+ if (h == 0) return;
+ free(h->name); free(h->sname); free(h->txt); free(h->ns); free(h->sns);
+ free(h);
+}
+
+static inline char **cnt_null(int l, char *str, int *_n)
+{
+ int n = 0;
+ char *p, **list;
+ *_n = 0;
+ if (l == 0 || str == 0) return 0;
+ for (p = str; p != str + l; ++p)
+ if (*p == 0) ++n;
+ *_n = n;
+ list = calloc(n, sizeof(void*));
+ list[0] = str;
+ for (p = str, n = 1; p < str + l - 1; ++p)
+ if (*p == 0) list[n++] = p + 1;
+ return list;
+}
+
+int bcf_hdr_sync(bcf_hdr_t *b)
+{
+ if (b == 0) return -1;
+ if (b->ns) free(b->ns);
+ if (b->sns) free(b->sns);
+ if (b->l_nm) b->ns = cnt_null(b->l_nm, b->name, &b->n_ref);
+ else b->ns = 0, b->n_ref = 0;
+ b->sns = cnt_null(b->l_smpl, b->sname, &b->n_smpl);
+ return 0;
+}
+
+int bcf_sync(bcf1_t *b)
+{
+ char *p, *tmp[5];
+ int i, n, n_smpl = b->n_smpl;
+ ks_tokaux_t aux;
+ // set ref, alt, flt, info, fmt
+ b->ref = b->alt = b->flt = b->info = b->fmt = 0;
+ for (p = b->str, n = 0; p < b->str + b->l_str; ++p)
+ if (*p == 0 && p+1 != b->str + b->l_str) tmp[n++] = p + 1;
+ if (n != 5) {
+ fprintf(stderr, "[%s] incorrect number of fields (%d != 5). Corrupted file?\n", __func__, n);
+ return -1;
+ }
+ b->ref = tmp[0]; b->alt = tmp[1]; b->flt = tmp[2]; b->info = tmp[3]; b->fmt = tmp[4];
+ // set n_alleles
+ if (*b->alt == 0) b->n_alleles = 1;
+ else {
+ for (p = b->alt, n = 1; *p; ++p)
+ if (*p == ',') ++n;
+ b->n_alleles = n + 1;
+ }
+ // set n_gi and gi[i].fmt
+ for (p = b->fmt, n = 1; *p; ++p)
+ if (*p == ':') ++n;
+ if (n > b->m_gi) {
+ int old_m = b->m_gi;
+ b->m_gi = n;
+ kroundup32(b->m_gi);
+ b->gi = realloc(b->gi, b->m_gi * sizeof(bcf_ginfo_t));
+ memset(b->gi + old_m, 0, (b->m_gi - old_m) * sizeof(bcf_ginfo_t));
+ }
+ b->n_gi = n;
+ for (p = kstrtok(b->fmt, ":", &aux), n = 0; p; p = kstrtok(0, 0, &aux))
+ b->gi[n++].fmt = bcf_str2int(p, aux.p - p);
+ // set gi[i].len
+ for (i = 0; i < b->n_gi; ++i) {
+ if (b->gi[i].fmt == bcf_str2int("PL", 2)) {
+ b->gi[i].len = b->n_alleles * (b->n_alleles + 1) / 2;
+ } else if (b->gi[i].fmt == bcf_str2int("DP", 2) || b->gi[i].fmt == bcf_str2int("HQ", 2)) {
+ b->gi[i].len = 2;
+ } else if (b->gi[i].fmt == bcf_str2int("GQ", 2) || b->gi[i].fmt == bcf_str2int("GT", 2)) {
+ b->gi[i].len = 1;
+ } else if (b->gi[i].fmt == bcf_str2int("GL", 2)) {
+ b->gi[i].len = b->n_alleles * (b->n_alleles + 1) / 2 * 4;
+ }
+ b->gi[i].data = realloc(b->gi[i].data, n_smpl * b->gi[i].len);
+ }
+ return 0;
+}
+
+int bcf_write(bcf_t *bp, const bcf_hdr_t *h, const bcf1_t *b)
+{
+ int i, l = 0;
+ if (b == 0) return -1;
+ bgzf_write(bp->fp, &b->tid, 4);
+ bgzf_write(bp->fp, &b->pos, 4);
+ bgzf_write(bp->fp, &b->qual, 4);
+ bgzf_write(bp->fp, &b->l_str, 4);
+ bgzf_write(bp->fp, b->str, b->l_str);
+ l = 12 + b->l_str;
+ for (i = 0; i < b->n_gi; ++i) {
+ bgzf_write(bp->fp, b->gi[i].data, b->gi[i].len * h->n_smpl);
+ l += b->gi[i].len * h->n_smpl;
+ }
+ return l;
+}
+
+int bcf_read(bcf_t *bp, const bcf_hdr_t *h, bcf1_t *b)
+{
+ int i, l = 0;
+ if (b == 0) return -1;
+ if (bgzf_read(bp->fp, &b->tid, 4) == 0) return -1;
+ b->n_smpl = h->n_smpl;
+ bgzf_read(bp->fp, &b->pos, 4);
+ bgzf_read(bp->fp, &b->qual, 4);
+ bgzf_read(bp->fp, &b->l_str, 4);
+ if (b->l_str > b->m_str) {
+ b->m_str = b->l_str;
+ kroundup32(b->m_str);
+ b->str = realloc(b->str, b->m_str);
+ }
+ bgzf_read(bp->fp, b->str, b->l_str);
+ l = 12 + b->l_str;
+ bcf_sync(b);
+ for (i = 0; i < b->n_gi; ++i) {
+ bgzf_read(bp->fp, b->gi[i].data, b->gi[i].len * h->n_smpl);
+ l += b->gi[i].len * h->n_smpl;
+ }
+ return l;
+}
+
+int bcf_destroy(bcf1_t *b)
+{
+ int i;
+ if (b == 0) return -1;
+ free(b->str);
+ for (i = 0; i < b->m_gi; ++i)
+ free(b->gi[i].data);
+ free(b->gi);
+ free(b);
+ return 0;
+}
+
+static inline void fmt_str(const char *p, kstring_t *s)
+{
+ if (*p == 0) kputc('.', s);
+ else kputs(p, s);
+}
+
+void bcf_fmt_core(const bcf_hdr_t *h, bcf1_t *b, kstring_t *s)
+{
+ int i, j, x;
+ s->l = 0;
+ if (h->n_ref) kputs(h->ns[b->tid], s);
+ else kputw(b->tid, s);
+ kputc('\t', s);
+ kputw(b->pos + 1, s); kputc('\t', s);
+ fmt_str(b->str, s); kputc('\t', s);
+ fmt_str(b->ref, s); kputc('\t', s);
+ fmt_str(b->alt, s); kputc('\t', s);
+ ksprintf(s, "%.3g", b->qual); kputc('\t', s);
+ fmt_str(b->flt, s); kputc('\t', s);
+ fmt_str(b->info, s);
+ if (b->fmt[0]) {
+ kputc('\t', s);
+ fmt_str(b->fmt, s);
+ }
+ x = b->n_alleles * (b->n_alleles + 1) / 2;
+ if (b->n_gi == 0) return;
+ for (j = 0; j < h->n_smpl; ++j) {
+ kputc('\t', s);
+ for (i = 0; i < b->n_gi; ++i) {
+ if (i) kputc(':', s);
+ if (b->gi[i].fmt == bcf_str2int("PL", 2)) {
+ uint8_t *d = (uint8_t*)b->gi[i].data + j * x;
+ int k;
+ for (k = 0; k < x; ++k) {
+ if (k > 0) kputc(',', s);
+ kputw(d[k], s);
+ }
+ } else if (b->gi[i].fmt == bcf_str2int("DP", 2)) {
+ kputw(((uint16_t*)b->gi[i].data)[j], s);
+ } else if (b->gi[i].fmt == bcf_str2int("GQ", 2)) {
+ kputw(((uint8_t*)b->gi[i].data)[j], s);
+ } else if (b->gi[i].fmt == bcf_str2int("GT", 2)) {
+ int y = ((uint8_t*)b->gi[i].data)[j];
+ if (y>>7&1) {
+ kputsn("./.", 3, s);
+ } else {
+ kputc('0' + (y>>3&7), s);
+ kputc("/|"[y>>6&1], s);
+ kputc('0' + (y&7), s);
+ }
+ } else if (b->gi[i].fmt == bcf_str2int("GL", 2)) {
+ float *d = (float*)b->gi[i].data + j * x;
+ int k;
+ //printf("- %lx\n", d);
+ for (k = 0; k < x; ++k) {
+ if (k > 0) kputc(',', s);
+ ksprintf(s, "%.2f", d[k]);
+ }
+ }
+ }
+ }
+}
+
+char *bcf_fmt(const bcf_hdr_t *h, bcf1_t *b)
+{
+ kstring_t s;
+ s.l = s.m = 0; s.s = 0;
+ bcf_fmt_core(h, b, &s);
+ return s.s;
+}
+
+int bcf_append_info(bcf1_t *b, const char *info, int l)
+{
+ int shift = b->fmt - b->str;
+ int l_fmt = b->l_str - shift;
+ char *ori = b->str;
+ if (b->l_str + l > b->m_str) { // enlarge if necessary
+ b->m_str = b->l_str + l;
+ kroundup32(b->m_str);
+ b->str = realloc(b->str, b->m_str);
+ }
+ memmove(b->str + shift + l, b->str + shift, l_fmt); // move the FORMAT field
+ memcpy(b->str + shift - 1, info, l); // append to the INFO field
+ b->str[shift + l - 1] = '\0';
+ b->fmt = b->str + shift + l;
+ b->l_str += l;
+ if (ori != b->str) bcf_sync(b); // synchronize when realloc changes the pointer
+ return 0;
+}
+
+int bcf_cpy(bcf1_t *r, const bcf1_t *b)
+{
+ char *t1 = r->str;
+ bcf_ginfo_t *t2 = r->gi;
+ int i, t3 = r->m_str, t4 = r->m_gi;
+ *r = *b;
+ r->str = t1; r->gi = t2; r->m_str = t3; r->m_gi = t4;
+ if (r->m_str < b->m_str) {
+ r->m_str = b->m_str;
+ r->str = realloc(r->str, r->m_str);
+ }
+ memcpy(r->str, b->str, r->m_str);
+ bcf_sync(r); // calling bcf_sync() is simple but inefficient
+ for (i = 0; i < r->n_gi; ++i)
+ memcpy(r->gi[i].data, b->gi[i].data, r->n_smpl * r->gi[i].len);
+ return 0;
+}
--- /dev/null
+/* The MIT License
+
+ Copyright (c) 2010 Broad Institute
+
+ Permission is hereby granted, free of charge, to any person obtaining
+ a copy of this software and associated documentation files (the
+ "Software"), to deal in the Software without restriction, including
+ without limitation the rights to use, copy, modify, merge, publish,
+ distribute, sublicense, and/or sell copies of the Software, and to
+ permit persons to whom the Software is furnished to do so, subject to
+ the following conditions:
+
+ The above copyright notice and this permission notice shall be
+ included in all copies or substantial portions of the Software.
+
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ SOFTWARE.
+*/
+
+/* Contact: Heng Li <lh3@live.co.uk> */
+
+#ifndef BCF_H
+#define BCF_H
+
+#include <stdint.h>
+#include <zlib.h>
+#include "bgzf.h"
+
+/*
+ A member in the structs below is said to "primary" if its content
+ cannot be inferred from other members in any of structs below; a
+ member is said to be "derived" if its content can be derived from
+ other members. For example, bcf1_t::str is primary as this comes from
+ the input data, while bcf1_t::info is derived as it can always be
+ correctly set if we know bcf1_t::str. Derived members are for quick
+ access to the content and must be synchronized with the primary data.
+ */
+
+typedef struct {
+ uint32_t fmt; // format of the block, set by bcf_str2int().
+ int len; // length of data for each individual
+ void *data; // concatenated data
+ // derived info: fmt, len (<-bcf1_t::fmt)
+} bcf_ginfo_t;
+
+typedef struct {
+ int32_t tid, pos; // refID and 0-based position
+ int32_t l_str, m_str; // length and the allocated size of ->str
+ float qual; // SNP quality
+ char *str; // concatenated string of variable length strings in VCF (from col.2 to col.7)
+ char *ref, *alt, *flt, *info, *fmt; // they all point to ->str; no memory allocation
+ int n_gi, m_gi; // number and the allocated size of geno fields
+ bcf_ginfo_t *gi; // array of geno fields
+ int n_alleles, n_smpl; // number of alleles and samples
+ // derived info: ref, alt, flt, info, fmt (<-str), n_gi (<-fmt), n_alleles (<-alt), n_smpl (<-bcf_hdr_t::n_smpl)
+} bcf1_t;
+
+typedef struct {
+ int32_t n_ref, n_smpl; // number of reference sequences and samples
+ int32_t l_nm; // length of concatenated sequence names; 0 padded
+ int32_t l_smpl; // length of concatenated sample names; 0 padded
+ int32_t l_txt; // length of header text (lines started with ##)
+ char *name, *sname, *txt; // concatenated sequence names, sample names and header text
+ char **ns, **sns; // array of sequence and sample names; point to name and sname, respectively
+ // derived info: n_ref (<-name), n_smpl (<-sname), ns (<-name), sns (<-sname)
+} bcf_hdr_t;
+
+typedef struct {
+ int is_vcf; // if the file in operation is a VCF
+ void *v; // auxillary data structure for VCF
+ BGZF *fp; // file handler for BCF
+} bcf_t;
+
+struct __bcf_idx_t;
+typedef struct __bcf_idx_t bcf_idx_t;
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+ // open a BCF file; for BCF file only
+ bcf_t *bcf_open(const char *fn, const char *mode);
+ // close file
+ int bcf_close(bcf_t *b);
+ // read one record from BCF; return -1 on end-of-file, and <-1 for errors
+ int bcf_read(bcf_t *bp, const bcf_hdr_t *h, bcf1_t *b);
+ // call this function if b->str is changed
+ int bcf_sync(bcf1_t *b);
+ // write a BCF record
+ int bcf_write(bcf_t *bp, const bcf_hdr_t *h, const bcf1_t *b);
+ // read the BCF header; BCF only
+ bcf_hdr_t *bcf_hdr_read(bcf_t *b);
+ // write the BCF header
+ int bcf_hdr_write(bcf_t *b, const bcf_hdr_t *h);
+ // set bcf_hdr_t::ns and bcf_hdr_t::sns
+ int bcf_hdr_sync(bcf_hdr_t *b);
+ // destroy the header
+ void bcf_hdr_destroy(bcf_hdr_t *h);
+ // destroy a record
+ int bcf_destroy(bcf1_t *b);
+ // BCF->VCF conversion
+ char *bcf_fmt(const bcf_hdr_t *h, bcf1_t *b);
+ // append more info
+ int bcf_append_info(bcf1_t *b, const char *info, int l);
+ // copy
+ int bcf_cpy(bcf1_t *r, const bcf1_t *b);
+
+ // open a VCF or BCF file if "b" is set in "mode"
+ bcf_t *vcf_open(const char *fn, const char *mode);
+ // close a VCF/BCF file
+ int vcf_close(bcf_t *bp);
+ // read the VCF/BCF header
+ bcf_hdr_t *vcf_hdr_read(bcf_t *bp);
+ // read a VCF/BCF record; return -1 on end-of-file and <-1 for errors
+ int vcf_read(bcf_t *bp, bcf_hdr_t *h, bcf1_t *b);
+ // write the VCF header
+ int vcf_hdr_write(bcf_t *bp, const bcf_hdr_t *h);
+ // write a VCF record
+ int vcf_write(bcf_t *bp, bcf_hdr_t *h, bcf1_t *b);
+
+ // keep the first n alleles and discard the rest
+ int bcf_shrink_alt(bcf1_t *b, int n);
+ // convert GL to PL
+ int bcf_gl2pl(bcf1_t *b);
+
+ // string hash table
+ void *bcf_build_refhash(bcf_hdr_t *h);
+ void bcf_str2id_destroy(void *_hash);
+ int bcf_str2id_add(void *_hash, const char *str);
+ int bcf_str2id(void *_hash, const char *str);
+ void *bcf_str2id_init();
+
+ // indexing related functions
+ int bcf_idx_build(const char *fn);
+ uint64_t bcf_idx_query(const bcf_idx_t *idx, int tid, int beg);
+ int bcf_parse_region(void *str2id, const char *str, int *tid, int *begin, int *end);
+ bcf_idx_t *bcf_idx_load(const char *fn);
+ void bcf_idx_destroy(bcf_idx_t *idx);
+
+#ifdef __cplusplus
+}
+#endif
+
+static inline uint32_t bcf_str2int(const char *str, int l)
+{
+ int i;
+ uint32_t x = 0;
+ for (i = 0; i < l && i < 4; ++i) {
+ if (str[i] == 0) return x;
+ x = x<<8 | str[i];
+ }
+ return x;
+}
+
+#endif
--- /dev/null
+\documentclass[10pt,pdftex]{article}
+\usepackage{color}
+\definecolor{gray}{rgb}{0.7,0.7,0.7}
+
+\setlength{\topmargin}{0.0cm}
+\setlength{\textheight}{21.5cm}
+\setlength{\oddsidemargin}{0cm}
+\setlength{\textwidth}{16.5cm}
+\setlength{\columnsep}{0.6cm}
+
+\begin{document}
+
+\begin{center}
+\begin{tabular}{|l|l|l|l|l|}
+\hline
+\multicolumn{2}{|c|}{\bf Field} & \multicolumn{1}{c|}{\bf Descrption} & \multicolumn{1}{c|}{\bf Type} & \multicolumn{1}{c|}{\bf Value} \\\hline\hline
+\multicolumn{2}{|l|}{\tt magic} & Magic string & {\tt char[4]} & {\tt BCF\char92 4} \\\hline
+\multicolumn{2}{|l|}{\tt l\_nm} & Length of concatenated sequence names & {\tt int32\_t} & \\\hline
+\multicolumn{2}{|l|}{\tt name} & Concatenated names, {\tt NULL} padded & {\tt char[l\_nm]} & \\\hline
+\multicolumn{2}{|l|}{\tt l\_smpl} & Length of concatenated sample names & {\tt int32\_t} & \\\hline
+\multicolumn{2}{|l|}{\tt sname} & Concatenated sample names & {\tt char[l\_smpl]} & \\\hline
+\multicolumn{2}{|l|}{\tt l\_txt} & Length of the meta text (double-hash lines)& {\tt int32\_t} & \\\hline
+\multicolumn{2}{|l|}{\tt text} & Meta text, {\tt NULL} terminated & {\tt char[l\_txt]} & \\\hline
+\multicolumn{5}{|c|}{\it \color{gray}{List of records until the end of the file}}\\\cline{2-5}
+& {\tt seq\_id} & Reference sequence ID & {\tt int32\_t} & \\\cline{2-5}
+& {\tt pos} & Position & {\tt int32\_t} & \\\cline{2-5}
+& {\tt qual} & Variant quality & {\tt float} & \\\cline{2-5}
+& {\tt l\_str} & Length of str & {\tt int32\_t} & \\\cline{2-5}
+& {\tt str} & {\tt ID+REF+ALT+FILTER+INFO+FORMAT}, {\tt NULL} padded & {\tt char[slen]} &\\\cline{2-5}
+& \multicolumn{4}{c|}{Blocks of data; \#blocks and formats defined by {\tt FORMAT} (table below)}\\
+\hline
+\end{tabular}
+\end{center}
+
+\begin{center}
+\begin{tabular}{cll}
+\hline
+\multicolumn{1}{l}{\bf Field} & \multicolumn{1}{l}{\bf Type} & \multicolumn{1}{l}{\bf Description} \\\hline
+{\tt DP} & {\tt uint16\_t[n]} & Read depth \\
+{\tt GL} & {\tt float[n*x]} & Log10 likelihood of data; $x=\frac{m(m+1)}{2}$, $m=\#\{alleles\}$\\
+{\tt GT} & {\tt uint8\_t[n]} & {\tt phase\char60\char60 6 | allele1\char60\char60 3 | allele2} \\
+{\tt GQ} & {\tt uint8\_t[n]} & {Genotype quality}\\
+{\tt HQ} & {\tt uint8\_t[n*2]} & {Haplotype quality}\\
+{\tt PL} & {\tt uint8\_t[n*x]} & {Phred-scaled likelihood of data}\\
+\emph{misc} & {\tt int32\_t+char*} & {\tt NULL} padded concatenated strings (integer equal to the length) \\
+\hline
+\end{tabular}
+\end{center}
+
+\begin{itemize}
+\item The file is {\tt BGZF} compressed.
+\item All integers are little-endian.
+\item In a string, a missing value `.' is an empty C string ``{\tt
+ \char92 0}'' (not ``{\tt .\char92 0}'')
+\item For {\tt GL} and {\tt PL}, likelihoods of genotypes appear in the
+ order of alleles in {\tt REF} and then {\tt ALT}. For example, if {\tt
+ REF=C}, {\tt ALT=T,A}, likelihoods appear in the order of {\tt
+ CC,CT,CA,TT,TA,AA}.
+\item {\tt GL} is an extension to and is backward compatible with the
+ {\tt GL} genotype field in {\tt VCFv4.0}.
+\end{itemize}
+
+\end{document}
\ No newline at end of file
--- /dev/null
+#include <errno.h>
+#include <math.h>
+#include <string.h>
+#include <stdlib.h>
+#include "bcf.h"
+
+static int8_t nt4_table[256] = {
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4 /*'-'*/, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 0, 4, 1, 4, 4, 4, 2, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 3, 4, 4, 4, -1, 4, 4, 4, 4, 4, 4, 4,
+ 4, 0, 4, 1, 4, 4, 4, 2, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 3, 4, 4, 4, -1, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4
+};
+
+static int read_I16(bcf1_t *b, int anno[16])
+{
+ char *p;
+ int i;
+ if ((p = strstr(b->info, "I16=")) == 0) return -1;
+ p += 4;
+ for (i = 0; i < 16; ++i) {
+ anno[i] = strtol(p, &p, 10);
+ if (anno[i] == 0 && (errno == EINVAL || errno == ERANGE)) return -2;
+ ++p;
+ }
+ return 0;
+}
+
+int bcf_2qcall(bcf_hdr_t *h, bcf1_t *b)
+{
+ int a[4], k, g[10], l, map[4], k1, j, i, i0, anno[16], dp, mq, d_rest;
+ char *s;
+ if (b->ref[1] != 0 || b->n_alleles > 4) return -1; // ref is not a single base
+ for (i = 0; i < b->n_gi; ++i)
+ if (b->gi[i].fmt == bcf_str2int("PL", 2)) break;
+ if (i == b->n_gi) return -1; // no PL
+ if (read_I16(b, anno) != 0) return -1; // no I16; FIXME: can be improved
+ d_rest = dp = anno[0] + anno[1] + anno[2] + anno[3];
+ if (dp == 0) return -1; // depth is zero
+ mq = (int)(sqrt((double)(anno[9] + anno[11]) / dp) + .499);
+ i0 = i;
+ a[0] = nt4_table[(int)b->ref[0]];
+ if (a[0] > 3) return -1; // ref is not A/C/G/T
+ a[1] = a[2] = a[3] = -2; // -1 has a special meaning
+ if (b->alt[0] == 0) return -1; // no alternate allele
+ map[0] = map[1] = map[2] = map[3] = -2;
+ map[a[0]] = 0;
+ for (k = 0, s = b->alt, k1 = -1; k < 3 && *s; ++k, s += 2) {
+ if (s[1] != ',' && s[1] != 0) return -1; // ALT is not single base
+ a[k+1] = nt4_table[(int)*s];
+ if (a[k+1] >= 0) map[a[k+1]] = k+1;
+ else k1 = k+1;
+ if (s[1] == 0) break;
+ }
+ for (k = 0; k < 4; ++k)
+ if (map[k] < 0) map[k] = k1;
+ for (i = 0; i < h->n_smpl; ++i) {
+ int d;
+ uint8_t *p = b->gi[i0].data + i * b->gi[i0].len;
+ for (j = 0; j < b->gi[i0].len; ++j)
+ if (p[j]) break;
+ d = (int)((double)d_rest / (h->n_smpl - i) + .499);
+ if (d == 0) d = 1;
+ if (j == b->gi[i0].len) d = 0;
+ d_rest -= d;
+ for (k = j = 0; k < 4; ++k) {
+ for (l = k; l < 4; ++l) {
+ int t, x = map[k], y = map[l];
+ if (x > y) t = x, x = y, y = t;
+ g[j++] = p[x * b->n_alleles - x * (x-1) / 2 + (y - x)];
+ }
+ }
+ printf("%s\t%d\t%c", h->ns[b->tid], b->pos+1, *b->ref);
+ printf("\t%d\t%d\t0", d, mq);
+ for (j = 0; j < 10; ++j)
+ printf("\t%d", g[j]);
+ printf("\t%s\n", h->sns[i]);
+ }
+ return 0;
+}
--- /dev/null
+.TH bcftools 1 "2 October 2010" "bcftools" "Bioinformatics tools"
+.SH NAME
+.PP
+bcftools - Utilities for the Binary Call Format (BCF) and VCF.
+.SH SYNOPSIS
+.PP
+bcftools index in.bcf
+.PP
+bcftools view in.bcf chr2:100-200 > out.vcf
+.PP
+bcftools view -vc in.bcf > out.vcf 2> out.afs
+
+.SH DESCRIPTION
+.PP
+Bcftools is a toolkit for processing VCF/BCF files, calling variants and
+estimating site allele frequencies and allele frequency spectrums.
+
+.SH COMMANDS AND OPTIONS
+
+.TP 10
+.B view
+.B bcftools view
+.RB [ \-cbuSAGgHvNQ ]
+.RB [ \-1
+.IR nGroup1 ]
+.RB [ \-l
+.IR listFile ]
+.RB [ \-t
+.IR mutRate ]
+.RB [ \-p
+.IR varThres ]
+.RB [ \-P
+.IR prior ]
+.I in.bcf
+.RI [ region ]
+
+Convert between BCF and VCF, call variant candidates and estimate allele
+frequencies.
+
+.B OPTIONS:
+.RS
+.TP 10
+.B -b
+Output in the BCF format. The default is VCF.
+.TP
+.B -c
+Call variants.
+.TP
+.B -v
+Output variant sites only (force -c)
+.TP
+.B -g
+Call per-sample genotypes at variant sites (force -c)
+.TP
+.B -u
+Uncompressed BCF output (force -b).
+.TP
+.B -S
+The input is VCF instead of BCF.
+.TP
+.B -A
+Retain all possible alternate alleles at variant sites. By default, this
+command discards unlikely alleles.
+.TP
+.B -G
+Suppress all individual genotype information.
+.TP
+.B -H
+Perform Hardy-Weiberg Equilibrium test. This will add computation time, sometimes considerably.
+.TP
+.B -N
+Skip sites where the REF field is not A/C/G/T
+.TP
+.B -Q
+Output the QCALL likelihood format
+.TP
+.BI "-1 " INT
+Number of group-1 samples. This option is used for dividing input into
+two groups for comparing. A zero value disables this functionality. [0]
+.TP
+.BI "-l " FILE
+List of sites at which information are outputted [all sites]
+.TP
+.BI "-t " FLOAT
+Scaled muttion rate for variant calling [0.001]
+.TP
+.BI "-p " FLOAT
+A site is considered to be a variant if P(ref|D)<FLOAT [0.5]
+.TP
+.BI "-P " STR
+Prior or initial allele frequency spectrum. If STR can be
+.IR full ,
+.IR cond2 ,
+.I flat
+or the file consisting of error output from a previous variant calling
+run.
+.RE
+
+.TP
+.B index
+.B bcftools index
+.I in.bcf
+
+Index sorted BCF for random access.
+.RE
+
+.TP
+.B cat
+.B bcftools cat
+.I in1.bcf
+.RI [ "in2.bcf " [ ... "]]]"
+
+Concatenate BCF files. The input files are required to be sorted and
+have identical samples appearing in the same order.
+.RE
--- /dev/null
+#include "bcf.h"
+#include "kstring.h"
+#include "khash.h"
+KHASH_MAP_INIT_STR(str2id, int)
+
+void *bcf_build_refhash(bcf_hdr_t *h)
+{
+ khash_t(str2id) *hash;
+ int i, ret;
+ hash = kh_init(str2id);
+ for (i = 0; i < h->n_ref; ++i) {
+ khint_t k;
+ k = kh_put(str2id, hash, h->ns[i], &ret); // FIXME: check ret
+ kh_val(hash, k) = i;
+ }
+ return hash;
+}
+
+void *bcf_str2id_init()
+{
+ return kh_init(str2id);
+}
+
+void bcf_str2id_destroy(void *_hash)
+{
+ khash_t(str2id) *hash = (khash_t(str2id)*)_hash;
+ if (hash) kh_destroy(str2id, hash); // Note that strings are not freed.
+}
+
+int bcf_str2id(void *_hash, const char *str)
+{
+ khash_t(str2id) *hash = (khash_t(str2id)*)_hash;
+ khint_t k;
+ if (!hash) return -1;
+ k = kh_get(str2id, hash, str);
+ return k == kh_end(hash)? -1 : kh_val(hash, k);
+}
+
+int bcf_str2id_add(void *_hash, const char *str)
+{
+ khint_t k;
+ int ret;
+ khash_t(str2id) *hash = (khash_t(str2id)*)_hash;
+ if (!hash) return -1;
+ k = kh_put(str2id, hash, str, &ret);
+ if (ret == 0) return kh_val(hash, k);
+ kh_val(hash, k) = kh_size(hash) - 1;
+ return kh_val(hash, k);
+}
+
+int bcf_shrink_alt(bcf1_t *b, int n)
+{
+ char *p;
+ int i, j, k, *z, n_smpl = b->n_smpl;
+ if (b->n_alleles <= n) return -1;
+ if (n > 1) {
+ for (p = b->alt, k = 1; *p; ++p)
+ if (*p == ',' && ++k == n) break;
+ *p = '\0';
+ } else p = b->alt, *p = '\0';
+ ++p;
+ memmove(p, b->flt, b->str + b->l_str - b->flt);
+ b->l_str -= b->flt - p;
+ z = alloca(sizeof(int) / 2 * n * (n+1));
+ for (i = k = 0; i < n; ++i)
+ for (j = 0; j < n - i; ++j)
+ z[k++] = i * b->n_alleles + j;
+ for (i = 0; i < b->n_gi; ++i) {
+ bcf_ginfo_t *g = b->gi + i;
+ if (g->fmt == bcf_str2int("PL", 2)) {
+ int l, x = b->n_alleles * (b->n_alleles + 1) / 2;
+ uint8_t *d = (uint8_t*)g->data;
+ g->len = n * (n + 1) / 2;
+ for (l = k = 0; l < n_smpl; ++l) {
+ uint8_t *dl = d + l * x;
+ for (j = 0; j < g->len; ++j) d[k++] = dl[z[j]];
+ }
+ } // FIXME: to add GL
+ }
+ b->n_alleles = n;
+ bcf_sync(b);
+ return 0;
+}
+
+int bcf_gl2pl(bcf1_t *b)
+{
+ char *p;
+ int i, n_smpl = b->n_smpl;
+ bcf_ginfo_t *g;
+ float *d0;
+ uint8_t *d1;
+ if (strstr(b->fmt, "PL")) return -1;
+ if ((p = strstr(b->fmt, "GL")) == 0) return -1;
+ *p = 'P';
+ for (i = 0; i < b->n_gi; ++i)
+ if (b->gi[i].fmt == bcf_str2int("GL", 2))
+ break;
+ g = b->gi + i;
+ g->fmt = bcf_str2int("PL", 2);
+ g->len /= 4; // 4 == sizeof(float)
+ d0 = (float*)g->data; d1 = (uint8_t*)g->data;
+ for (i = 0; i < n_smpl * g->len; ++i) {
+ int x = (int)(-10. * d0[i] + .499);
+ if (x > 255) x = 255;
+ if (x < 0) x = 0;
+ d1[i] = x;
+ }
+ return 0;
+}
--- /dev/null
+#include <unistd.h>
+#include <stdlib.h>
+#include <math.h>
+#include <zlib.h>
+#include <errno.h>
+#include "bcf.h"
+#include "prob1.h"
+#include "kstring.h"
+
+#include "khash.h"
+KHASH_SET_INIT_INT64(set64)
+
+#include "kseq.h"
+KSTREAM_INIT(gzFile, gzread, 16384)
+
+#define VC_NO_GENO 2
+#define VC_BCFOUT 4
+#define VC_CALL 8
+#define VC_VARONLY 16
+#define VC_VCFIN 32
+#define VC_UNCOMP 64
+#define VC_HWE 128
+#define VC_KEEPALT 256
+#define VC_ACGT_ONLY 512
+#define VC_QCALL 1024
+#define VC_CALL_GT 2048
+#define VC_ADJLD 4096
+
+typedef struct {
+ int flag, prior_type, n1;
+ char *fn_list, *prior_file;
+ double theta, pref;
+} viewconf_t;
+
+khash_t(set64) *bcf_load_pos(const char *fn, bcf_hdr_t *_h)
+{
+ void *str2id;
+ gzFile fp;
+ kstream_t *ks;
+ int ret, dret, lineno = 1;
+ kstring_t *str;
+ khash_t(set64) *hash = 0;
+
+ hash = kh_init(set64);
+ str2id = bcf_build_refhash(_h);
+ str = calloc(1, sizeof(kstring_t));
+ fp = strcmp(fn, "-")? gzopen(fn, "r") : gzdopen(fileno(stdin), "r");
+ ks = ks_init(fp);
+ while (ks_getuntil(ks, 0, str, &dret) >= 0) {
+ int tid = bcf_str2id(str2id, str->s);
+ if (tid >= 0 && dret != '\n') {
+ if (ks_getuntil(ks, 0, str, &dret) >= 0) {
+ uint64_t x = (uint64_t)tid<<32 | (atoi(str->s) - 1);
+ kh_put(set64, hash, x, &ret);
+ } else break;
+ } else fprintf(stderr, "[%s] %s is not a reference name (line %d).\n", __func__, str->s, lineno);
+ if (dret != '\n') while ((dret = ks_getc(ks)) > 0 && dret != '\n');
+ if (dret < 0) break;
+ ++lineno;
+ }
+ bcf_str2id_destroy(str2id);
+ ks_destroy(ks);
+ gzclose(fp);
+ free(str->s); free(str);
+ return hash;
+}
+
+static double test_hwe(const double g[3])
+{
+ extern double kf_gammaq(double p, double x);
+ double fexp, chi2, f[3], n;
+ int i;
+ n = g[0] + g[1] + g[2];
+ fexp = (2. * g[2] + g[1]) / (2. * n);
+ if (fexp > 1. - 1e-10) fexp = 1. - 1e-10;
+ if (fexp < 1e-10) fexp = 1e-10;
+ f[0] = n * (1. - fexp) * (1. - fexp);
+ f[1] = n * 2. * fexp * (1. - fexp);
+ f[2] = n * fexp * fexp;
+ for (i = 0, chi2 = 0.; i < 3; ++i)
+ chi2 += (g[i] - f[i]) * (g[i] - f[i]) / f[i];
+ return kf_gammaq(.5, chi2 / 2.);
+}
+
+typedef struct {
+ double p[4];
+ int mq, depth, is_tested, d[4];
+} anno16_t;
+
+static double ttest(int n1, int n2, int a[4])
+{
+ extern double kf_betai(double a, double b, double x);
+ double t, v, u1, u2;
+ if (n1 == 0 || n2 == 0 || n1 + n2 < 3) return 1.0;
+ u1 = (double)a[0] / n1; u2 = (double)a[2] / n2;
+ if (u1 <= u2) return 1.;
+ t = (u1 - u2) / sqrt(((a[1] - n1 * u1 * u1) + (a[3] - n2 * u2 * u2)) / (n1 + n2 - 2) * (1./n1 + 1./n2));
+ v = n1 + n2 - 2;
+// printf("%d,%d,%d,%d,%lf,%lf,%lf\n", a[0], a[1], a[2], a[3], t, u1, u2);
+ return t < 0.? 1. : .5 * kf_betai(.5*v, .5, v/(v+t*t));
+}
+
+static int test16_core(int anno[16], anno16_t *a)
+{
+ extern double kt_fisher_exact(int n11, int n12, int n21, int n22, double *_left, double *_right, double *two);
+ double left, right;
+ int i;
+ a->p[0] = a->p[1] = a->p[2] = a->p[3] = 1.;
+ memcpy(a->d, anno, 4 * sizeof(int));
+ a->depth = anno[0] + anno[1] + anno[2] + anno[3];
+ a->is_tested = (anno[0] + anno[1] > 0 && anno[2] + anno[3] > 0);
+ if (a->depth == 0) return -1;
+ a->mq = (int)(sqrt((anno[9] + anno[11]) / a->depth) + .499);
+ kt_fisher_exact(anno[0], anno[1], anno[2], anno[3], &left, &right, &a->p[0]);
+ for (i = 1; i < 4; ++i)
+ a->p[i] = ttest(anno[0] + anno[1], anno[2] + anno[3], anno+4*i);
+ return 0;
+}
+
+static int test16(bcf1_t *b, anno16_t *a)
+{
+ char *p;
+ int i, anno[16];
+ a->p[0] = a->p[1] = a->p[2] = a->p[3] = 1.;
+ a->d[0] = a->d[1] = a->d[2] = a->d[3] = 0.;
+ a->mq = a->depth = a->is_tested = 0;
+ if ((p = strstr(b->info, "I16=")) == 0) return -1;
+ p += 4;
+ for (i = 0; i < 16; ++i) {
+ anno[i] = strtol(p, &p, 10);
+ if (anno[i] == 0 && (errno == EINVAL || errno == ERANGE)) return -2;
+ ++p;
+ }
+ return test16_core(anno, a);
+}
+
+static void rm_info(bcf1_t *b, const char *key)
+{
+ char *p, *q;
+ if ((p = strstr(b->info, key)) == 0) return;
+ for (q = p; *q && *q != ';'; ++q);
+ if (p > b->info && *(p-1) == ';') --p;
+ memmove(p, q, b->l_str - (q - b->str));
+ b->l_str -= q - p;
+ bcf_sync(b);
+}
+
+static int update_bcf1(int n_smpl, bcf1_t *b, const bcf_p1aux_t *pa, const bcf_p1rst_t *pr, double pref, int flag)
+{
+ kstring_t s;
+ int is_var = (pr->p_ref < pref);
+ double p_hwe, r = is_var? pr->p_ref : 1. - pr->p_ref;
+ anno16_t a;
+
+ p_hwe = pr->g[0] >= 0.? test_hwe(pr->g) : 1.0; // only do HWE g[] is calculated
+ test16(b, &a);
+ rm_info(b, "I16=");
+
+ memset(&s, 0, sizeof(kstring_t));
+ kputc('\0', &s); kputs(b->ref, &s); kputc('\0', &s);
+ kputs(b->alt, &s); kputc('\0', &s); kputc('\0', &s);
+ kputs(b->info, &s);
+ if (b->info[0]) kputc(';', &s);
+ ksprintf(&s, "AF1=%.3lf;AFE=%.3lf", 1.-pr->f_em, 1.-pr->f_exp);
+ ksprintf(&s, ";DP4=%d,%d,%d,%d;MQ=%d", a.d[0], a.d[1], a.d[2], a.d[3], a.mq);
+ if (a.is_tested) {
+ if (pr->pc[0] >= 0.) ksprintf(&s, ";PC4=%lg,%lg,%lg,%lg", pr->pc[0], pr->pc[1], pr->pc[2], pr->pc[3]);
+ ksprintf(&s, ";PV4=%.2lg,%.2lg,%.2lg,%.2lg", a.p[0], a.p[1], a.p[2], a.p[3]);
+ }
+ if (pr->g[0] >= 0. && p_hwe <= .2)
+ ksprintf(&s, ";GC=%.2lf,%.2lf,%.2lf;HWE=%.3lf", pr->g[2], pr->g[1], pr->g[0], p_hwe);
+ kputc('\0', &s);
+ kputs(b->fmt, &s); kputc('\0', &s);
+ free(b->str);
+ b->m_str = s.m; b->l_str = s.l; b->str = s.s;
+ b->qual = r < 1e-100? 99 : -4.343 * log(r);
+ if (b->qual > 99) b->qual = 99;
+ bcf_sync(b);
+ if (!is_var) bcf_shrink_alt(b, 1);
+ else if (!(flag&VC_KEEPALT))
+ bcf_shrink_alt(b, pr->rank0 < 2? 2 : pr->rank0+1);
+ if (is_var && (flag&VC_CALL_GT)) { // call individual genotype
+ int i, x, old_n_gi = b->n_gi;
+ s.m = b->m_str; s.l = b->l_str - 1; s.s = b->str;
+ kputs(":GT:GQ", &s); kputc('\0', &s);
+ b->m_str = s.m; b->l_str = s.l; b->str = s.s;
+ bcf_sync(b);
+ for (i = 0; i < b->n_smpl; ++i) {
+ x = bcf_p1_call_gt(pa, pr->f_em, i);
+ ((uint8_t*)b->gi[old_n_gi].data)[i] = (x&3) == 0? 1<<3|1 : (x&3) == 1? 1 : 0;
+ ((uint8_t*)b->gi[old_n_gi+1].data)[i] = x>>2;
+ }
+ }
+ return is_var;
+}
+
+double bcf_ld_freq(const bcf1_t *b0, const bcf1_t *b1, double f[4]);
+
+int bcfview(int argc, char *argv[])
+{
+ extern int bcf_2qcall(bcf_hdr_t *h, bcf1_t *b);
+ bcf_t *bp, *bout = 0;
+ bcf1_t *b, *blast;
+ int c;
+ uint64_t n_processed = 0;
+ viewconf_t vc;
+ bcf_p1aux_t *p1 = 0;
+ bcf_hdr_t *h;
+ int tid, begin, end;
+ char moder[4], modew[4];
+ khash_t(set64) *hash = 0;
+
+ tid = begin = end = -1;
+ memset(&vc, 0, sizeof(viewconf_t));
+ vc.prior_type = vc.n1 = -1; vc.theta = 1e-3; vc.pref = 0.5;
+ while ((c = getopt(argc, argv, "N1:l:cHAGvbSuP:t:p:QgL")) >= 0) {
+ switch (c) {
+ case '1': vc.n1 = atoi(optarg); break;
+ case 'l': vc.fn_list = strdup(optarg); break;
+ case 'N': vc.flag |= VC_ACGT_ONLY; break;
+ case 'G': vc.flag |= VC_NO_GENO; break;
+ case 'A': vc.flag |= VC_KEEPALT; break;
+ case 'b': vc.flag |= VC_BCFOUT; break;
+ case 'S': vc.flag |= VC_VCFIN; break;
+ case 'c': vc.flag |= VC_CALL; break;
+ case 'v': vc.flag |= VC_VARONLY | VC_CALL; break;
+ case 'u': vc.flag |= VC_UNCOMP | VC_BCFOUT; break;
+ case 'H': vc.flag |= VC_HWE; break;
+ case 'g': vc.flag |= VC_CALL_GT | VC_CALL; break;
+ case 't': vc.theta = atof(optarg); break;
+ case 'p': vc.pref = atof(optarg); break;
+ case 'Q': vc.flag |= VC_QCALL; break;
+ case 'L': vc.flag |= VC_ADJLD; break;
+ case 'P':
+ if (strcmp(optarg, "full") == 0) vc.prior_type = MC_PTYPE_FULL;
+ else if (strcmp(optarg, "cond2") == 0) vc.prior_type = MC_PTYPE_COND2;
+ else if (strcmp(optarg, "flat") == 0) vc.prior_type = MC_PTYPE_FLAT;
+ else vc.prior_file = strdup(optarg);
+ break;
+ }
+ }
+ if (argc == optind) {
+ fprintf(stderr, "\n");
+ fprintf(stderr, "Usage: bcftools view [options] <in.bcf> [reg]\n\n");
+ fprintf(stderr, "Options: -c SNP calling\n");
+ fprintf(stderr, " -v output potential variant sites only (force -c)\n");
+ fprintf(stderr, " -g call genotypes at variant sites (force -c)\n");
+ fprintf(stderr, " -b output BCF instead of VCF\n");
+ fprintf(stderr, " -u uncompressed BCF output (force -b)\n");
+ fprintf(stderr, " -S input is VCF\n");
+ fprintf(stderr, " -A keep all possible alternate alleles at variant sites\n");
+ fprintf(stderr, " -G suppress all individual genotype information\n");
+ fprintf(stderr, " -H perform Hardy-Weinberg test (slower)\n");
+ fprintf(stderr, " -N skip sites where REF is not A/C/G/T\n");
+ fprintf(stderr, " -Q output the QCALL likelihood format\n");
+ fprintf(stderr, " -L calculate LD for adjacent sites\n");
+ fprintf(stderr, " -1 INT number of group-1 samples [0]\n");
+ fprintf(stderr, " -l FILE list of sites to output [all sites]\n");
+ fprintf(stderr, " -t FLOAT scaled mutation rate [%.4lg]\n", vc.theta);
+ fprintf(stderr, " -p FLOAT variant if P(ref|D)<FLOAT [%.3lg]\n", vc.pref);
+ fprintf(stderr, " -P STR type of prior: full, cond2, flat [full]\n");
+ fprintf(stderr, "\n");
+ return 1;
+ }
+
+ b = calloc(1, sizeof(bcf1_t));
+ blast = calloc(1, sizeof(bcf1_t));
+ strcpy(moder, "r");
+ if (!(vc.flag & VC_VCFIN)) strcat(moder, "b");
+ strcpy(modew, "w");
+ if (vc.flag & VC_BCFOUT) strcat(modew, "b");
+ if (vc.flag & VC_UNCOMP) strcat(modew, "u");
+ bp = vcf_open(argv[optind], moder);
+ h = vcf_hdr_read(bp);
+ bout = vcf_open("-", modew);
+ if (!(vc.flag & VC_QCALL)) vcf_hdr_write(bout, h);
+ if (vc.flag & VC_CALL) {
+ p1 = bcf_p1_init(h->n_smpl);
+ if (vc.prior_file) {
+ if (bcf_p1_read_prior(p1, vc.prior_file) < 0) {
+ fprintf(stderr, "[%s] fail to read the prior AFS.\n", __func__);
+ return 1;
+ }
+ } else bcf_p1_init_prior(p1, vc.prior_type, vc.theta);
+ if (vc.n1 > 0) {
+ bcf_p1_set_n1(p1, vc.n1);
+ bcf_p1_init_subprior(p1, vc.prior_type, vc.theta);
+ }
+ }
+ if (vc.fn_list) hash = bcf_load_pos(vc.fn_list, h);
+ if (optind + 1 < argc && !(vc.flag&VC_VCFIN)) {
+ void *str2id = bcf_build_refhash(h);
+ if (bcf_parse_region(str2id, argv[optind+1], &tid, &begin, &end) >= 0) {
+ bcf_idx_t *idx;
+ idx = bcf_idx_load(argv[optind]);
+ if (idx) {
+ uint64_t off;
+ off = bcf_idx_query(idx, tid, begin);
+ if (off == 0) {
+ fprintf(stderr, "[%s] no records in the query region.\n", __func__);
+ return 1; // FIXME: a lot of memory leaks...
+ }
+ bgzf_seek(bp->fp, off, SEEK_SET);
+ bcf_idx_destroy(idx);
+ }
+ }
+ }
+ while (vcf_read(bp, h, b) > 0) {
+ if (vc.flag & VC_ACGT_ONLY) {
+ int x;
+ if (b->ref[0] == 0 || b->ref[1] != 0) continue;
+ x = toupper(b->ref[0]);
+ if (x != 'A' && x != 'C' && x != 'G' && x != 'T') continue;
+ }
+ if (hash) {
+ uint64_t x = (uint64_t)b->tid<<32 | b->pos;
+ khint_t k = kh_get(set64, hash, x);
+ if (kh_size(hash) == 0) break;
+ if (k == kh_end(hash)) continue;
+ kh_del(set64, hash, k);
+ }
+ if (tid >= 0) {
+ int l = strlen(b->ref);
+ l = b->pos + (l > 0? l : 1);
+ if (b->tid != tid || b->pos >= end) break;
+ if (!(l > begin && end > b->pos)) continue;
+ }
+ ++n_processed;
+ if (vc.flag & VC_QCALL) { // output QCALL format; STOP here
+ bcf_2qcall(h, b);
+ continue;
+ }
+ if (vc.flag & (VC_CALL|VC_ADJLD)) bcf_gl2pl(b);
+ if (vc.flag & VC_CALL) { // call variants
+ bcf_p1rst_t pr;
+ bcf_p1_cal(b, p1, &pr); // pr.g[3] is not calculated here
+ if (vc.flag&VC_HWE) bcf_p1_cal_g3(p1, pr.g);
+ if (n_processed % 100000 == 0) {
+ fprintf(stderr, "[%s] %ld sites processed.\n", __func__, (long)n_processed);
+ bcf_p1_dump_afs(p1);
+ }
+ if (pr.p_ref >= vc.pref && (vc.flag & VC_VARONLY)) continue;
+ update_bcf1(h->n_smpl, b, p1, &pr, vc.pref, vc.flag);
+ }
+ if (vc.flag & VC_ADJLD) { // compute LD
+ double f[4], r2;
+ if ((r2 = bcf_ld_freq(blast, b, f)) >= 0) {
+ kstring_t s;
+ s.m = s.l = 0; s.s = 0;
+ if (*b->info) kputc(';', &s);
+ ksprintf(&s, "NEIR=%.3lf;NEIF=%.3lf,%.3lf", r2, f[0]+f[2], f[0]+f[1]);
+ bcf_append_info(b, s.s, s.l);
+ free(s.s);
+ }
+ bcf_cpy(blast, b);
+ }
+ if (vc.flag & VC_NO_GENO) { // do not output GENO fields
+ b->n_gi = 0;
+ b->fmt[0] = '\0';
+ }
+ vcf_write(bout, h, b);
+ }
+ if (vc.prior_file) free(vc.prior_file);
+ if (vc.flag & VC_CALL) bcf_p1_dump_afs(p1);
+ bcf_hdr_destroy(h);
+ bcf_destroy(b); bcf_destroy(blast);
+ vcf_close(bp); vcf_close(bout);
+ if (hash) kh_destroy(set64, hash);
+ if (vc.fn_list) free(vc.fn_list);
+ if (p1) bcf_p1_destroy(p1);
+ return 0;
+}
--- /dev/null
+#include <math.h>
+#include <stdlib.h>
+
+/* This program is implemented with ideas from this web page:
+ *
+ * http://www.langsrud.com/fisher.htm
+ */
+
+// log\binom{n}{k}
+static double lbinom(int n, int k)
+{
+ if (k == 0 || n == k) return 0;
+ return lgamma(n+1) - lgamma(k+1) - lgamma(n-k+1);
+}
+
+// n11 n12 | n1_
+// n21 n22 | n2_
+//-----------+----
+// n_1 n_2 | n
+
+// hypergeometric distribution
+static double hypergeo(int n11, int n1_, int n_1, int n)
+{
+ return exp(lbinom(n1_, n11) + lbinom(n-n1_, n_1-n11) - lbinom(n, n_1));
+}
+
+typedef struct {
+ int n11, n1_, n_1, n;
+ double p;
+} hgacc_t;
+
+// incremental version of hypergenometric distribution
+static double hypergeo_acc(int n11, int n1_, int n_1, int n, hgacc_t *aux)
+{
+ if (n1_ || n_1 || n) {
+ aux->n11 = n11; aux->n1_ = n1_; aux->n_1 = n_1; aux->n = n;
+ } else { // then only n11 changed; the rest fixed
+ if (n11%11 && n11 + aux->n - aux->n1_ - aux->n_1) {
+ if (n11 == aux->n11 + 1) { // incremental
+ aux->p *= (double)(aux->n1_ - aux->n11) / n11
+ * (aux->n_1 - aux->n11) / (n11 + aux->n - aux->n1_ - aux->n_1);
+ aux->n11 = n11;
+ return aux->p;
+ }
+ if (n11 == aux->n11 - 1) { // incremental
+ aux->p *= (double)aux->n11 / (aux->n1_ - n11)
+ * (aux->n11 + aux->n - aux->n1_ - aux->n_1) / (aux->n_1 - n11);
+ aux->n11 = n11;
+ return aux->p;
+ }
+ }
+ aux->n11 = n11;
+ }
+ aux->p = hypergeo(aux->n11, aux->n1_, aux->n_1, aux->n);
+ return aux->p;
+}
+
+double kt_fisher_exact(int n11, int n12, int n21, int n22, double *_left, double *_right, double *two)
+{
+ int i, j, max, min;
+ double p, q, left, right;
+ hgacc_t aux;
+ int n1_, n_1, n;
+
+ n1_ = n11 + n12; n_1 = n11 + n21; n = n11 + n12 + n21 + n22; // calculate n1_, n_1 and n
+ max = (n_1 < n1_) ? n_1 : n1_; // max n11, for right tail
+ min = (n1_ + n_1 - n < 0) ? 0 : (n1_ + n_1 - n < 0); // min n11, for left tail
+ *two = *_left = *_right = 1.;
+ if (min == max) return 1.; // no need to do test
+ q = hypergeo_acc(n11, n1_, n_1, n, &aux); // the probability of the current table
+ // left tail
+ p = hypergeo_acc(min, 0, 0, 0, &aux);
+ for (left = 0., i = min + 1; p < 0.99999999 * q; ++i) // loop until underflow
+ left += p, p = hypergeo_acc(i, 0, 0, 0, &aux);
+ --i;
+ if (p < 1.00000001 * q) left += p;
+ else --i;
+ // right tail
+ p = hypergeo_acc(max, 0, 0, 0, &aux);
+ for (right = 0., j = max - 1; p < 0.99999999 * q; --j) // loop until underflow
+ right += p, p = hypergeo_acc(j, 0, 0, 0, &aux);
+ if (p < 1.00000001 * q) right += p;
+ else ++j;
+ // two-tail
+ *two = left + right;
+ if (*two > 1.) *two = 1.;
+ // adjust left and right
+ if (abs(i - n11) < abs(j - n11)) right = 1. - left + q;
+ else left = 1.0 - right + q;
+ *_left = left; *_right = right;
+ return q;
+}
+
+#ifdef FET_MAIN
+#include <stdio.h>
+
+int main(int argc, char *argv[])
+{
+ char id[1024];
+ int n11, n12, n21, n22;
+ double left, right, twotail, prob;
+
+ while (scanf("%s%d%d%d%d", id, &n11, &n12, &n21, &n22) == 5) {
+ prob = kt_fisher_exact(n11, n12, n21, n22, &left, &right, &twotail);
+ printf("%s\t%d\t%d\t%d\t%d\t%.6g\t%.6g\t%.6g\t%.6g\n", id, n11, n12, n21, n22,
+ prob, left, right, twotail);
+ }
+ return 0;
+}
+#endif
--- /dev/null
+#include <assert.h>
+#include <ctype.h>
+#include <sys/stat.h>
+#include "bam_endian.h"
+#include "kstring.h"
+#include "bcf.h"
+#ifdef _USE_KNETFILE
+#include "knetfile.h"
+#endif
+
+#define TAD_LIDX_SHIFT 13
+
+typedef struct {
+ int32_t n, m;
+ uint64_t *offset;
+} bcf_lidx_t;
+
+struct __bcf_idx_t {
+ int32_t n;
+ bcf_lidx_t *index2;
+};
+
+/************
+ * indexing *
+ ************/
+
+static inline void insert_offset2(bcf_lidx_t *index2, int _beg, int _end, uint64_t offset)
+{
+ int i, beg, end;
+ beg = _beg >> TAD_LIDX_SHIFT;
+ end = (_end - 1) >> TAD_LIDX_SHIFT;
+ if (index2->m < end + 1) {
+ int old_m = index2->m;
+ index2->m = end + 1;
+ kroundup32(index2->m);
+ index2->offset = (uint64_t*)realloc(index2->offset, index2->m * 8);
+ memset(index2->offset + old_m, 0, 8 * (index2->m - old_m));
+ }
+ if (beg == end) {
+ if (index2->offset[beg] == 0) index2->offset[beg] = offset;
+ } else {
+ for (i = beg; i <= end; ++i)
+ if (index2->offset[i] == 0) index2->offset[i] = offset;
+ }
+ if (index2->n < end + 1) index2->n = end + 1;
+}
+
+bcf_idx_t *bcf_idx_core(bcf_t *bp, bcf_hdr_t *h)
+{
+ bcf_idx_t *idx;
+ int32_t last_coor, last_tid;
+ uint64_t last_off;
+ kstring_t *str;
+ BGZF *fp = bp->fp;
+ bcf1_t *b;
+ int ret;
+
+ b = calloc(1, sizeof(bcf1_t));
+ str = calloc(1, sizeof(kstring_t));
+ idx = (bcf_idx_t*)calloc(1, sizeof(bcf_idx_t));
+ idx->n = h->n_ref;
+ idx->index2 = calloc(h->n_ref, sizeof(bcf_lidx_t));
+
+ last_tid = 0xffffffffu;
+ last_off = bgzf_tell(fp); last_coor = 0xffffffffu;
+ while ((ret = bcf_read(bp, h, b)) > 0) {
+ int end, tmp;
+ if (last_tid != b->tid) { // change of chromosomes
+ last_tid = b->tid;
+ } else if (last_coor > b->pos) {
+ fprintf(stderr, "[bcf_idx_core] the input is out of order\n");
+ free(str->s); free(str); free(idx); bcf_destroy(b);
+ return 0;
+ }
+ tmp = strlen(b->ref);
+ end = b->pos + (tmp > 0? tmp : 1);
+ insert_offset2(&idx->index2[b->tid], b->pos, end, last_off);
+ last_off = bgzf_tell(fp);
+ last_coor = b->pos;
+ }
+ free(str->s); free(str); bcf_destroy(b);
+ return idx;
+}
+
+void bcf_idx_destroy(bcf_idx_t *idx)
+{
+ int i;
+ if (idx == 0) return;
+ for (i = 0; i < idx->n; ++i) free(idx->index2[i].offset);
+ free(idx->index2);
+ free(idx);
+}
+
+/******************
+ * index file I/O *
+ ******************/
+
+void bcf_idx_save(const bcf_idx_t *idx, BGZF *fp)
+{
+ int32_t i, ti_is_be;
+ ti_is_be = bam_is_big_endian();
+ bgzf_write(fp, "BCI\4", 4);
+ if (ti_is_be) {
+ uint32_t x = idx->n;
+ bgzf_write(fp, bam_swap_endian_4p(&x), 4);
+ } else bgzf_write(fp, &idx->n, 4);
+ for (i = 0; i < idx->n; ++i) {
+ bcf_lidx_t *index2 = idx->index2 + i;
+ // write linear index (index2)
+ if (ti_is_be) {
+ int x = index2->n;
+ bgzf_write(fp, bam_swap_endian_4p(&x), 4);
+ } else bgzf_write(fp, &index2->n, 4);
+ if (ti_is_be) { // big endian
+ int x;
+ for (x = 0; (int)x < index2->n; ++x)
+ bam_swap_endian_8p(&index2->offset[x]);
+ bgzf_write(fp, index2->offset, 8 * index2->n);
+ for (x = 0; (int)x < index2->n; ++x)
+ bam_swap_endian_8p(&index2->offset[x]);
+ } else bgzf_write(fp, index2->offset, 8 * index2->n);
+ }
+}
+
+static bcf_idx_t *bcf_idx_load_core(BGZF *fp)
+{
+ int i, ti_is_be;
+ char magic[4];
+ bcf_idx_t *idx;
+ ti_is_be = bam_is_big_endian();
+ if (fp == 0) {
+ fprintf(stderr, "[%s] fail to load index.\n", __func__);
+ return 0;
+ }
+ bgzf_read(fp, magic, 4);
+ if (strncmp(magic, "BCI\4", 4)) {
+ fprintf(stderr, "[%s] wrong magic number.\n", __func__);
+ return 0;
+ }
+ idx = (bcf_idx_t*)calloc(1, sizeof(bcf_idx_t));
+ bgzf_read(fp, &idx->n, 4);
+ if (ti_is_be) bam_swap_endian_4p(&idx->n);
+ idx->index2 = (bcf_lidx_t*)calloc(idx->n, sizeof(bcf_lidx_t));
+ for (i = 0; i < idx->n; ++i) {
+ bcf_lidx_t *index2 = idx->index2 + i;
+ int j;
+ bgzf_read(fp, &index2->n, 4);
+ if (ti_is_be) bam_swap_endian_4p(&index2->n);
+ index2->m = index2->n;
+ index2->offset = (uint64_t*)calloc(index2->m, 8);
+ bgzf_read(fp, index2->offset, index2->n * 8);
+ if (ti_is_be)
+ for (j = 0; j < index2->n; ++j) bam_swap_endian_8p(&index2->offset[j]);
+ }
+ return idx;
+}
+
+bcf_idx_t *bcf_idx_load_local(const char *fnidx)
+{
+ BGZF *fp;
+ fp = bgzf_open(fnidx, "r");
+ if (fp) {
+ bcf_idx_t *idx = bcf_idx_load_core(fp);
+ bgzf_close(fp);
+ return idx;
+ } else return 0;
+}
+
+#ifdef _USE_KNETFILE
+static void download_from_remote(const char *url)
+{
+ const int buf_size = 1 * 1024 * 1024;
+ char *fn;
+ FILE *fp;
+ uint8_t *buf;
+ knetFile *fp_remote;
+ int l;
+ if (strstr(url, "ftp://") != url && strstr(url, "http://") != url) return;
+ l = strlen(url);
+ for (fn = (char*)url + l - 1; fn >= url; --fn)
+ if (*fn == '/') break;
+ ++fn; // fn now points to the file name
+ fp_remote = knet_open(url, "r");
+ if (fp_remote == 0) {
+ fprintf(stderr, "[download_from_remote] fail to open remote file.\n");
+ return;
+ }
+ if ((fp = fopen(fn, "w")) == 0) {
+ fprintf(stderr, "[download_from_remote] fail to create file in the working directory.\n");
+ knet_close(fp_remote);
+ return;
+ }
+ buf = (uint8_t*)calloc(buf_size, 1);
+ while ((l = knet_read(fp_remote, buf, buf_size)) != 0)
+ fwrite(buf, 1, l, fp);
+ free(buf);
+ fclose(fp);
+ knet_close(fp_remote);
+}
+#else
+static void download_from_remote(const char *url)
+{
+ return;
+}
+#endif
+
+static char *get_local_version(const char *fn)
+{
+ struct stat sbuf;
+ char *fnidx = (char*)calloc(strlen(fn) + 5, 1);
+ strcat(strcpy(fnidx, fn), ".bci");
+ if ((strstr(fnidx, "ftp://") == fnidx || strstr(fnidx, "http://") == fnidx)) {
+ char *p, *url;
+ int l = strlen(fnidx);
+ for (p = fnidx + l - 1; p >= fnidx; --p)
+ if (*p == '/') break;
+ url = fnidx; fnidx = strdup(p + 1);
+ if (stat(fnidx, &sbuf) == 0) {
+ free(url);
+ return fnidx;
+ }
+ fprintf(stderr, "[%s] downloading the index file...\n", __func__);
+ download_from_remote(url);
+ free(url);
+ }
+ if (stat(fnidx, &sbuf) == 0) return fnidx;
+ free(fnidx); return 0;
+}
+
+bcf_idx_t *bcf_idx_load(const char *fn)
+{
+ bcf_idx_t *idx;
+ char *fname = get_local_version(fn);
+ if (fname == 0) return 0;
+ idx = bcf_idx_load_local(fname);
+ free(fname);
+ return idx;
+}
+
+int bcf_idx_build2(const char *fn, const char *_fnidx)
+{
+ char *fnidx;
+ BGZF *fpidx;
+ bcf_t *bp;
+ bcf_idx_t *idx;
+ bcf_hdr_t *h;
+ if ((bp = bcf_open(fn, "r")) == 0) {
+ fprintf(stderr, "[bcf_idx_build2] fail to open the BAM file.\n");
+ return -1;
+ }
+ h = bcf_hdr_read(bp);
+ idx = bcf_idx_core(bp, h);
+ bcf_close(bp);
+ if (_fnidx == 0) {
+ fnidx = (char*)calloc(strlen(fn) + 5, 1);
+ strcpy(fnidx, fn); strcat(fnidx, ".bci");
+ } else fnidx = strdup(_fnidx);
+ fpidx = bgzf_open(fnidx, "w");
+ if (fpidx == 0) {
+ fprintf(stderr, "[bcf_idx_build2] fail to create the index file.\n");
+ free(fnidx);
+ return -1;
+ }
+ bcf_idx_save(idx, fpidx);
+ bcf_idx_destroy(idx);
+ bgzf_close(fpidx);
+ free(fnidx);
+ return 0;
+}
+
+int bcf_idx_build(const char *fn)
+{
+ return bcf_idx_build2(fn, 0);
+}
+
+/********************************************
+ * parse a region in the format chr:beg-end *
+ ********************************************/
+
+int bcf_parse_region(void *str2id, const char *str, int *tid, int *begin, int *end)
+{
+ char *s, *p;
+ int i, l, k;
+ l = strlen(str);
+ p = s = (char*)malloc(l+1);
+ /* squeeze out "," */
+ for (i = k = 0; i != l; ++i)
+ if (str[i] != ',' && !isspace(str[i])) s[k++] = str[i];
+ s[k] = 0;
+ for (i = 0; i != k; ++i) if (s[i] == ':') break;
+ s[i] = 0;
+ if ((*tid = bcf_str2id(str2id, s)) < 0) {
+ free(s);
+ return -1;
+ }
+ if (i == k) { /* dump the whole sequence */
+ *begin = 0; *end = 1<<29; free(s);
+ return 0;
+ }
+ for (p = s + i + 1; i != k; ++i) if (s[i] == '-') break;
+ *begin = atoi(p);
+ if (i < k) {
+ p = s + i + 1;
+ *end = atoi(p);
+ } else *end = 1<<29;
+ if (*begin > 0) --*begin;
+ free(s);
+ if (*begin > *end) return -1;
+ return 0;
+}
+
+/*******************************
+ * retrieve a specified region *
+ *******************************/
+
+uint64_t bcf_idx_query(const bcf_idx_t *idx, int tid, int beg)
+{
+ uint64_t min_off, *offset;
+ int i;
+ if (beg < 0) beg = 0;
+ offset = idx->index2[tid].offset;
+ for (i = beg>>TAD_LIDX_SHIFT; i < idx->index2[tid].n && offset[i] == 0; ++i);
+ min_off = (i == idx->index2[tid].n)? offset[idx->index2[tid].n-1] : offset[i];
+ return min_off;
+}
+
+int bcf_main_index(int argc, char *argv[])
+{
+ if (argc == 1) {
+ fprintf(stderr, "Usage: bcftools index <in.bcf>\n");
+ return 1;
+ }
+ bcf_idx_build(argv[1]);
+ return 0;
+}
--- /dev/null
+#include <math.h>
+
+
+/* Log gamma function
+ * \log{\Gamma(z)}
+ * AS245, 2nd algorithm, http://lib.stat.cmu.edu/apstat/245
+ */
+double kf_lgamma(double z)
+{
+ double x = 0;
+ x += 0.1659470187408462e-06 / (z+7);
+ x += 0.9934937113930748e-05 / (z+6);
+ x -= 0.1385710331296526 / (z+5);
+ x += 12.50734324009056 / (z+4);
+ x -= 176.6150291498386 / (z+3);
+ x += 771.3234287757674 / (z+2);
+ x -= 1259.139216722289 / (z+1);
+ x += 676.5203681218835 / z;
+ x += 0.9999999999995183;
+ return log(x) - 5.58106146679532777 - z + (z-0.5) * log(z+6.5);
+}
+
+/* complementary error function
+ * \frac{2}{\sqrt{\pi}} \int_x^{\infty} e^{-t^2} dt
+ * AS66, 2nd algorithm, http://lib.stat.cmu.edu/apstat/66
+ */
+double kf_erfc(double x)
+{
+ const double p0 = 220.2068679123761;
+ const double p1 = 221.2135961699311;
+ const double p2 = 112.0792914978709;
+ const double p3 = 33.912866078383;
+ const double p4 = 6.37396220353165;
+ const double p5 = .7003830644436881;
+ const double p6 = .03526249659989109;
+ const double q0 = 440.4137358247522;
+ const double q1 = 793.8265125199484;
+ const double q2 = 637.3336333788311;
+ const double q3 = 296.5642487796737;
+ const double q4 = 86.78073220294608;
+ const double q5 = 16.06417757920695;
+ const double q6 = 1.755667163182642;
+ const double q7 = .08838834764831844;
+ double expntl, z, p;
+ z = fabs(x) * M_SQRT2;
+ if (z > 37.) return x > 0.? 0. : 2.;
+ expntl = exp(z * z * - .5);
+ if (z < 10. / M_SQRT2) // for small z
+ p = expntl * ((((((p6 * z + p5) * z + p4) * z + p3) * z + p2) * z + p1) * z + p0)
+ / (((((((q7 * z + q6) * z + q5) * z + q4) * z + q3) * z + q2) * z + q1) * z + q0);
+ else p = expntl / 2.506628274631001 / (z + 1. / (z + 2. / (z + 3. / (z + 4. / (z + .65)))));
+ return x > 0.? 2. * p : 2. * (1. - p);
+}
+
+/* The following computes regularized incomplete gamma functions.
+ * Formulas are taken from Wiki, with additional input from Numerical
+ * Recipes in C (for modified Lentz's algorithm) and AS245
+ * (http://lib.stat.cmu.edu/apstat/245).
+ *
+ * A good online calculator is available at:
+ *
+ * http://www.danielsoper.com/statcalc/calc23.aspx
+ *
+ * It calculates upper incomplete gamma function, which equals
+ * kf_gammaq(s,z)*tgamma(s).
+ */
+
+#define KF_GAMMA_EPS 1e-14
+#define KF_TINY 1e-290
+
+// regularized lower incomplete gamma function, by series expansion
+static double _kf_gammap(double s, double z)
+{
+ double sum, x;
+ int k;
+ for (k = 1, sum = x = 1.; k < 100; ++k) {
+ sum += (x *= z / (s + k));
+ if (x / sum < KF_GAMMA_EPS) break;
+ }
+ return exp(s * log(z) - z - kf_lgamma(s + 1.) + log(sum));
+}
+// regularized upper incomplete gamma function, by continued fraction
+static double _kf_gammaq(double s, double z)
+{
+ int j;
+ double C, D, f;
+ f = 1. + z - s; C = f; D = 0.;
+ // Modified Lentz's algorithm for computing continued fraction
+ // See Numerical Recipes in C, 2nd edition, section 5.2
+ for (j = 1; j < 100; ++j) {
+ double a = j * (s - j), b = (j<<1) + 1 + z - s, d;
+ D = b + a * D;
+ if (D < KF_TINY) D = KF_TINY;
+ C = b + a / C;
+ if (C < KF_TINY) C = KF_TINY;
+ D = 1. / D;
+ d = C * D;
+ f *= d;
+ if (fabs(d - 1.) < KF_GAMMA_EPS) break;
+ }
+ return exp(s * log(z) - z - kf_lgamma(s) - log(f));
+}
+
+double kf_gammap(double s, double z)
+{
+ return z <= 1. || z < s? _kf_gammap(s, z) : 1. - _kf_gammaq(s, z);
+}
+
+double kf_gammaq(double s, double z)
+{
+ return z <= 1. || z < s? 1. - _kf_gammap(s, z) : _kf_gammaq(s, z);
+}
+
+/* Regularized incomplete beta function. The method is taken from
+ * Numerical Recipe in C, 2nd edition, section 6.4. The following web
+ * page calculates the incomplete beta function, which equals
+ * kf_betai(a,b,x) * gamma(a) * gamma(b) / gamma(a+b):
+ *
+ * http://www.danielsoper.com/statcalc/calc36.aspx
+ */
+static double kf_betai_aux(double a, double b, double x)
+{
+ double C, D, f;
+ int j;
+ if (x == 0.) return 0.;
+ if (x == 1.) return 1.;
+ f = 1.; C = f; D = 0.;
+ // Modified Lentz's algorithm for computing continued fraction
+ for (j = 1; j < 200; ++j) {
+ double aa, d;
+ int m = j>>1;
+ aa = (j&1)? -(a + m) * (a + b + m) * x / ((a + 2*m) * (a + 2*m + 1))
+ : m * (b - m) * x / ((a + 2*m - 1) * (a + 2*m));
+ D = 1. + aa * D;
+ if (D < KF_TINY) D = KF_TINY;
+ C = 1. + aa / C;
+ if (C < KF_TINY) C = KF_TINY;
+ D = 1. / D;
+ d = C * D;
+ f *= d;
+ if (fabs(d - 1.) < KF_GAMMA_EPS) break;
+ }
+ return exp(kf_lgamma(a+b) - kf_lgamma(a) - kf_lgamma(b) + a * log(x) + b * log(1.-x)) / a / f;
+}
+double kf_betai(double a, double b, double x)
+{
+ return x < (a + 1.) / (a + b + 2.)? kf_betai_aux(a, b, x) : 1. - kf_betai_aux(b, a, 1. - x);
+}
+
+#ifdef KF_MAIN
+#include <stdio.h>
+int main(int argc, char *argv[])
+{
+ double x = 5.5, y = 3;
+ double a, b;
+ printf("erfc(%lg): %lg, %lg\n", x, erfc(x), kf_erfc(x));
+ printf("upper-gamma(%lg,%lg): %lg\n", x, y, kf_gammaq(y, x)*tgamma(y));
+ a = 2; b = 2; x = 0.5;
+ printf("incomplete-beta(%lg,%lg,%lg): %lg\n", a, b, x, kf_betai(a, b, x) / exp(kf_lgamma(a+b) - kf_lgamma(a) - kf_lgamma(b)));
+ return 0;
+}
+#endif
--- /dev/null
+#include <stdlib.h>
+#include <string.h>
+#include <math.h>
+#include "bcf.h"
+
+static double g_q2p[256];
+
+#define LD_ITER_MAX 50
+#define LD_ITER_EPS 1e-4
+
+#define _G1(h, k) ((h>>1&1) + (k>>1&1))
+#define _G2(h, k) ((h&1) + (k&1))
+
+// 0: the previous site; 1: the current site
+static int freq_iter(int n, double *pdg[2], double f[4])
+{
+ double ff[4];
+ int i, k, h;
+ memset(ff, 0, 4 * sizeof(double));
+ for (i = 0; i < n; ++i) {
+ double *p[2], sum, tmp;
+ p[0] = pdg[0] + i * 3; p[1] = pdg[1] + i * 3;
+ for (k = 0, sum = 0.; k < 4; ++k)
+ for (h = 0; h < 4; ++h)
+ sum += f[k] * f[h] * p[0][_G1(k,h)] * p[1][_G2(k,h)];
+ for (k = 0; k < 4; ++k) {
+ tmp = f[0] * (p[0][_G1(0,k)] * p[1][_G2(0,k)] + p[0][_G1(k,0)] * p[1][_G2(k,0)])
+ + f[1] * (p[0][_G1(1,k)] * p[1][_G2(1,k)] + p[0][_G1(k,1)] * p[1][_G2(k,1)])
+ + f[2] * (p[0][_G1(2,k)] * p[1][_G2(2,k)] + p[0][_G1(k,2)] * p[1][_G2(k,2)])
+ + f[3] * (p[0][_G1(3,k)] * p[1][_G2(3,k)] + p[0][_G1(k,3)] * p[1][_G2(k,3)]);
+ ff[k] += f[k] * tmp / sum;
+ }
+ }
+ for (k = 0; k < 4; ++k) f[k] = ff[k] / (2 * n);
+ return 0;
+}
+
+double bcf_ld_freq(const bcf1_t *b0, const bcf1_t *b1, double f[4])
+{
+ const bcf1_t *b[2];
+ uint8_t *PL[2];
+ int i, j, PL_len[2], n_smpl;
+ double *pdg[2], flast[4], r;
+ // initialize g_q2p if necessary
+ if (g_q2p[0] == 0.)
+ for (i = 0; i < 256; ++i)
+ g_q2p[i] = pow(10., -i / 10.);
+ // initialize others
+ if (b0->n_smpl != b1->n_smpl) return -1; // different number of samples
+ n_smpl = b0->n_smpl;
+ b[0] = b0; b[1] = b1;
+ f[0] = f[1] = f[2] = f[3] = -1.;
+ if (b[0]->n_alleles < 2 || b[1]->n_alleles < 2) return -1; // one allele only
+ // set PL and PL_len
+ for (j = 0; j < 2; ++j) {
+ const bcf1_t *bj = b[j];
+ for (i = 0; i < bj->n_gi; ++i) {
+ if (bj->gi[i].fmt == bcf_str2int("PL", 2)) {
+ PL[j] = (uint8_t*)bj->gi[i].data;
+ PL_len[j] = bj->gi[i].len;
+ break;
+ }
+ }
+ if (i == bj->n_gi) return -1; // no PL
+ }
+ // fill pdg[2]
+ pdg[0] = malloc(3 * n_smpl * sizeof(double));
+ pdg[1] = malloc(3 * n_smpl * sizeof(double));
+ for (j = 0; j < 2; ++j) {
+ for (i = 0; i < n_smpl; ++i) {
+ const uint8_t *pi = PL[j] + i * PL_len[j];
+ double *p = pdg[j] + i * 3;
+ p[0] = g_q2p[pi[b[j]->n_alleles]]; p[1] = g_q2p[pi[1]]; p[2] = g_q2p[pi[0]];
+ }
+ }
+ // iteration
+ f[0] = f[1] = f[2] = f[3] = 0.25; // this is a really bad guess...
+ for (j = 0; j < LD_ITER_MAX; ++j) {
+ double eps = 0;
+ memcpy(flast, f, 4 * sizeof(double));
+ freq_iter(n_smpl, pdg, f);
+ for (i = 0; i < 4; ++i) {
+ double x = fabs(f[0] - flast[0]);
+ if (x > eps) eps = x;
+ }
+ if (eps < LD_ITER_EPS) break;
+ }
+ // free
+ free(pdg[0]); free(pdg[1]);
+ { // calculate r^2
+ double p[2], q[2], D;
+ p[0] = f[0] + f[1]; q[0] = 1 - p[0];
+ p[1] = f[0] + f[2]; q[1] = 1 - p[1];
+ D = f[0] * f[3] - f[1] * f[2];
+ r = sqrt(D * D / (p[0] * p[1] * q[0] * q[1]));
+ // fprintf(stderr, "R(%lf,%lf,%lf,%lf)=%lf\n", f[0], f[1], f[2], f[3], r2);
+ if (isnan(r)) r = -1.;
+ }
+ return r;
+}
--- /dev/null
+#include <string.h>
+#include <stdlib.h>
+#include <sys/stat.h>
+#include "bcf.h"
+
+int bcfview(int argc, char *argv[]);
+int bcf_main_index(int argc, char *argv[]);
+
+#define BUF_SIZE 0x10000
+
+int bcf_cat(int n, char * const *fn)
+{
+ int i;
+ bcf_t *out;
+ uint8_t *buf;
+ buf = malloc(BUF_SIZE);
+ out = bcf_open("-", "w");
+ for (i = 0; i < n; ++i) {
+ bcf_t *in;
+ bcf_hdr_t *h;
+ off_t end;
+ struct stat s;
+ in = bcf_open(fn[i], "r");
+ h = bcf_hdr_read(in);
+ if (i == 0) bcf_hdr_write(out, h);
+ bcf_hdr_destroy(h);
+#ifdef _USE_KNETFILE
+ fstat(knet_fileno(in->fp->x.fpr), &s);
+ end = s.st_size - 28;
+ while (knet_tell(in->fp->x.fpr) < end) {
+ int size = knet_tell(in->fp->x.fpr) + BUF_SIZE < end? BUF_SIZE : end - knet_tell(in->fp->x.fpr);
+ knet_read(in->fp->x.fpr, buf, size);
+ fwrite(buf, 1, size, out->fp->x.fpw);
+ }
+#else
+ abort(); // FIXME: not implemented
+#endif
+ bcf_close(in);
+ }
+ bcf_close(out);
+ free(buf);
+ return 0;
+}
+
+int main(int argc, char *argv[])
+{
+ if (argc == 1) {
+ fprintf(stderr, "\n");
+ fprintf(stderr, "Usage: bcftools <command> <arguments>\n\n");
+ fprintf(stderr, "Command: view print, extract, convert and call SNPs from BCF\n");
+ fprintf(stderr, " index index BCF\n");
+ fprintf(stderr, " cat concatenate BCFs\n");
+ fprintf(stderr, "\n");
+ return 1;
+ }
+ if (strcmp(argv[1], "view") == 0) return bcfview(argc-1, argv+1);
+ else if (strcmp(argv[1], "index") == 0) return bcf_main_index(argc-1, argv+1);
+ else if (strcmp(argv[1], "cat") == 0) return bcf_cat(argc-2, argv+2);
+ else {
+ fprintf(stderr, "[main] Unrecognized command.\n");
+ return 1;
+ }
+ return 0;
+}
--- /dev/null
+#include <math.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stdio.h>
+#include <errno.h>
+#include "prob1.h"
+
+#include "kseq.h"
+KSTREAM_INIT(gzFile, gzread, 16384)
+
+#define MC_AVG_ERR 0.007
+#define MC_MAX_EM_ITER 16
+#define MC_EM_EPS 1e-4
+
+unsigned char seq_nt4_table[256] = {
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4 /*'-'*/, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 0, 4, 1, 4, 4, 4, 2, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 0, 4, 1, 4, 4, 4, 2, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,
+ 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4
+};
+
+struct __bcf_p1aux_t {
+ int n, M, n1;
+ double *q2p, *pdg; // pdg -> P(D|g)
+ double *phi;
+ double *z, *zswap; // aux for afs
+ double *z1, *z2, *phi1, *phi2; // only calculated when n1 is set
+ double t, t1, t2;
+ double *afs, *afs1; // afs: accumulative AFS; afs1: site posterior distribution
+ const uint8_t *PL; // point to PL
+ int PL_len;
+};
+
+static void init_prior(int type, double theta, int M, double *phi)
+{
+ int i;
+ if (type == MC_PTYPE_COND2) {
+ for (i = 0; i <= M; ++i)
+ phi[i] = 2. * (i + 1) / (M + 1) / (M + 2);
+ } else if (type == MC_PTYPE_FLAT) {
+ for (i = 0; i <= M; ++i)
+ phi[i] = 1. / (M + 1);
+ } else {
+ double sum;
+ for (i = 0, sum = 0.; i < M; ++i)
+ sum += (phi[i] = theta / (M - i));
+ phi[M] = 1. - sum;
+ }
+}
+
+void bcf_p1_init_prior(bcf_p1aux_t *ma, int type, double theta)
+{
+ init_prior(type, theta, ma->M, ma->phi);
+}
+
+void bcf_p1_init_subprior(bcf_p1aux_t *ma, int type, double theta)
+{
+ if (ma->n1 <= 0 || ma->n1 >= ma->M) return;
+ init_prior(type, theta, 2*ma->n1, ma->phi1);
+ init_prior(type, theta, 2*(ma->n - ma->n1), ma->phi2);
+}
+
+int bcf_p1_read_prior(bcf_p1aux_t *ma, const char *fn)
+{
+ gzFile fp;
+ kstring_t s;
+ kstream_t *ks;
+ long double sum;
+ int dret, k;
+ memset(&s, 0, sizeof(kstring_t));
+ fp = strcmp(fn, "-")? gzopen(fn, "r") : gzdopen(fileno(stdin), "r");
+ ks = ks_init(fp);
+ memset(ma->phi, 0, sizeof(double) * (ma->M + 1));
+ while (ks_getuntil(ks, '\n', &s, &dret) >= 0) {
+ if (strstr(s.s, "[afs] ") == s.s) {
+ char *p = s.s + 6;
+ for (k = 0; k <= ma->M; ++k) {
+ int x;
+ double y;
+ x = strtol(p, &p, 10);
+ if (x != k && (errno == EINVAL || errno == ERANGE)) return -1;
+ ++p;
+ y = strtod(p, &p);
+ if (y == 0. && (errno == EINVAL || errno == ERANGE)) return -1;
+ ma->phi[ma->M - k] += y;
+ }
+ }
+ }
+ ks_destroy(ks);
+ gzclose(fp);
+ free(s.s);
+ for (sum = 0., k = 0; k <= ma->M; ++k) sum += ma->phi[k];
+ fprintf(stderr, "[prior]");
+ for (k = 0; k <= ma->M; ++k) ma->phi[k] /= sum;
+ for (k = 0; k <= ma->M; ++k) fprintf(stderr, " %d:%.3lg", k, ma->phi[ma->M - k]);
+ fputc('\n', stderr);
+ for (sum = 0., k = 1; k < ma->M; ++k) sum += ma->phi[ma->M - k] * (2.* k * (ma->M - k) / ma->M / (ma->M - 1));
+ fprintf(stderr, "[%s] heterozygosity=%lf, ", __func__, (double)sum);
+ for (sum = 0., k = 1; k <= ma->M; ++k) sum += k * ma->phi[ma->M - k] / ma->M;
+ fprintf(stderr, "theta=%lf\n", (double)sum);
+ return 0;
+}
+
+bcf_p1aux_t *bcf_p1_init(int n)
+{
+ bcf_p1aux_t *ma;
+ int i;
+ ma = calloc(1, sizeof(bcf_p1aux_t));
+ ma->n1 = -1;
+ ma->n = n; ma->M = 2 * n;
+ ma->q2p = calloc(256, sizeof(double));
+ ma->pdg = calloc(3 * ma->n, sizeof(double));
+ ma->phi = calloc(ma->M + 1, sizeof(double));
+ ma->phi1 = calloc(ma->M + 1, sizeof(double));
+ ma->phi2 = calloc(ma->M + 1, sizeof(double));
+ ma->z = calloc(2 * ma->n + 1, sizeof(double));
+ ma->zswap = calloc(2 * ma->n + 1, sizeof(double));
+ ma->z1 = calloc(ma->M + 1, sizeof(double)); // actually we do not need this large
+ ma->z2 = calloc(ma->M + 1, sizeof(double));
+ ma->afs = calloc(2 * ma->n + 1, sizeof(double));
+ ma->afs1 = calloc(2 * ma->n + 1, sizeof(double));
+ for (i = 0; i < 256; ++i)
+ ma->q2p[i] = pow(10., -i / 10.);
+ bcf_p1_init_prior(ma, MC_PTYPE_FULL, 1e-3); // the simplest prior
+ return ma;
+}
+
+int bcf_p1_set_n1(bcf_p1aux_t *b, int n1)
+{
+ if (n1 == 0 || n1 >= b->n) return -1;
+ b->n1 = n1;
+ return 0;
+}
+
+void bcf_p1_destroy(bcf_p1aux_t *ma)
+{
+ if (ma) {
+ free(ma->q2p); free(ma->pdg);
+ free(ma->phi); free(ma->phi1); free(ma->phi2);
+ free(ma->z); free(ma->zswap); free(ma->z1); free(ma->z2);
+ free(ma->afs); free(ma->afs1);
+ free(ma);
+ }
+}
+
+static int cal_pdg(const bcf1_t *b, bcf_p1aux_t *ma)
+{
+ int i, j, k;
+ long *p, tmp;
+ p = alloca(b->n_alleles * sizeof(long));
+ memset(p, 0, sizeof(long) * b->n_alleles);
+ for (j = 0; j < ma->n; ++j) {
+ const uint8_t *pi = ma->PL + j * ma->PL_len;
+ double *pdg = ma->pdg + j * 3;
+ pdg[0] = ma->q2p[pi[b->n_alleles]]; pdg[1] = ma->q2p[pi[1]]; pdg[2] = ma->q2p[pi[0]];
+ for (i = k = 0; i < b->n_alleles; ++i) {
+ p[i] += (int)pi[k];
+ k += b->n_alleles - i;
+ }
+ }
+ for (i = 0; i < b->n_alleles; ++i) p[i] = p[i]<<4 | i;
+ for (i = 1; i < b->n_alleles; ++i) // insertion sort
+ for (j = i; j > 0 && p[j] < p[j-1]; --j)
+ tmp = p[j], p[j] = p[j-1], p[j-1] = tmp;
+ for (i = b->n_alleles - 1; i >= 0; --i)
+ if ((p[i]&0xf) == 0) break;
+ return i;
+}
+// f0 is the reference allele frequency
+static double mc_freq_iter(double f0, const bcf_p1aux_t *ma)
+{
+ double f, f3[3];
+ int i;
+ f3[0] = (1.-f0)*(1.-f0); f3[1] = 2.*f0*(1.-f0); f3[2] = f0*f0;
+ for (i = 0, f = 0.; i < ma->n; ++i) {
+ double *pdg;
+ pdg = ma->pdg + i * 3;
+ f += (pdg[1] * f3[1] + 2. * pdg[2] * f3[2])
+ / (pdg[0] * f3[0] + pdg[1] * f3[1] + pdg[2] * f3[2]);
+ }
+ f /= ma->n * 2.;
+ return f;
+}
+
+int bcf_p1_call_gt(const bcf_p1aux_t *ma, double f0, int k)
+{
+ double sum, g[3];
+ double max, f3[3], *pdg = ma->pdg + k * 3;
+ int q, i, max_i;
+ f3[0] = (1.-f0)*(1.-f0); f3[1] = 2.*f0*(1.-f0); f3[2] = f0*f0;
+ for (i = 0, sum = 0.; i < 3; ++i)
+ sum += (g[i] = pdg[i] * f3[i]);
+ for (i = 0, max = -1., max_i = 0; i < 3; ++i) {
+ g[i] /= sum;
+ if (g[i] > max) max = g[i], max_i = i;
+ }
+ max = 1. - max;
+ if (max < 1e-308) max = 1e-308;
+ q = (int)(-4.343 * log(max) + .499);
+ if (q > 99) q = 99;
+ return q<<2|max_i;
+}
+
+#define TINY 1e-20
+
+static void mc_cal_y_core(bcf_p1aux_t *ma, int beg)
+{
+ double *z[2], *tmp, *pdg;
+ int _j, last_min, last_max;
+ z[0] = ma->z;
+ z[1] = ma->zswap;
+ pdg = ma->pdg;
+ memset(z[0], 0, sizeof(double) * (ma->M + 1));
+ memset(z[1], 0, sizeof(double) * (ma->M + 1));
+ z[0][0] = 1.;
+ last_min = last_max = 0;
+ ma->t = 0.;
+ for (_j = beg; _j < ma->n; ++_j) {
+ int k, j = _j - beg, _min = last_min, _max = last_max;
+ double p[3], sum;
+ pdg = ma->pdg + _j * 3;
+ p[0] = pdg[0]; p[1] = 2. * pdg[1]; p[2] = pdg[2];
+ for (; _min < _max && z[0][_min] < TINY; ++_min) z[0][_min] = z[1][_min] = 0.;
+ for (; _max > _min && z[0][_max] < TINY; --_max) z[0][_max] = z[1][_max] = 0.;
+ _max += 2;
+ if (_min == 0)
+ k = 0, z[1][k] = (2*j+2-k)*(2*j-k+1) * p[0] * z[0][k];
+ if (_min <= 1)
+ k = 1, z[1][k] = (2*j+2-k)*(2*j-k+1) * p[0] * z[0][k] + k*(2*j+2-k) * p[1] * z[0][k-1];
+ for (k = _min < 2? 2 : _min; k <= _max; ++k)
+ z[1][k] = (2*j+2-k)*(2*j-k+1) * p[0] * z[0][k]
+ + k*(2*j+2-k) * p[1] * z[0][k-1]
+ + k*(k-1)* p[2] * z[0][k-2];
+ for (k = _min, sum = 0.; k <= _max; ++k) sum += z[1][k];
+ ma->t += log(sum / ((2. * j + 2) * (2. * j + 1)));
+ for (k = _min; k <= _max; ++k) z[1][k] /= sum;
+ if (_min >= 1) z[1][_min-1] = 0.;
+ if (_min >= 2) z[1][_min-2] = 0.;
+ if (j < ma->n - 1) z[1][_max+1] = z[1][_max+2] = 0.;
+ if (_j == ma->n1 - 1) { // set pop1
+ ma->t1 = ma->t;
+ memcpy(ma->z1, z[1], sizeof(double) * (ma->n1 * 2 + 1));
+ }
+ tmp = z[0]; z[0] = z[1]; z[1] = tmp;
+ last_min = _min; last_max = _max;
+ }
+ if (z[0] != ma->z) memcpy(ma->z, z[0], sizeof(double) * (ma->M + 1));
+}
+
+static void mc_cal_y(bcf_p1aux_t *ma)
+{
+ if (ma->n1 > 0 && ma->n1 < ma->n) {
+ int k;
+ long double x;
+ memset(ma->z1, 0, sizeof(double) * (2 * ma->n1 + 1));
+ memset(ma->z2, 0, sizeof(double) * (2 * (ma->n - ma->n1) + 1));
+ ma->t1 = ma->t2 = 0.;
+ mc_cal_y_core(ma, ma->n1);
+ ma->t2 = ma->t;
+ memcpy(ma->z2, ma->z, sizeof(double) * (2 * (ma->n - ma->n1) + 1));
+ mc_cal_y_core(ma, 0);
+ // rescale z
+ x = expl(ma->t - (ma->t1 + ma->t2));
+ for (k = 0; k <= ma->M; ++k) ma->z[k] *= x;
+ } else mc_cal_y_core(ma, 0);
+}
+
+static void contrast(bcf_p1aux_t *ma, double pc[4]) // mc_cal_y() must be called before hand
+{
+ int k, n1 = ma->n1, n2 = ma->n - ma->n1;
+ long double sum1, sum2;
+ pc[0] = pc[1] = pc[2] = pc[3] = -1.;
+ if (n1 <= 0 || n2 <= 0) return;
+ for (k = 0, sum1 = 0.; k <= 2*n1; ++k) sum1 += ma->phi1[k] * ma->z1[k];
+ for (k = 0, sum2 = 0.; k <= 2*n2; ++k) sum2 += ma->phi2[k] * ma->z2[k];
+ pc[2] = ma->phi1[2*n1] * ma->z1[2*n1] / sum1;
+ pc[3] = ma->phi2[2*n2] * ma->z2[2*n2] / sum2;
+ for (k = 2; k < 4; ++k) {
+ pc[k] = pc[k] > .5? -(-4.343 * log(1. - pc[k] + TINY) + .499) : -4.343 * log(pc[k] + TINY) + .499;
+ pc[k] = (int)pc[k];
+ if (pc[k] > 99) pc[k] = 99;
+ if (pc[k] < -99) pc[k] = -99;
+ }
+ pc[0] = ma->phi2[2*n2] * ma->z2[2*n2] / sum2 * (1. - ma->phi1[2*n1] * ma->z1[2*n1] / sum1);
+ pc[1] = ma->phi1[2*n1] * ma->z1[2*n1] / sum1 * (1. - ma->phi2[2*n2] * ma->z2[2*n2] / sum2);
+ pc[0] = pc[0] == 1.? 99 : (int)(-4.343 * log(1. - pc[0]) + .499);
+ pc[1] = pc[1] == 1.? 99 : (int)(-4.343 * log(1. - pc[1]) + .499);
+}
+
+static double mc_cal_afs(bcf_p1aux_t *ma)
+{
+ int k;
+ long double sum = 0.;
+ memset(ma->afs1, 0, sizeof(double) * (ma->M + 1));
+ mc_cal_y(ma);
+ for (k = 0, sum = 0.; k <= ma->M; ++k)
+ sum += (long double)ma->phi[k] * ma->z[k];
+ for (k = 0; k <= ma->M; ++k) {
+ ma->afs1[k] = ma->phi[k] * ma->z[k] / sum;
+ if (isnan(ma->afs1[k]) || isinf(ma->afs1[k])) return -1.;
+ }
+ for (k = 0, sum = 0.; k <= ma->M; ++k) {
+ ma->afs[k] += ma->afs1[k];
+ sum += k * ma->afs1[k];
+ }
+ return sum / ma->M;
+}
+
+long double bcf_p1_cal_g3(bcf_p1aux_t *p1a, double g[3])
+{
+ long double pd = 0., g2[3];
+ int i, k;
+ memset(g2, 0, sizeof(long double) * 3);
+ for (k = 0; k < p1a->M; ++k) {
+ double f = (double)k / p1a->M, f3[3], g1[3];
+ long double z = 1.;
+ g1[0] = g1[1] = g1[2] = 0.;
+ f3[0] = (1. - f) * (1. - f); f3[1] = 2. * f * (1. - f); f3[2] = f * f;
+ for (i = 0; i < p1a->n; ++i) {
+ double *pdg = p1a->pdg + i * 3;
+ double x = pdg[0] * f3[0] + pdg[1] * f3[1] + pdg[2] * f3[2];
+ z *= x;
+ g1[0] += pdg[0] * f3[0] / x;
+ g1[1] += pdg[1] * f3[1] / x;
+ g1[2] += pdg[2] * f3[2] / x;
+ }
+ pd += p1a->phi[k] * z;
+ for (i = 0; i < 3; ++i)
+ g2[i] += p1a->phi[k] * z * g1[i];
+ }
+ for (i = 0; i < 3; ++i) g[i] = g2[i] / pd;
+ return pd;
+}
+
+int bcf_p1_cal(bcf1_t *b, bcf_p1aux_t *ma, bcf_p1rst_t *rst)
+{
+ int i, k;
+ long double sum = 0.;
+ // set PL and PL_len
+ for (i = 0; i < b->n_gi; ++i) {
+ if (b->gi[i].fmt == bcf_str2int("PL", 2)) {
+ ma->PL = (uint8_t*)b->gi[i].data;
+ ma->PL_len = b->gi[i].len;
+ break;
+ }
+ }
+ if (b->n_alleles < 2) return -1; // FIXME: find a better solution
+ //
+ rst->rank0 = cal_pdg(b, ma);
+ rst->f_exp = mc_cal_afs(ma);
+ rst->p_ref = ma->afs1[ma->M];
+ // calculate f_flat and f_em
+ for (k = 0, sum = 0.; k <= ma->M; ++k)
+ sum += (long double)ma->z[k];
+ rst->f_flat = 0.;
+ for (k = 0; k <= ma->M; ++k) {
+ double p = ma->z[k] / sum;
+ rst->f_flat += k * p;
+ }
+ rst->f_flat /= ma->M;
+ { // calculate f_em
+ double flast = rst->f_flat;
+ for (i = 0; i < MC_MAX_EM_ITER; ++i) {
+ rst->f_em = mc_freq_iter(flast, ma);
+ if (fabs(rst->f_em - flast) < MC_EM_EPS) break;
+ flast = rst->f_em;
+ }
+ }
+ rst->g[0] = rst->g[1] = rst->g[2] = -1.;
+ contrast(ma, rst->pc);
+ return 0;
+}
+
+void bcf_p1_dump_afs(bcf_p1aux_t *ma)
+{
+ int k;
+ fprintf(stderr, "[afs]");
+ for (k = 0; k <= ma->M; ++k)
+ fprintf(stderr, " %d:%.3lf", k, ma->afs[ma->M - k]);
+ fprintf(stderr, "\n");
+ memset(ma->afs, 0, sizeof(double) * (ma->M + 1));
+}
--- /dev/null
+#ifndef BCF_PROB1_H
+#define BCF_PROB1_H
+
+#include "bcf.h"
+
+struct __bcf_p1aux_t;
+typedef struct __bcf_p1aux_t bcf_p1aux_t;
+
+typedef struct {
+ int rank0;
+ double f_em, f_exp, f_flat, p_ref;
+ double pc[4];
+ double g[3];
+} bcf_p1rst_t;
+
+#define MC_PTYPE_FULL 1
+#define MC_PTYPE_COND2 2
+#define MC_PTYPE_FLAT 3
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+ bcf_p1aux_t *bcf_p1_init(int n);
+ void bcf_p1_init_prior(bcf_p1aux_t *ma, int type, double theta);
+ void bcf_p1_init_subprior(bcf_p1aux_t *ma, int type, double theta);
+ void bcf_p1_destroy(bcf_p1aux_t *ma);
+ int bcf_p1_cal(bcf1_t *b, bcf_p1aux_t *ma, bcf_p1rst_t *rst);
+ int bcf_p1_call_gt(const bcf_p1aux_t *ma, double f0, int k);
+ void bcf_p1_dump_afs(bcf_p1aux_t *ma);
+ int bcf_p1_read_prior(bcf_p1aux_t *ma, const char *fn);
+ long double bcf_p1_cal_g3(bcf_p1aux_t *p1a, double g[3]);
+ int bcf_p1_set_n1(bcf_p1aux_t *b, int n1);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
--- /dev/null
+#include <zlib.h>
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include "bcf.h"
+#include "kstring.h"
+#include "kseq.h"
+KSTREAM_INIT(gzFile, gzread, 4096)
+
+typedef struct {
+ gzFile fp;
+ FILE *fpout;
+ kstream_t *ks;
+ void *refhash;
+ kstring_t line;
+ int max_ref;
+} vcf_t;
+
+bcf_hdr_t *vcf_hdr_read(bcf_t *bp)
+{
+ kstring_t meta, smpl;
+ int dret;
+ vcf_t *v;
+ bcf_hdr_t *h;
+ if (!bp->is_vcf) return bcf_hdr_read(bp);
+ h = calloc(1, sizeof(bcf_hdr_t));
+ v = (vcf_t*)bp->v;
+ v->line.l = 0;
+ memset(&meta, 0, sizeof(kstring_t));
+ memset(&smpl, 0, sizeof(kstring_t));
+ while (ks_getuntil(v->ks, '\n', &v->line, &dret) >= 0) {
+ if (v->line.l < 2) continue;
+ if (v->line.s[0] != '#') return 0; // no sample line
+ if (v->line.s[0] == '#' && v->line.s[1] == '#') {
+ kputsn(v->line.s, v->line.l, &meta); kputc('\n', &meta);
+ } else if (v->line.s[0] == '#') {
+ int k;
+ ks_tokaux_t aux;
+ char *p;
+ for (p = kstrtok(v->line.s, "\t\n", &aux), k = 0; p; p = kstrtok(0, 0, &aux), ++k) {
+ if (k >= 9) {
+ kputsn(p, aux.p - p, &smpl);
+ kputc('\0', &smpl);
+ }
+ }
+ break;
+ }
+ }
+ kputc('\0', &meta);
+ h->name = 0;
+ h->sname = smpl.s; h->l_smpl = smpl.l;
+ h->txt = meta.s; h->l_txt = meta.l;
+ bcf_hdr_sync(h);
+ return h;
+}
+
+bcf_t *vcf_open(const char *fn, const char *mode)
+{
+ bcf_t *bp;
+ vcf_t *v;
+ if (strchr(mode, 'b')) return bcf_open(fn, mode);
+ bp = calloc(1, sizeof(bcf_t));
+ v = calloc(1, sizeof(vcf_t));
+ bp->is_vcf = 1;
+ bp->v = v;
+ v->refhash = bcf_str2id_init();
+ if (strchr(mode, 'r')) {
+ v->fp = strcmp(fn, "-")? gzopen(fn, "r") : gzdopen(fileno(stdin), "r");
+ v->ks = ks_init(v->fp);
+ } else if (strchr(mode, 'w'))
+ v->fpout = strcmp(fn, "-")? fopen(fn, "w") : stdout;
+ return bp;
+}
+
+int vcf_close(bcf_t *bp)
+{
+ vcf_t *v;
+ if (bp == 0) return -1;
+ if (!bp->is_vcf) return bcf_close(bp);
+ v = (vcf_t*)bp->v;
+ if (v->fp) {
+ ks_destroy(v->ks);
+ gzclose(v->fp);
+ }
+ if (v->fpout) fclose(v->fpout);
+ free(v->line.s);
+ bcf_str2id_destroy(v->refhash);
+ free(v);
+ free(bp);
+ return 0;
+}
+
+int vcf_hdr_write(bcf_t *bp, const bcf_hdr_t *h)
+{
+ vcf_t *v = (vcf_t*)bp->v;
+ int i, has_ref = 0, has_ver = 0;
+ if (!bp->is_vcf) return bcf_hdr_write(bp, h);
+ if (h->l_txt > 0) {
+ if (strstr(h->txt, "##fileformat=")) has_ver = 1;
+ if (has_ver == 0) fprintf(v->fpout, "##fileformat=VCFv4.0\n");
+ fwrite(h->txt, 1, h->l_txt - 1, v->fpout);
+ if (strstr(h->txt, "##SQ=")) has_ref = 1;
+ }
+ if (has_ver == 0) fprintf(v->fpout, "##fileformat=VCFv4.0\n");
+ fprintf(v->fpout, "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT");
+ for (i = 0; i < h->n_smpl; ++i)
+ fprintf(v->fpout, "\t%s", h->sns[i]);
+ fputc('\n', v->fpout);
+ return 0;
+}
+
+int vcf_write(bcf_t *bp, bcf_hdr_t *h, bcf1_t *b)
+{
+ vcf_t *v = (vcf_t*)bp->v;
+ extern void bcf_fmt_core(const bcf_hdr_t *h, bcf1_t *b, kstring_t *s);
+ if (!bp->is_vcf) return bcf_write(bp, h, b);
+ bcf_fmt_core(h, b, &v->line);
+ fwrite(v->line.s, 1, v->line.l, v->fpout);
+ fputc('\n', v->fpout);
+ return v->line.l + 1;
+}
+
+int vcf_read(bcf_t *bp, bcf_hdr_t *h, bcf1_t *b)
+{
+ int dret, k, i, sync = 0;
+ vcf_t *v = (vcf_t*)bp->v;
+ char *p, *q;
+ kstring_t str, rn;
+ ks_tokaux_t aux, a2;
+ if (!bp->is_vcf) return bcf_read(bp, h, b);
+ v->line.l = 0;
+ str.l = 0; str.m = b->m_str; str.s = b->str;
+ rn.l = rn.m = h->l_nm; rn.s = h->name;
+ if (ks_getuntil(v->ks, '\n', &v->line, &dret) < 0) return -1;
+ b->n_smpl = h->n_smpl;
+ for (p = kstrtok(v->line.s, "\t", &aux), k = 0; p; p = kstrtok(0, 0, &aux), ++k) {
+ *(char*)aux.p = 0;
+ if (k == 0) { // ref
+ int tid = bcf_str2id(v->refhash, p);
+ if (tid < 0) {
+ tid = bcf_str2id_add(v->refhash, p);
+ kputs(p, &rn); kputc('\0', &rn);
+ sync = 1;
+ }
+ b->tid = tid;
+ } else if (k == 1) { // pos
+ b->pos = atoi(p) - 1;
+ } else if (k == 5) { // qual
+ b->qual = (p[0] >= '0' && p[0] <= '9')? atof(p) : 0;
+ } else if (k <= 8) { // variable length strings
+ kputs(p, &str); kputc('\0', &str);
+ b->l_str = str.l; b->m_str = str.m; b->str = str.s;
+ if (k == 8) bcf_sync(b);
+ } else { // k > 9
+ if (strncmp(p, "./.", 3) == 0) {
+ for (i = 0; i < b->n_gi; ++i) {
+ if (b->gi[i].fmt == bcf_str2int("GT", 2)) {
+ ((uint8_t*)b->gi[i].data)[k-9] = 1<<7;
+ } else if (b->gi[i].fmt == bcf_str2int("GQ", 2)) {
+ ((uint8_t*)b->gi[i].data)[k-9] = 0;
+ } else if (b->gi[i].fmt == bcf_str2int("DP", 2)) {
+ ((uint16_t*)b->gi[i].data)[k-9] = 0;
+ } else if (b->gi[i].fmt == bcf_str2int("PL", 2)) {
+ int y = b->n_alleles * (b->n_alleles + 1) / 2;
+ memset((uint8_t*)b->gi[i].data + (k - 9) * y, 0, y);
+ } else if (b->gi[i].fmt == bcf_str2int("GL", 2)) {
+ int y = b->n_alleles * (b->n_alleles + 1) / 2;
+ memset((float*)b->gi[i].data + (k - 9) * y, 0, y * 4);
+ }
+ }
+ goto endblock;
+ }
+ for (q = kstrtok(p, ":", &a2), i = 0; q && i < b->n_gi; q = kstrtok(0, 0, &a2), ++i) {
+ if (b->gi[i].fmt == bcf_str2int("GT", 2)) {
+ ((uint8_t*)b->gi[i].data)[k-9] = (q[0] - '0')<<3 | (q[2] - '0') | (q[1] == '/'? 0 : 1) << 6;
+ } else if (b->gi[i].fmt == bcf_str2int("GQ", 2)) {
+ double _x = strtod(q, &q);
+ int x = (int)(_x + .499);
+ if (x > 255) x = 255;
+ ((uint8_t*)b->gi[i].data)[k-9] = x;
+ } else if (b->gi[i].fmt == bcf_str2int("DP", 2)) {
+ int x = strtol(q, &q, 10);
+ if (x > 0xffff) x = 0xffff;
+ ((uint16_t*)b->gi[i].data)[k-9] = x;
+ } else if (b->gi[i].fmt == bcf_str2int("PL", 2)) {
+ int x, y, j;
+ uint8_t *data = (uint8_t*)b->gi[i].data;
+ y = b->n_alleles * (b->n_alleles + 1) / 2;
+ for (j = 0; j < y; ++j) {
+ x = strtol(q, &q, 10);
+ if (x > 255) x = 255;
+ data[(k-9) * y + j] = x;
+ ++q;
+ }
+ } else if (b->gi[i].fmt == bcf_str2int("GL", 2)) {
+ int j, y;
+ float x, *data = (float*)b->gi[i].data;
+ y = b->n_alleles * (b->n_alleles + 1) / 2;
+ for (j = 0; j < y; ++j) {
+ x = strtod(q, &q);
+ data[(k-9) * y + j] = x;
+ ++q;
+ }
+ }
+ }
+ endblock: i = i;
+ }
+ }
+ h->l_nm = rn.l; h->name = rn.s;
+ if (sync) bcf_hdr_sync(h);
+ return v->line.l + 1;
+}
--- /dev/null
+#!/usr/bin/perl -w
+
+# Author: lh3
+
+use strict;
+use warnings;
+use Getopt::Std;
+
+&main;
+exit;
+
+sub main {
+ my $version = '0.1.0';
+ &usage if (@ARGV < 1);
+ my $command = shift(@ARGV);
+ my %func = (subsam=>\&subsam, listsam=>\&listsam, fillac=>\&fillac, qstats=>\&qstats, varFilter=>\&varFilter,
+ hapmap2vcf=>\&hapmap2vcf, ucscsnp2vcf=>\&ucscsnp2vcf, filter4vcf=>\&filter4vcf, ldstats=>\&ldstats);
+ die("Unknown command \"$command\".\n") if (!defined($func{$command}));
+ &{$func{$command}};
+}
+
+sub subsam {
+ die(qq/Usage: vcfutils.pl subsam <in.vcf> [samples]\n/) if (@ARGV == 0);
+ my ($fh, %h);
+ my $fn = shift(@ARGV);
+ my @col;
+ open($fh, ($fn =~ /\.gz$/)? "gzip -dc $fn |" : $fn) || die;
+ $h{$_} = 1 for (@ARGV);
+ while (<$fh>) {
+ if (/^##/) {
+ print;
+ } elsif (/^#/) {
+ my @t = split;
+ my @s = @t[0..8]; # all fixed fields + FORMAT
+ for (9 .. $#t) {
+ if ($h{$t[$_]}) {
+ push(@s, $t[$_]);
+ push(@col, $_);
+ }
+ }
+ pop(@s) if (@s == 9); # no sample selected; remove the FORMAT field
+ print join("\t", @s), "\n";
+ } else {
+ my @t = split;
+ if (@col == 0) {
+ print join("\t", @t[0..7]), "\n";
+ } else {
+ print join("\t", @t[0..8], map {$t[$_]} @col), "\n";
+ }
+ }
+ }
+ close($fh);
+}
+
+sub listsam {
+ die(qq/Usage: vcfutils.pl listsam <in.vcf>\n/) if (@ARGV == 0 && -t STDIN);
+ while (<>) {
+ if (/^#/ && !/^##/) {
+ my @t = split;
+ print join("\n", @t[9..$#t]), "\n";
+ exit;
+ }
+ }
+}
+
+sub fillac {
+ die(qq/Usage: vcfutils.pl fillac <in.vcf>\n\nNote: The GT field MUST BE present and always appear as the first field.\n/) if (@ARGV == 0 && -t STDIN);
+ while (<>) {
+ if (/^#/) {
+ print;
+ } else {
+ my @t = split;
+ my @c = (0);
+ my $n = 0;
+ my $s = -1;
+ @_ = split(":", $t[8]);
+ for (0 .. $#_) {
+ if ($_[$_] eq 'GT') { $s = $_; last; }
+ }
+ if ($s < 0) {
+ print join("\t", @t), "\n";
+ next;
+ }
+ for (9 .. $#t) {
+ if ($t[$_] =~ /^0,0,0/) {
+ } elsif ($t[$_] =~ /^([^\s:]+:){$s}(\d+).(\d+)/) {
+ ++$c[$2]; ++$c[$3];
+ $n += 2;
+ }
+ }
+ my $AC = "AC=" . join("\t", @c[1..$#c]) . ";AN=$n";
+ my $info = $t[7];
+ $info =~ s/(;?)AC=(\d+)//;
+ $info =~ s/(;?)AN=(\d+)//;
+ if ($info eq '.') {
+ $info = $AC;
+ } else {
+ $info .= ";$AC";
+ }
+ $t[7] = $info;
+ print join("\t", @t), "\n";
+ }
+ }
+}
+
+sub ldstats {
+ my %opts = (t=>0.9);
+ getopts('t:', \%opts);
+ die("Usage: vcfutils.pl ldstats [-t $opts{t}] <in.vcf>\n") if (@ARGV == 0 && -t STDIN);
+ my $cutoff = $opts{t};
+ my ($last, $lastchr) = (0x7fffffff, '');
+ my ($x, $y, $n) = (0, 0, 0);
+ while (<>) {
+ if (/^([^#\s]+)\s(\d+)/) {
+ my ($chr, $pos) = ($1, $2);
+ if (/NEIR=([\d\.]+)/) {
+ ++$n;
+ ++$y, $x += $pos - $last if ($lastchr eq $chr && $pos > $last && $1 > $cutoff);
+ }
+ $last = $pos; $lastchr = $chr;
+ }
+ }
+ print "Number of SNP intervals in strong LD (r > $opts{t}): $y\n";
+ print "Fraction: ", $y/$n, "\n";
+ print "Length: $x\n";
+}
+
+sub qstats {
+ my %opts = (r=>'', s=>0.02, v=>undef);
+ getopts('r:s:v', \%opts);
+ die("Usage: vcfutils.pl qstats [-r ref.vcf] <in.vcf>\n
+Note: This command discards indels. Output: QUAL #non-indel #SNPs #transitions #joint ts/tv #joint/#ref #joint/#non-indel \n") if (@ARGV == 0 && -t STDIN);
+ my %ts = (AG=>1, GA=>1, CT=>1, TC=>1);
+ my %h = ();
+ my $is_vcf = defined($opts{v})? 1 : 0;
+ if ($opts{r}) { # read the reference positions
+ my $fh;
+ open($fh, $opts{r}) || die;
+ while (<$fh>) {
+ next if (/^#/);
+ if ($is_vcf) {
+ my @t = split;
+ $h{$t[0],$t[1]} = $t[4];
+ } else {
+ $h{$1,$2} = 1 if (/^(\S+)\s+(\d+)/);
+ }
+ }
+ close($fh);
+ }
+ my $hsize = scalar(keys %h);
+ my @a;
+ while (<>) {
+ next if (/^#/);
+ my @t = split;
+ next if (length($t[3]) != 1 || uc($t[3]) eq 'N');
+ $t[3] = uc($t[3]); $t[4] = uc($t[4]);
+ my @s = split(',', $t[4]);
+ $t[5] = 3 if ($t[5] < 0);
+ next if (length($s[0]) != 1);
+ my $hit;
+ if ($is_vcf) {
+ $hit = 0;
+ my $aa = $h{$t[0],$t[1]};
+ if (defined($aa)) {
+ my @aaa = split(",", $aa);
+ for (@aaa) {
+ $hit = 1 if ($_ eq $s[0]);
+ }
+ }
+ } else {
+ $hit = defined($h{$t[0],$t[1]})? 1 : 0;
+ }
+ push(@a, [$t[5], ($t[4] eq '.' || $t[4] eq $t[3])? 0 : 1, $ts{$t[3].$s[0]}? 1 : 0, $hit]);
+ }
+ push(@a, [-1, 0, 0, 0]); # end marker
+ die("[qstats] No SNP data!\n") if (@a == 0);
+ @a = sort {$b->[0]<=>$a->[0]} @a;
+ my $next = $opts{s};
+ my $last = $a[0];
+ my @c = (0, 0, 0, 0);
+ my @lc;
+ $lc[1] = $lc[2] = 0;
+ for my $p (@a) {
+ if ($p->[0] == -1 || ($p->[0] != $last && $c[0]/@a > $next)) {
+ my @x;
+ $x[0] = sprintf("%.4f", $c[1]-$c[2]? $c[2] / ($c[1] - $c[2]) : 100);
+ $x[1] = sprintf("%.4f", $hsize? $c[3] / $hsize : 0);
+ $x[2] = sprintf("%.4f", $c[3] / $c[1]);
+ my $a = $c[1] - $lc[1];
+ my $b = $c[2] - $lc[2];
+ $x[3] = sprintf("%.4f", $a-$b? $b / ($a-$b) : 100);
+ print join("\t", $last, @c, @x), "\n";
+ $next = $c[0]/@a + $opts{s};
+ $lc[1] = $c[1]; $lc[2] = $c[2];
+ }
+ ++$c[0]; $c[1] += $p->[1]; $c[2] += $p->[2]; $c[3] += $p->[3];
+ $last = $p->[0];
+ }
+}
+
+sub varFilter {
+ my %opts = (d=>1, D=>10000, l=>30, Q=>25, q=>10, G=>25, s=>100, w=>10, W=>10, N=>2, p=>undef, F=>.001);
+ getopts('pq:d:D:l:Q:w:W:N:G:F:', \%opts);
+ die(qq/
+Usage: vcfutils.pl varFilter [options] <in.vcf>
+
+Options: -Q INT minimum RMS mapping quality for SNPs [$opts{Q}]
+ -q INT minimum RMS mapping quality for gaps [$opts{q}]
+ -d INT minimum read depth [$opts{d}]
+ -D INT maximum read depth [$opts{D}]
+
+ -G INT min indel score for nearby SNP filtering [$opts{G}]
+ -w INT SNP within INT bp around a gap to be filtered [$opts{w}]
+
+ -W INT window size for filtering dense SNPs [$opts{W}]
+ -N INT max number of SNPs in a window [$opts{N}]
+
+ -l INT window size for filtering adjacent gaps [$opts{l}]
+
+ -p print filtered variants
+\n/) if (@ARGV == 0 && -t STDIN);
+
+ # calculate the window size
+ my ($ol, $ow, $oW) = ($opts{l}, $opts{w}, $opts{W});
+ my $max_dist = $ol > $ow? $ol : $ow;
+ $max_dist = $oW if ($max_dist < $oW);
+ # the core loop
+ my @staging; # (indel_filtering_score, flt_tag)
+ while (<>) {
+ my @t = split;
+ next if (/^#/);
+ next if ($t[4] eq '.'); # skip non-var sites
+ my $is_snp = 1;
+ if (length($t[3]) > 1) {
+ $is_snp = 0;
+ } else {
+ my @s = split(',', $t[4]);
+ for (@s) {
+ $is_snp = 0 if (length > 1);
+ }
+ }
+ # clear the out-of-range elements
+ while (@staging) {
+ # Still on the same chromosome and the first element's window still affects this position?
+ last if ($staging[0][3] eq $t[0] && $staging[0][4] + $staging[0][2] + $max_dist >= $t[1]);
+ varFilter_aux(shift(@staging), $opts{p}); # calling a function is a bit slower, not much
+ }
+ my ($flt, $score) = (0, -1);
+
+ # collect key annotations
+ my ($dp, $mq, $af) = (-1, -1, 1);
+ if ($t[7] =~ /DP=(\d+)/i) {
+ $dp = $1;
+ } elsif ($t[7] =~ /DP4=(\d+),(\d+),(\d+),(\d+)/i) {
+ $dp = $1 + $2 + $3 + $4;
+ }
+ if ($t[7] =~ /MQ=(\d+)/i) {
+ $mq = $1;
+ }
+ if ($t[7] =~ /AF=([^\s;=]+)/i) {
+ $af = $1;
+ } elsif ($t[7] =~ /AF1=([^\s;=]+)/i) {
+ $af = $1;
+ }
+ # the depth filter
+ if ($dp >= 0) {
+ if ($dp < $opts{d}) {
+ $flt = 2;
+ } elsif ($dp > $opts{D}) {
+ $flt = 3;
+ }
+ }
+
+ # site dependent filters
+ my $dlen = 0;
+ if ($flt == 0) {
+ if (!$is_snp) { # an indel
+ # If deletion, remember the length of the deletion
+ $dlen = length($t[3]) - 1;
+ $flt = 1 if ($mq < $opts{q});
+ # filtering SNPs
+ if ($t[5] >= $opts{G}) {
+ for my $x (@staging) {
+ # Is it a SNP and is it outside the SNP filter window?
+ next if ($x->[0] >= 0 || $x->[4] + $x->[2] + $ow < $t[1]);
+ $x->[1] = 5 if ($x->[1] == 0);
+ }
+ }
+ # the indel filtering score
+ $score = $t[5];
+ # check the staging list for indel filtering
+ for my $x (@staging) {
+ # Is it a SNP and is it outside the gap filter window
+ next if ($x->[0] < 0 || $x->[4] + $x->[2] + $ol < $t[1]);
+ if ($x->[0] < $score) {
+ $x->[1] = 6;
+ } else {
+ $flt = 6; last;
+ }
+ }
+ } else { # a SNP
+ $flt = 1 if ($mq < $opts{Q});
+ # check adjacent SNPs
+ my $k = 1;
+ for my $x (@staging) {
+ ++$k if ($x->[0] < 0 && -($x->[0] + 1) > $opts{F} && $x->[4] + $x->[2] + $oW >= $t[1] && ($x->[1] == 0 || $x->[1] == 4 || $x->[1] == 5));
+ }
+ # filtering is necessary
+ if ($k > $opts{N}) {
+ $flt = 4;
+ for my $x (@staging) {
+ $x->[1] = 4 if ($x->[0] < 0 && $x->[4] + $x->[2] + $oW >= $t[1] && $x->[1] == 0);
+ }
+ } else { # then check gap filter
+ for my $x (@staging) {
+ next if ($x->[0] < 0 || $x->[4] + $x->[2] + $ow < $t[1]);
+ if ($x->[0] >= $opts{G}) {
+ $flt = 5; last;
+ }
+ }
+ }
+ }
+ }
+ push(@staging, [$score < 0? -$af-1 : $score, $flt, $dlen, @t]);
+ }
+ # output the last few elements in the staging list
+ while (@staging) {
+ varFilter_aux(shift @staging, $opts{p});
+ }
+}
+
+sub varFilter_aux {
+ my ($first, $is_print) = @_;
+ if ($first->[1] == 0) {
+ print join("\t", @$first[3 .. @$first-1]), "\n";
+ } elsif ($is_print) {
+ print STDERR join("\t", substr("UQdDWGgsiX", $first->[1], 1), @$first[3 .. @$first-1]), "\n";
+ }
+}
+
+sub filter4vcf {
+ my %opts = (d=>3, D=>2000, 1=>1e-4, 2=>1e-100, 3=>0, 4=>1e-4, Q=>10, q=>3);
+ getopts('d:D:1:2:3:4:Q:q:', \%opts);
+ die(qq/
+Usage: vcfutils.pl filter4vcf [options] <in.vcf>
+
+Options: -d INT min total depth (given DP or DP4) [$opts{d}]
+ -D INT max total depth [$opts{D}]
+ -q INT min SNP quality [$opts{q}]
+ -Q INT min RMS mapQ (given MQ) [$opts{Q}]
+ -1 FLOAT min P-value for strand bias (given PV4) [$opts{1}]
+ -2 FLOAT min P-value for baseQ bias [$opts{2}]
+ -3 FLOAT min P-value for mapQ bias [$opts{3}]
+ -4 FLOAT min P-value for end distance bias [$opts{4}]\n
+/) if (@ARGV == 0 && -t STDIN);
+
+ my %ts = (AG=>1, GA=>1, CT=>1, TC=>1);
+
+ my @n = (0, 0);
+ while (<>) {
+ if (/^#/) {
+ print;
+ next;
+ }
+ next if (/PV4=([^,]+),([^,]+),([^,]+),([^,;\t]+)/ && ($1<$opts{1} || $2<$opts{2} || $3<$opts{3} || $4<$opts{4}));
+ my $depth = -1;
+ $depth = $1 if (/DP=(\d+)/);
+ $depth = $1+$2+$3+$4 if (/DP4=(\d+),(\d+),(\d+),(\d+)/);
+ next if ($depth > 0 && ($depth < $opts{d} || $depth > $opts{D}));
+ next if (/MQ=(\d+)/ && $1 < $opts{Q});
+ my @t = split;
+ next if ($t[5] >= 0 && $t[5] < $opts{q});
+ ++$n[0];
+ my @s = split(',', $t[4]);
+ ++$n[1] if ($ts{$t[3].$s[0]});
+ print;
+ }
+}
+
+sub ucscsnp2vcf {
+ die("Usage: vcfutils.pl <in.ucsc.snp>\n") if (@ARGV == 0 && -t STDIN);
+ print "##fileformat=VCFv4.0\n";
+ print join("\t", "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO"), "\n";
+ while (<>) {
+ my @t = split("\t");
+ my $indel = ($t[9] =~ /^[ACGT](\/[ACGT])+$/)? 0 : 1;
+ my $pos = $t[2] + 1;
+ my @alt;
+ push(@alt, $t[7]);
+ if ($t[6] eq '-') {
+ $t[9] = reverse($t[9]);
+ $t[9] =~ tr/ACGTRYMKWSNacgtrymkwsn/TGCAYRKMWSNtgcayrkmwsn/;
+ }
+ my @a = split("/", $t[9]);
+ for (@a) {
+ push(@alt, $_) if ($_ ne $alt[0]);
+ }
+ if ($indel) {
+ --$pos;
+ for (0 .. $#alt) {
+ $alt[$_] =~ tr/-//d;
+ $alt[$_] = "N$alt[$_]";
+ }
+ }
+ my $ref = shift(@alt);
+ my $af = $t[13] > 0? ";AF=$t[13]" : '';
+ my $valid = ($t[12] eq 'unknown')? '' : ";valid=$t[12]";
+ my $info = "molType=$t[10];class=$t[11]$valid$af";
+ print join("\t", $t[1], $pos, $t[4], $ref, join(",", @alt), 0, '.', $info), "\n";
+ }
+}
+
+sub hapmap2vcf {
+ die("Usage: vcfutils.pl <in.ucsc.snp> <in.hapmap>\n") if (@ARGV == 0);
+ my $fn = shift(@ARGV);
+ # parse UCSC SNP
+ warn("Parsing UCSC SNPs...\n");
+ my ($fh, %map);
+ open($fh, ($fn =~ /\.gz$/)? "gzip -dc $fn |" : $fn) || die;
+ while (<$fh>) {
+ my @t = split;
+ next if ($t[3] - $t[2] != 1); # not SNP
+ @{$map{$t[4]}} = @t[1,3,7];
+ }
+ close($fh);
+ # write VCF
+ warn("Writing VCF...\n");
+ print "##fileformat=VCFv4.0\n";
+ while (<>) {
+ my @t = split;
+ if ($t[0] eq 'rs#') { # the first line
+ print join("\t", "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT", @t[11..$#t]), "\n";
+ } else {
+ next unless ($map{$t[0]});
+ next if (length($t[1]) != 3); # skip non-SNPs
+ my $a = \@{$map{$t[0]}};
+ my $ref = $a->[2];
+ my @u = split('/', $t[1]);
+ if ($u[1] eq $ref) {
+ $u[1] = $u[0]; $u[0] = $ref;
+ } elsif ($u[0] ne $ref) { next; }
+ my $alt = $u[1];
+ my %w;
+ $w{$u[0]} = 0; $w{$u[1]} = 1;
+ my @s = (@$a[0,1], $t[0], $ref, $alt, 0, '.', '.', 'GT');
+ my $is_tri = 0;
+ for (@t[11..$#t]) {
+ if ($_ eq 'NN') {
+ push(@s, './.');
+ } else {
+ my @a = ($w{substr($_,0,1)}, $w{substr($_,1,1)});
+ if (!defined($a[0]) || !defined($a[1])) {
+ $is_tri = 1;
+ last;
+ }
+ push(@s, "$a[0]/$a[1]");
+ }
+ }
+ next if ($is_tri);
+ print join("\t", @s), "\n";
+ }
+ }
+}
+
+sub usage {
+ die(qq/
+Usage: vcfutils.pl <command> [<arguments>]\n
+Command: subsam get a subset of samples
+ listsam list the samples
+ fillac fill the allele count field
+ qstats SNP stats stratified by QUAL
+ varFilter filtering short variants
+ filter4vcf filtering VCFs produced by samtools+bcftools
+ hapmap2vcf convert the hapmap format to VCF
+ ucscsnp2vcf convert UCSC SNP SQL dump to VCF
+\n/);
+}
bgzf_open(const char* __restrict path, const char* __restrict mode)
{
BGZF* fp = NULL;
- if (mode[0] == 'r' || mode[0] == 'R') { /* The reading mode is preferred. */
+ if (strchr(mode, 'r') || strchr(mode, 'R')) { /* The reading mode is preferred. */
#ifdef _USE_KNETFILE
knetFile *file = knet_open(path, mode);
if (file == 0) return 0;
if (fd == -1) return 0;
fp = open_read(fd);
#endif
- } else if (mode[0] == 'w' || mode[0] == 'W') {
+ } else if (strchr(mode, 'w') || strchr(mode, 'W')) {
int fd, oflag = O_WRONLY | O_CREAT | O_TRUNC;
#ifdef _WIN32
oflag |= O_BINARY;
#endif
fd = open(path, oflag, 0666);
if (fd == -1) return 0;
- fp = open_write(fd, strstr(mode, "u")? 1 : 0);
+ fp = open_write(fd, strchr(mode, 'u')? 1 : 0);
}
if (fp != NULL) fp->owned_file = 1;
return fp;
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
+#include <sys/select.h>
+#include <sys/stat.h>
#include "bgzf.h"
static const int WINDOW_SIZE = 64 * 1024;
static int bgzip_main_usage()
{
- printf("\n");
- printf("Usage: bgzip [options] [file] ...\n\n");
- printf("Options: -c write on standard output, keep original files unchanged\n");
- printf(" -d decompress\n");
- // printf(" -l list compressed file contents\n");
- printf(" -b INT decompress at virtual file pointer INT\n");
- printf(" -s INT decompress INT bytes in the uncompressed file\n");
- printf(" -h give this help\n");
- printf("\n");
- return 0;
+ fprintf(stderr, "\n");
+ fprintf(stderr, "Usage: bgzip [options] [file] ...\n\n");
+ fprintf(stderr, "Options: -c write on standard output, keep original files unchanged\n");
+ fprintf(stderr, " -d decompress\n");
+ fprintf(stderr, " -f overwrite files without asking\n");
+ fprintf(stderr, " -b INT decompress at virtual file pointer INT\n");
+ fprintf(stderr, " -s INT decompress INT bytes in the uncompressed file\n");
+ fprintf(stderr, " -h give this help\n");
+ fprintf(stderr, "\n");
+ return 1;
}
static int write_open(const char *fn, int is_forced)
char c;
if (!is_forced) {
if ((fd = open(fn, O_WRONLY | O_CREAT | O_TRUNC | O_EXCL, 0666)) < 0 && errno == EEXIST) {
- printf("bgzip: %s already exists; do you wish to overwrite (y or n)? ", fn);
+ fprintf(stderr, "[bgzip] %s already exists; do you wish to overwrite (y or n)? ", fn);
scanf("%c", &c);
if (c != 'Y' && c != 'y') {
- printf("bgzip: not overwritten\n");
+ fprintf(stderr, "[bgzip] not overwritten\n");
exit(1);
}
}
}
if (fd < 0) {
if ((fd = open(fn, O_WRONLY | O_CREAT | O_TRUNC, 0666)) < 0) {
- fprintf(stderr, "bgzip: %s: Fail to write\n", fn);
+ fprintf(stderr, "[bgzip] %s: Fail to write\n", fn);
exit(1);
}
}
return fd;
}
-static
-void
-fail(BGZF* fp)
+static void fail(BGZF* fp)
{
- printf("Error: %s\n", fp->error);
+ fprintf(stderr, "Error: %s\n", fp->error);
exit(1);
}
int main(int argc, char **argv)
{
int c, compress, pstdout, is_forced;
- BGZF *rz;
+ BGZF *fp;
void *buffer;
long start, end, size;
compress = 1; pstdout = 0; start = 0; size = -1; end = -1; is_forced = 0;
- while((c = getopt(argc, argv, "cdlhfb:s:")) >= 0){
+ while((c = getopt(argc, argv, "cdhfb:s:")) >= 0){
switch(c){
case 'h': return bgzip_main_usage();
case 'd': compress = 0; break;
case 'c': pstdout = 1; break;
- // case 'l': compress = 2; break;
case 'b': start = atol(optarg); break;
case 's': size = atol(optarg); break;
case 'f': is_forced = 1; break;
}
}
if (size >= 0) end = start + size;
- if(end >= 0 && end < start){
- fprintf(stderr, " -- Illegal region: [%ld, %ld] --\n", start, end);
+ if (end >= 0 && end < start) {
+ fprintf(stderr, "[bgzip] Illegal region: [%ld, %ld]\n", start, end);
return 1;
}
- if(compress == 1){
- int f_src, f_dst = -1;
- if(argc > optind){
- if((f_src = open(argv[optind], O_RDONLY)) < 0){
- fprintf(stderr, " -- Cannot open file: %s --\n", argv[optind]);
+ if (compress == 1) {
+ struct stat sbuf;
+ int f_src = fileno(stdin);
+ int f_dst = fileno(stdout);
+
+ if ( argc>optind )
+ {
+ if ( stat(argv[optind],&sbuf)<0 )
+ {
+ fprintf(stderr, "[bgzip] %s: %s\n", strerror(errno), argv[optind]);
+ return 1;
+ }
+
+ if ((f_src = open(argv[optind], O_RDONLY)) < 0) {
+ fprintf(stderr, "[bgzip] %s: %s\n", strerror(errno), argv[optind]);
return 1;
}
- if(pstdout){
+
+ if (pstdout)
f_dst = fileno(stdout);
- } else {
- char *name = malloc(sizeof(strlen(argv[optind]) + 5));
+ else
+ {
+ char *name = malloc(strlen(argv[optind]) + 5);
strcpy(name, argv[optind]);
strcat(name, ".gz");
f_dst = write_open(name, is_forced);
if (f_dst < 0) return 1;
free(name);
}
- } else if(pstdout){
- f_src = fileno(stdin);
- f_dst = fileno(stdout);
- } else return bgzip_main_usage();
- rz = bgzf_fdopen(f_dst, "w");
+ }
+ else if (!pstdout && isatty(fileno((FILE *)stdout)) )
+ return bgzip_main_usage();
+
+ fp = bgzf_fdopen(f_dst, "w");
buffer = malloc(WINDOW_SIZE);
- while((c = read(f_src, buffer, WINDOW_SIZE)) > 0) {
- if (bgzf_write(rz, buffer, c) < 0) {
- fail(rz);
- }
- }
- // f_dst will be closed here
- if (bgzf_close(rz) < 0) {
- fail(rz);
- }
- if (argc > optind) unlink(argv[optind]);
+ while ((c = read(f_src, buffer, WINDOW_SIZE)) > 0)
+ if (bgzf_write(fp, buffer, c) < 0) fail(fp);
+ // f_dst will be closed here
+ if (bgzf_close(fp) < 0) fail(fp);
+ if (argc > optind && !pstdout) unlink(argv[optind]);
free(buffer);
close(f_src);
return 0;
} else {
- if(argc <= optind) return bgzip_main_usage();
- int f_dst;
- if (argc > optind && !pstdout) {
- char *name;
- if (strstr(argv[optind], ".gz") - argv[optind] != strlen(argv[optind]) - 3) {
- printf("bgzip: %s: unknown suffix -- ignored\n", argv[optind]);
- return 1;
- }
- name = strdup(argv[optind]);
- name[strlen(name) - 3] = '\0';
- f_dst = write_open(name, is_forced);
- free(name);
- } else f_dst = fileno(stdout);
- rz = bgzf_open(argv[optind], "r");
- if (rz == NULL) {
- printf("Could not open file: %s\n", argv[optind]);
- return 1;
- }
- buffer = malloc(WINDOW_SIZE);
- if (bgzf_seek(rz, start, SEEK_SET) < 0) {
- fail(rz);
- }
- while(1){
- if(end < 0) c = bgzf_read(rz, buffer, WINDOW_SIZE);
- else c = bgzf_read(rz, buffer, (end - start > WINDOW_SIZE)? WINDOW_SIZE:(end - start));
- if(c == 0) break;
- if (c < 0) fail(rz);
- start += c;
- write(f_dst, buffer, c);
- if(end >= 0 && start >= end) break;
- }
- free(buffer);
- if (bgzf_close(rz) < 0) {
- fail(rz);
- }
- if (!pstdout) unlink(argv[optind]);
+ struct stat sbuf;
+ int f_dst;
+
+ if ( argc>optind )
+ {
+ if ( stat(argv[optind],&sbuf)<0 )
+ {
+ fprintf(stderr, "[bgzip] %s: %s\n", strerror(errno), argv[optind]);
+ return 1;
+ }
+ char *name;
+ int len = strlen(argv[optind]);
+ if ( strcmp(argv[optind]+len-3,".gz") )
+ {
+ fprintf(stderr, "[bgzip] %s: unknown suffix -- ignored\n", argv[optind]);
+ return 1;
+ }
+ fp = bgzf_open(argv[optind], "r");
+ if (fp == NULL) {
+ fprintf(stderr, "[bgzip] Could not open file: %s\n", argv[optind]);
+ return 1;
+ }
+
+ if (pstdout) {
+ f_dst = fileno(stdout);
+ }
+ else {
+ name = strdup(argv[optind]);
+ name[strlen(name) - 3] = '\0';
+ f_dst = write_open(name, is_forced);
+ free(name);
+ }
+ }
+ else if (!pstdout && isatty(fileno((FILE *)stdin)) )
+ return bgzip_main_usage();
+ else
+ {
+ f_dst = fileno(stdout);
+ fp = bgzf_fdopen(fileno(stdin), "r");
+ if (fp == NULL) {
+ fprintf(stderr, "[bgzip] Could not read from stdin: %s\n", strerror(errno));
+ return 1;
+ }
+ }
+ buffer = malloc(WINDOW_SIZE);
+ if (bgzf_seek(fp, start, SEEK_SET) < 0) fail(fp);
+ while (1) {
+ if (end < 0) c = bgzf_read(fp, buffer, WINDOW_SIZE);
+ else c = bgzf_read(fp, buffer, (end - start > WINDOW_SIZE)? WINDOW_SIZE:(end - start));
+ if (c == 0) break;
+ if (c < 0) fail(fp);
+ start += c;
+ write(f_dst, buffer, c);
+ if (end >= 0 && start >= end) break;
+ }
+ free(buffer);
+ if (bgzf_close(fp) < 0) fail(fp);
+ if (!pstdout) unlink(argv[optind]);
return 0;
}
}
-
--- /dev/null
+#include <math.h>
+#include "errmod.h"
+#include "ksort.h"
+KSORT_INIT_GENERIC(uint16_t)
+
+typedef struct __errmod_coef_t {
+ double *fk, *beta, *lhet;
+} errmod_coef_t;
+
+typedef struct {
+ double fsum[16], bsum[16];
+ uint32_t c[16];
+} call_aux_t;
+
+static errmod_coef_t *cal_coef(double depcorr, double eta)
+{
+ int k, n, q;
+ long double sum, sum1;
+ double *lC;
+ errmod_coef_t *ec;
+
+ ec = calloc(1, sizeof(errmod_coef_t));
+ // initialize ->fk
+ ec->fk = (double*)calloc(256, sizeof(double));
+ ec->fk[0] = 1.0;
+ for (n = 1; n != 256; ++n)
+ ec->fk[n] = pow(1. - depcorr, n) * (1.0 - eta) + eta;
+ // initialize ->coef
+ ec->beta = (double*)calloc(256 * 256 * 64, sizeof(double));
+ lC = (double*)calloc(256 * 256, sizeof(double));
+ for (n = 1; n != 256; ++n) {
+ double lgn = lgamma(n+1);
+ for (k = 1; k <= n; ++k)
+ lC[n<<8|k] = lgn - lgamma(k+1) - lgamma(n-k+1);
+ }
+ for (q = 1; q != 64; ++q) {
+ double e = pow(10.0, -q/10.0);
+ double le = log(e);
+ double le1 = log(1.0 - e);
+ for (n = 1; n <= 255; ++n) {
+ double *beta = ec->beta + (q<<16|n<<8);
+ sum1 = sum = 0.0;
+ for (k = n; k >= 0; --k, sum1 = sum) {
+ sum = sum1 + expl(lC[n<<8|k] + k*le + (n-k)*le1);
+ beta[k] = -10. / M_LN10 * logl(sum1 / sum);
+ }
+ }
+ }
+ // initialize ->lhet
+ ec->lhet = (double*)calloc(256 * 256, sizeof(double));
+ for (n = 0; n < 256; ++n)
+ for (k = 0; k < 256; ++k)
+ ec->lhet[n<<8|k] = lC[n<<8|k] - M_LN2 * n;
+ free(lC);
+ return ec;
+}
+
+errmod_t *errmod_init(float depcorr)
+{
+ errmod_t *em;
+ em = (errmod_t*)calloc(1, sizeof(errmod_t));
+ em->depcorr = depcorr;
+ em->coef = cal_coef(depcorr, 0.03);
+ return em;
+}
+
+void errmod_destroy(errmod_t *em)
+{
+ if (em == 0) return;
+ free(em->coef->lhet); free(em->coef->fk); free(em->coef->beta);
+ free(em->coef); free(em);
+}
+// qual:6, strand:1, base:4
+int errmod_cal(const errmod_t *em, int n, int m, uint16_t *bases, float *q)
+{
+ call_aux_t aux;
+ int i, j, k, w[32];
+
+ if (m > m) return -1;
+ memset(q, 0, m * m * sizeof(float));
+ if (n == 0) return 0;
+ // calculate aux.esum and aux.fsum
+ if (n > 255) { // then sample 255 bases
+ ks_shuffle(uint16_t, n, bases);
+ n = 255;
+ }
+ ks_introsort(uint16_t, n, bases);
+ memset(w, 0, 32 * sizeof(int));
+ memset(&aux, 0, sizeof(call_aux_t));
+ for (j = n - 1; j >= 0; --j) { // calculate esum and fsum
+ uint16_t b = bases[j];
+ int q = b>>5 < 4? 4 : b>>5;
+ if (q > 63) q = 63;
+ k = b&0x1f;
+ aux.fsum[k&0xf] += em->coef->fk[w[k]];
+ aux.bsum[k&0xf] += em->coef->fk[w[k]] * em->coef->beta[q<<16|n<<8|aux.c[k&0xf]];
+ ++aux.c[k&0xf];
+ ++w[k];
+ }
+ // generate likelihood
+ for (j = 0; j != m; ++j) {
+ float tmp1, tmp3;
+ int tmp2, bar_e;
+ // homozygous
+ for (k = 0, tmp1 = tmp3 = 0.0, tmp2 = 0; k != m; ++k) {
+ if (k == j) continue;
+ tmp1 += aux.bsum[k]; tmp2 += aux.c[k]; tmp3 += aux.fsum[k];
+ }
+ if (tmp2) {
+ bar_e = (int)(tmp1 / tmp3 + 0.499);
+ if (bar_e > 63) bar_e = 63;
+ q[j*m+j] = tmp1;
+ }
+ // heterozygous
+ for (k = j + 1; k < m; ++k) {
+ int cjk = aux.c[j] + aux.c[k];
+ for (i = 0, tmp2 = 0, tmp1 = tmp3 = 0.0; i < m; ++i) {
+ if (i == j || i == k) continue;
+ tmp1 += aux.bsum[i]; tmp2 += aux.c[i]; tmp3 += aux.fsum[i];
+ }
+ if (tmp2) {
+ bar_e = (int)(tmp1 / tmp3 + 0.499);
+ if (bar_e > 63) bar_e = 63;
+ q[j*m+k] = q[k*m+j] = -4.343 * em->coef->lhet[cjk<<8|aux.c[k]] + tmp1;
+ } else q[j*m+k] = q[k*m+j] = -4.343 * em->coef->lhet[cjk<<8|aux.c[k]]; // all the bases are either j or k
+ }
+ for (k = 0; k != m; ++k) if (q[j*m+k] < 0.0) q[j*m+k] = 0.0;
+ }
+ return 0;
+}
--- /dev/null
+#ifndef ERRMOD_H
+#define ERRMOD_H
+
+#include <stdint.h>
+
+struct __errmod_coef_t;
+
+typedef struct {
+ double depcorr;
+ struct __errmod_coef_t *coef;
+} errmod_t;
+
+errmod_t *errmod_init(float depcorr);
+void errmod_destroy(errmod_t *em);
+int errmod_cal(const errmod_t *em, int n, int m, uint16_t *bases, float *q);
+
+#endif
-all:../libbam.a ../samtools ex1.glf ex1.pileup.gz ex1.bam.bai ex1f-rmduppe.bam ex1f-rmdupse.bam ex1.glfview.gz calDepth
+all:../libbam.a ../samtools ../bcftools/bcftools \
+ ex1.glf ex1.pileup.gz ex1.bam.bai ex1f-rmduppe.bam ex1f-rmdupse.bam ex1.glfview.gz ex1.bcf calDepth
@echo; echo \# You can now launch the viewer with: \'samtools tview ex1.bam ex1.fa\'; echo;
ex1.fa.fai:ex1.fa
ex1b.bam:ex1.bam
../samtools view -h ex1.bam | awk 'BEGIN{FS=OFS="\t"}{if(/^@/)print;else{$$1=$$1"b";print}}' | ../samtools view -bS - > $@
ex1f.rg:
- (echo "@RG ID:ex1 LB:ex1"; echo "@RG ID:ex1a LB:ex1"; echo "@RG ID:ex1b LB:ex1b") > $@
+ (echo "@RG ID:ex1 LB:ex1 SM:ex1"; echo "@RG ID:ex1a LB:ex1 SM:ex1"; echo "@RG ID:ex1b LB:ex1b SM:ex1b") > $@
ex1f.bam:ex1.bam ex1a.bam ex1b.bam ex1f.rg
../samtools merge -rh ex1f.rg $@ ex1.bam ex1a.bam ex1b.bam
ex1f-rmduppe.bam:ex1f.bam
ex1f-rmdupse.bam:ex1f.bam
../samtools rmdup -S ex1f.bam $@
+ex1.bcf:ex1.bam ex1.fa.fai
+ ../samtools mpileup -gf ex1.fa ex1.bam > $@
+
+../bcftools/bcftools:
+ (cd ../bcftools; make bcftools)
+
../samtools:
(cd ..; make samtools)
gcc -g -Wall -O2 -I.. calDepth.c -o $@ -lm -lz -L.. -lbam
clean:
- rm -fr *.bam *.bai *.glf* *.fai *.pileup* *~ calDepth *.dSYM ex1*.rg
\ No newline at end of file
+ rm -fr *.bam *.bai *.glf* *.fai *.pileup* *~ calDepth *.dSYM ex1*.rg ex1.bcf
+
+# ../samtools pileup ex1.bam|perl -ape '$_=$F[4];s/(\d+)(??{".{$1}"})|\^.//g;@_=(tr/A-Z//,tr/a-z//);$_=join("\t",@F[0,1],@_)."\n"'
+
+# ../samtools pileup -cf ex1.fa ex1.bam|perl -ape '$_=$F[8];s/\^.//g;s/(\d+)(??{".{$1}"})|\^.//g;@_=(tr/A-Za-z//,tr/,.//);$_=join("\t",@F[0,1],@_)."\n"'
\ No newline at end of file
>ref
AGCATGTTAGATAAGATAGCTGTGCTAGTAGGCAGTCAGCGCCAT
+>ref2
+aggttttataaaacaattaagtctacagagcaactacgcg
@SQ SN:ref LN:45
-r001 163 ref 7 30 8M2I4M1D3M = 37 39 TTAGATAAAGGATACTG *
-r002 0 ref 9 30 1S2I6M1P1I4M2I * 0 0 AAAAGATAAGGATAAA *
+@SQ SN:ref2 LN:40
+r001 163 ref 7 30 8M4I4M1D3M = 37 39 TTAGATAAAGAGGATACTG *
+r002 0 ref 9 30 1S2I6M1P1I1P1I4M2I * 0 0 AAAAGATAAGGGATAAA *
r003 0 ref 9 30 5H6M * 0 0 AGCTAA *
r004 0 ref 16 30 6M14N1I5M * 0 0 ATAGCTCTCAGC *
r003 16 ref 29 30 6H5M * 0 0 TAGGC *
-r001 83 ref 37 30 9M = 7 -39 CAGCGCCAT *
\ No newline at end of file
+r001 83 ref 37 30 9M = 7 -39 CAGCGCCAT *
+x1 0 ref2 1 30 20M * 0 0 aggttttataaaacaaataa ????????????????????
+x2 0 ref2 2 30 21M * 0 0 ggttttataaaacaaataatt ?????????????????????
+x3 0 ref2 6 30 9M4I13M * 0 0 ttataaaacAAATaattaagtctaca ??????????????????????????
+x4 0 ref2 10 30 25M * 0 0 CaaaTaattaagtctacagagcaac ?????????????????????????
+x5 0 ref2 12 30 24M * 0 0 aaTaattaagtctacagagcaact ????????????????????????
+x6 0 ref2 14 30 23M * 0 0 Taattaagtctacagagcaacta ???????????????????????
#include <stdio.h>
#include <string.h>
#include <stdint.h>
+#include <math.h>
#include "kaln.h"
#define FROM_M 0
-2, -2, -2, -2, -2
};
-ka_param_t ka_param_blast = { 5, 2, 2, aln_sm_blast, 5, 50 };
-ka_param_t ka_param_aa2aa = { 10, 2, 2, aln_sm_blosum62, 22, 50 };
+ka_param_t ka_param_blast = { 5, 2, 5, 2, aln_sm_blast, 5, 50 };
+ka_param_t ka_param_aa2aa = { 10, 2, 10, 2, aln_sm_blosum62, 22, 50 };
static uint32_t *ka_path2cigar32(const path_t *path, int path_len, int *n_cigar)
{
}
#define set_end_I(II, cur, p) \
{ \
- if (gap_end >= 0) { \
- if ((p)->M - gap_open > (p)->I) { \
+ if (gap_end_ext >= 0) { \
+ if ((p)->M - gap_end_open > (p)->I) { \
(cur)->It = FROM_M; \
- (II) = (p)->M - gap_open - gap_end; \
+ (II) = (p)->M - gap_end_open - gap_end_ext; \
} else { \
(cur)->It = FROM_I; \
- (II) = (p)->I - gap_end; \
+ (II) = (p)->I - gap_end_ext; \
} \
} else set_I(II, cur, p); \
}
}
#define set_end_D(DD, cur, p) \
{ \
- if (gap_end >= 0) { \
- if ((p)->M - gap_open > (p)->D) { \
+ if (gap_end_ext >= 0) { \
+ if ((p)->M - gap_end_open > (p)->D) { \
(cur)->Dt = FROM_M; \
- (DD) = (p)->M - gap_open - gap_end; \
+ (DD) = (p)->M - gap_end_open - gap_end_ext; \
} else { \
(cur)->Dt = FROM_D; \
- (DD) = (p)->D - gap_end; \
+ (DD) = (p)->D - gap_end_ext; \
} \
} else set_D(DD, cur, p); \
}
uint8_t type, ctype;
uint32_t *cigar = 0;
- int gap_open, gap_ext, gap_end, b;
+ int gap_open, gap_ext, gap_end_open, gap_end_ext, b;
int *score_matrix, N_MATRIX_ROW;
/* initialize some align-related parameters. just for compatibility */
gap_open = ap->gap_open;
gap_ext = ap->gap_ext;
- gap_end = ap->gap_end;
+ gap_end_open = ap->gap_end_open;
+ gap_end_ext = ap->gap_end_ext;
b = ap->band_width;
score_matrix = ap->matrix;
N_MATRIX_ROW = ap->row;
return cigar;
}
+
+/*****************************************
+ * Probabilistic banded glocal alignment *
+ *****************************************/
+
+static float g_qual2prob[256];
+
+#define EI .25
+#define EM .3333333333333
+#define set_u(u, b, i, k) { int x=(i)-(b); x=x>0?x:0; (u)=((k)-x+1)*3; }
+
+ka_probpar_t ka_probpar_def = { 0.001, 0.1, 10 };
+
+/*
+ The topology of the profile HMM:
+
+ /\ /\ /\ /\
+ I[1] I[k-1] I[k] I[L]
+ ^ \ \ ^ \ ^ \ \ ^
+ | \ \ | \ | \ \ |
+ M[0] M[1] -> ... -> M[k-1] -> M[k] -> ... -> M[L] M[L+1]
+ \ \/ \/ \/ /
+ \ /\ /\ /\ /
+ -> D[k-1] -> D[k] ->
+
+ M[0] points to every {M,I}[k] and every {M,I}[k] points M[L+1].
+
+ On input, _ref is the reference sequence and _query is the query
+ sequence. Both are sequences of 0/1/2/3/4 where 4 stands for an
+ ambiguous residue. iqual is the base quality. c sets the gap open
+ probability, gap extension probability and band width.
+
+ On output, state and q are arrays of length l_query. The higher 30
+ bits give the reference position the query base is matched to and the
+ lower two bits can be 0 (an alignment match) or 1 (an
+ insertion). q[i] gives the phred scaled posterior probability of
+ state[i] being wrong.
+ */
+int ka_prob_glocal(const uint8_t *_ref, int l_ref, const uint8_t *_query, int l_query, const uint8_t *iqual,
+ const ka_probpar_t *c, int *state, uint8_t *q)
+{
+ double **f, **b, *s, m[9], sI, sM, bI, bM, pb;
+ float *qual, *_qual;
+ const uint8_t *ref, *query;
+ int bw, bw2, i, k, is_diff = 0;
+
+ /*** initialization ***/
+ ref = _ref - 1; query = _query - 1; // change to 1-based coordinate
+ bw = l_ref > l_query? l_ref : l_query;
+ if (bw > c->bw) bw = c->bw;
+ if (bw < abs(l_ref - l_query)) bw = abs(l_ref - l_query);
+ bw2 = bw * 2 + 1;
+ // allocate the forward and backward matrices f[][] and b[][] and the scaling array s[]
+ f = calloc(l_query+1, sizeof(void*));
+ b = calloc(l_query+1, sizeof(void*));
+ for (i = 0; i <= l_query; ++i) {
+ f[i] = calloc(bw2 * 3 + 6, sizeof(double)); // FIXME: this is over-allocated for very short seqs
+ b[i] = calloc(bw2 * 3 + 6, sizeof(double));
+ }
+ s = calloc(l_query+2, sizeof(double)); // s[] is the scaling factor to avoid underflow
+ // initialize qual
+ _qual = calloc(l_query, sizeof(float));
+ if (g_qual2prob[0] == 0)
+ for (i = 0; i < 256; ++i)
+ g_qual2prob[i] = pow(10, -i/10.);
+ for (i = 0; i < l_query; ++i) _qual[i] = g_qual2prob[iqual? iqual[i] : 30];
+ qual = _qual - 1;
+ // initialize transition probability
+ sM = sI = 1. / (2 * l_query + 2); // the value here seems not to affect results; FIXME: need proof
+ m[0*3+0] = (1 - c->d - c->d) * (1 - sM); m[0*3+1] = m[0*3+2] = c->d * (1 - sM);
+ m[1*3+0] = (1 - c->e) * (1 - sI); m[1*3+1] = c->e * (1 - sI); m[1*3+2] = 0.;
+ m[2*3+0] = 1 - c->e; m[2*3+1] = 0.; m[2*3+2] = c->e;
+ bM = (1 - c->d) / l_query; bI = c->d / l_query; // (bM+bI)*l_query==1
+ /*** forward ***/
+ // f[0]
+ set_u(k, bw, 0, 0);
+ f[0][k] = s[0] = 1.;
+ { // f[1]
+ double *fi = f[1], sum;
+ int beg = 1, end = l_ref < bw + 1? l_ref : bw + 1, _beg, _end;
+ for (k = beg, sum = 0.; k <= end; ++k) {
+ int u;
+ double e = (ref[k] > 3 || query[1] > 3)? 1. : ref[k] == query[1]? 1. - qual[1] : qual[1] * EM;
+ set_u(u, bw, 1, k);
+ fi[u+0] = e * bM; fi[u+1] = EI * bI;
+ sum += fi[u] + fi[u+1];
+ }
+ // rescale
+ s[1] = sum;
+ set_u(_beg, bw, 1, beg); set_u(_end, bw, 1, end); _end += 2;
+ for (k = _beg; k <= _end; ++k) fi[k] /= sum;
+ }
+ // f[2..l_query]
+ for (i = 2; i <= l_query; ++i) {
+ double *fi = f[i], *fi1 = f[i-1], sum, qli = qual[i];
+ int beg = 1, end = l_ref, x, _beg, _end;
+ uint8_t qyi = query[i];
+ x = i - bw; beg = beg > x? beg : x; // band start
+ x = i + bw; end = end < x? end : x; // band end
+ for (k = beg, sum = 0.; k <= end; ++k) {
+ int u, v11, v01, v10;
+ double e;
+ e = (ref[k] > 3 || qyi > 3)? 1. : ref[k] == qyi? 1. - qli : qli * EM;
+ set_u(u, bw, i, k); set_u(v11, bw, i-1, k-1); set_u(v10, bw, i-1, k); set_u(v01, bw, i, k-1);
+ fi[u+0] = e * (m[0] * fi1[v11+0] + m[3] * fi1[v11+1] + m[6] * fi1[v11+2]);
+ fi[u+1] = EI * (m[1] * fi1[v10+0] + m[4] * fi1[v10+1]);
+ fi[u+2] = m[2] * fi[v01+0] + m[8] * fi[v01+2];
+ sum += fi[u] + fi[u+1] + fi[u+2];
+// fprintf(stderr, "F (%d,%d;%d): %lg,%lg,%lg\n", i, k, u, fi[u], fi[u+1], fi[u+2]); // DEBUG
+ }
+ // rescale
+ s[i] = sum;
+ set_u(_beg, bw, i, beg); set_u(_end, bw, i, end); _end += 2;
+ for (k = _beg, sum = 1./sum; k <= _end; ++k) fi[k] *= sum;
+ }
+ { // f[l_query+1]
+ double sum;
+ for (k = 1, sum = 0.; k <= l_ref; ++k) {
+ int u;
+ set_u(u, bw, l_query, k);
+ if (u < 3 || u >= bw2*3+3) continue;
+ sum += f[l_query][u+0] * sM + f[l_query][u+1] * sI;
+ }
+ s[l_query+1] = sum; // the last scaling factor
+ }
+ /*** backward ***/
+ // b[l_query] (b[l_query+1][0]=1 and thus \tilde{b}[][]=1/s[l_query+1]; this is where s[l_query+1] comes from)
+ for (k = 1; k <= l_ref; ++k) {
+ int u;
+ double *bi = b[l_query];
+ set_u(u, bw, l_query, k);
+ if (u < 3 || u >= bw2*3+3) continue;
+ bi[u+0] = sM / s[l_query] / s[l_query+1]; bi[u+1] = sI / s[l_query] / s[l_query+1];
+ }
+ // b[l_query-1..1]
+ for (i = l_query - 1; i >= 1; --i) {
+ int beg = 1, end = l_ref, x, _beg, _end;
+ double *bi = b[i], *bi1 = b[i+1], y = (i > 1), qli1 = qual[i+1];
+ uint8_t qyi1 = query[i+1];
+ x = i - bw; beg = beg > x? beg : x;
+ x = i + bw; end = end < x? end : x;
+ for (k = end; k >= beg; --k) {
+ int u, v11, v01, v10;
+ double e;
+ set_u(u, bw, i, k); set_u(v11, bw, i+1, k+1); set_u(v10, bw, i+1, k); set_u(v01, bw, i, k+1);
+ e = (k >= l_ref? 0 : (ref[k+1] > 3 || qyi1 > 3)? 1. : ref[k+1] == qyi1? 1. - qli1 : qli1 * EM) * bi1[v11];
+ bi[u+0] = e * m[0] + EI * m[1] * bi1[v10+1] + m[2] * bi[v01+2]; // bi1[v11] has been foled into e.
+ bi[u+1] = e * m[3] + EI * m[4] * bi1[v10+1];
+ bi[u+2] = (e * m[6] + m[8] * bi[v01+2]) * y;
+// fprintf(stderr, "B (%d,%d;%d): %lg,%lg,%lg\n", i, k, u, bi[u], bi[u+1], bi[u+2]); // DEBUG
+ }
+ // rescale
+ set_u(_beg, bw, i, beg); set_u(_end, bw, i, end); _end += 2;
+ for (k = _beg, y = 1./s[i]; k <= _end; ++k) bi[k] *= y;
+ }
+ { // b[0]
+ int beg = 1, end = l_ref < bw + 1? l_ref : bw + 1;
+ double sum = 0.;
+ for (k = end; k >= beg; --k) {
+ int u;
+ double e = (ref[k] > 3 || query[1] > 3)? 1. : ref[k] == query[1]? 1. - qual[1] : qual[1] * EM;
+ set_u(u, bw, 1, k);
+ if (u < 3 || u >= bw2*3+3) continue;
+ sum += e * b[1][u+0] * bM + EI * b[1][u+1] * bI;
+ }
+ set_u(k, bw, 0, 0);
+ pb = b[0][k] = sum / s[0]; // if everything works as is expected, pb == 1.0
+ }
+ is_diff = fabs(pb - 1.) > 1e-7? 1 : 0;
+ /*** MAP ***/
+ for (i = 1; i <= l_query; ++i) {
+ double sum = 0., *fi = f[i], *bi = b[i], max = 0.;
+ int beg = 1, end = l_ref, x, max_k = -1;
+ x = i - bw; beg = beg > x? beg : x;
+ x = i + bw; end = end < x? end : x;
+ for (k = beg; k <= end; ++k) {
+ int u;
+ double z;
+ set_u(u, bw, i, k);
+ z = fi[u+0] * bi[u+0]; if (z > max) max = z, max_k = (k-1)<<2 | 0; sum += z;
+ z = fi[u+1] * bi[u+1]; if (z > max) max = z, max_k = (k-1)<<2 | 1; sum += z;
+ }
+ max /= sum; sum *= s[i]; // if everything works as is expected, sum == 1.0
+ if (state) state[i-1] = max_k;
+ if (q) k = (int)(-4.343 * log(1. - max) + .499), q[i-1] = k > 100? 99 : k;
+#ifdef _MAIN
+ fprintf(stderr, "(%.10lg,%.10lg) (%d,%d:%d)~%lg\n", pb, sum, i-1, max_k>>2, max_k&3, max); // DEBUG
+#endif
+ }
+ /*** free ***/
+ for (i = 0; i <= l_query; ++i) {
+ free(f[i]); free(b[i]);
+ }
+ free(f); free(b); free(s); free(_qual);
+ return 0;
+}
+
+#ifdef _MAIN
+int main()
+{
+ int l_ref = 5, l_query = 4;
+ uint8_t *ref = (uint8_t*)"\0\1\3\3\1";
+ uint8_t *query = (uint8_t*)"\0\3\3\1";
+// uint8_t *query = (uint8_t*)"\1\3\3\1";
+ static uint8_t qual[4] = {20, 20, 20, 20};
+ ka_prob_glocal(ref, l_ref, query, l_query, qual, &ka_probpar_def, 0, 0);
+ return 0;
+}
+#endif
typedef struct {
int gap_open;
int gap_ext;
- int gap_end;
+ int gap_end_open;
+ int gap_end_ext;
int *matrix;
int row;
int band_width;
} ka_param_t;
+typedef struct {
+ float d, e;
+ int bw;
+} ka_probpar_t;
+
#ifdef __cplusplus
extern "C" {
#endif
- uint32_t *ka_global_core(uint8_t *seq1, int len1, uint8_t *seq2, int len2, const ka_param_t *ap, int *_score, int *n_cigar);
+ uint32_t *ka_global_core(uint8_t *seq1, int len1, uint8_t *seq2, int len2, const ka_param_t *ap,
+ int *_score, int *n_cigar);
+ int ka_prob_glocal(const uint8_t *_ref, int l_ref, const uint8_t *_query, int l_query, const uint8_t *iqual,
+ const ka_probpar_t *c, int *state, uint8_t *q);
#ifdef __cplusplus
}
#endif
-extern ka_param_t ka_param_blast; /* = { 5, 2, 2, aln_sm_blast, 5, 50 }; */
+extern ka_param_t ka_param_blast; /* = { 5, 2, 5, 2, aln_sm_blast, 5, 50 }; */
+extern ka_probpar_t ka_probpar_def; /* { 0.0001, 0.1, 10 } */
#endif
if (hh <= k) low = ll; \
if (hh >= k) high = hh - 1; \
} \
+ } \
+ void ks_shuffle_##name(size_t n, type_t a[]) \
+ { \
+ int i, j; \
+ for (i = n; i > 1; --i) { \
+ type_t tmp; \
+ j = (int)(drand48() * i); \
+ tmp = a[j]; a[j] = a[i-1]; a[i-1] = tmp; \
+ } \
}
#define ks_mergesort(name, n, a, t) ks_mergesort_##name(n, a, t)
#define ks_heapmake(name, n, a) ks_heapmake_##name(n, a)
#define ks_heapadjust(name, i, n, a) ks_heapadjust_##name(i, n, a)
#define ks_ksmall(name, n, a, k) ks_ksmall_##name(n, a, k)
+#define ks_shuffle(name, n, a) ks_shuffle_##name(n, a)
#define ks_lt_generic(a, b) ((a) < (b))
#define ks_lt_str(a, b) (strcmp((a), (b)) < 0)
return l;
}
+char *kstrtok(const char *str, const char *sep, ks_tokaux_t *aux)
+{
+ const char *p, *start;
+ if (sep) { // set up the table
+ if (str == 0 && (aux->tab[0]&1)) return 0; // no need to set up if we have finished
+ aux->tab[0] = aux->tab[1] = aux->tab[2] = aux->tab[3] = 0;
+ for (p = sep; *p; ++p)
+ aux->tab[*p/64] |= 1ull<<(*p%64);
+ }
+ if (str) aux->p = str - 1, aux->tab[0] &= ~1ull;
+ else if (aux->tab[0]&1) return 0;
+ for (p = start = aux->p + 1; *p; ++p)
+ if (aux->tab[*p/64]>>(*p%64)&1) break;
+ aux->p = p; // end of token
+ if (*p == 0) aux->tab[0] |= 1; // no more tokens
+ return (char*)start;
+}
+
// s MUST BE a null terminated string; l = strlen(s)
int ksplit_core(char *s, int delimiter, int *_max, int **_offsets)
{
* Boyer-Moore search *
**********************/
+typedef unsigned char ubyte_t;
+
// reference: http://www-igm.univ-mlv.fr/~lecroq/string/node14.html
-int *ksBM_prep(const uint8_t *pat, int m)
+static int *ksBM_prep(const ubyte_t *pat, int m)
{
int i, *suff, *prep, *bmGs, *bmBc;
- prep = calloc(m + 256, 1);
+ prep = calloc(m + 256, sizeof(int));
bmGs = prep; bmBc = prep + m;
{ // preBmBc()
for (i = 0; i < 256; ++i) bmBc[i] = m;
return prep;
}
-int *ksBM_search(const uint8_t *str, int n, const uint8_t *pat, int m, int *_prep, int *n_matches)
+void *kmemmem(const void *_str, int n, const void *_pat, int m, int **_prep)
{
- int i, j, *prep, *bmGs, *bmBc;
- int *matches = 0, mm = 0, nm = 0;
- prep = _prep? _prep : ksBM_prep(pat, m);
+ int i, j, *prep = 0, *bmGs, *bmBc;
+ const ubyte_t *str, *pat;
+ str = (const ubyte_t*)_str; pat = (const ubyte_t*)_pat;
+ prep = (_prep == 0 || *_prep == 0)? ksBM_prep(pat, m) : *_prep;
+ if (_prep && *_prep == 0) *_prep = prep;
bmGs = prep; bmBc = prep + m;
j = 0;
while (j <= n - m) {
for (i = m - 1; i >= 0 && pat[i] == str[i+j]; --i);
- if (i < 0) {
- if (nm == mm) {
- mm = mm? mm<<1 : 1;
- matches = realloc(matches, mm * sizeof(int));
- }
- matches[nm++] = j;
- j += bmGs[0];
- } else {
+ if (i >= 0) {
int max = bmBc[str[i+j]] - m + 1 + i;
if (max < bmGs[i]) max = bmGs[i];
j += max;
- }
+ } else return (void*)(str + j);
}
- *n_matches = nm;
if (_prep == 0) free(prep);
- return matches;
+ return 0;
}
+char *kstrstr(const char *str, const char *pat, int **_prep)
+{
+ return (char*)kmemmem(str, strlen(str), pat, strlen(pat), _prep);
+}
+
+char *kstrnstr(const char *str, const char *pat, int n, int **_prep)
+{
+ return (char*)kmemmem(str, n, pat, strlen(pat), _prep);
+}
+
+/***********************
+ * The main() function *
+ ***********************/
+
#ifdef KSTRING_MAIN
#include <stdio.h>
int main()
{
kstring_t *s;
int *fields, n, i;
+ ks_tokaux_t aux;
+ char *p;
s = (kstring_t*)calloc(1, sizeof(kstring_t));
// test ksprintf()
ksprintf(s, " abcdefg: %d ", 100);
fields = ksplit(s, 0, &n);
for (i = 0; i < n; ++i)
printf("field[%d] = '%s'\n", i, s->s + fields[i]);
- free(s);
+ // test kstrtok()
+ s->l = 0;
+ for (p = kstrtok("ab:cde:fg/hij::k", ":/", &aux); p; p = kstrtok(0, 0, &aux)) {
+ kputsn(p, aux.p - p, s);
+ kputc('\n', s);
+ }
+ printf("%s", s->s);
+ // free
+ free(s->s); free(s); free(fields);
{
- static char *str = "abcdefgcdg";
+ static char *str = "abcdefgcdgcagtcakcdcd";
static char *pat = "cd";
- int n, *matches;
- matches = ksBM_search(str, strlen(str), pat, strlen(pat), 0, &n);
- printf("%d: \n", n);
- for (i = 0; i < n; ++i)
- printf("- %d\n", matches[i]);
- free(matches);
+ char *ret, *s = str;
+ int *prep = 0;
+ while ((ret = kstrstr(s, pat, &prep)) != 0) {
+ printf("match: %s\n", ret);
+ s = ret + prep[0];
+ }
+ free(prep);
}
return 0;
}
} kstring_t;
#endif
-int ksprintf(kstring_t *s, const char *fmt, ...);
-int ksplit_core(char *s, int delimiter, int *_max, int **_offsets);
+typedef struct {
+ uint64_t tab[4];
+ const char *p; // end of the current token
+} ks_tokaux_t;
-// calculate the auxiliary array, allocated by calloc()
-int *ksBM_prep(const uint8_t *pat, int m);
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+ int ksprintf(kstring_t *s, const char *fmt, ...);
+ int ksplit_core(char *s, int delimiter, int *_max, int **_offsets);
+ char *kstrstr(const char *str, const char *pat, int **_prep);
+ char *kstrnstr(const char *str, const char *pat, int n, int **_prep);
+ void *kmemmem(const void *_str, int n, const void *_pat, int m, int **_prep);
-/* Search pat in str and returned the list of matches. The size of the
- * list is returned as n_matches. _prep is the array returned by
- * ksBM_prep(). If it is a NULL pointer, ksBM_prep() will be called. */
-int *ksBM_search(const uint8_t *str, int n, const uint8_t *pat, int m, int *_prep, int *n_matches);
+ /* kstrtok() is similar to strtok_r() except that str is not
+ * modified and both str and sep can be NULL. For efficiency, it is
+ * actually recommended to set both to NULL in the subsequent calls
+ * if sep is not changed. */
+ char *kstrtok(const char *str, const char *sep, ks_tokaux_t *aux);
+#ifdef __cplusplus
+}
+#endif
+
static inline int kputsn(const char *p, int l, kstring_t *s)
{
if (s->l + l + 1 >= s->m) {
kroundup32(s->m);
s->s = (char*)realloc(s->s, s->m);
}
- strncpy(s->s + s->l, p, l);
+ memcpy(s->s + s->l, p, l);
s->l += l;
s->s[s->l] = 0;
return l;
--- /dev/null
+import java.io.*;
+import java.lang.*;
+
+public class HmmGlocal
+{
+ private double[] qual2prob;
+ private double cd, ce; // gap open probility [1e-3], gap extension probability [0.1]
+ private int cb; // band width [7]
+
+ public HmmGlocal(final double d, final double e, final int b) {
+ cd = d; ce = e; cb = b;
+ qual2prob = new double[256];
+ for (int i = 0; i < 256; ++i)
+ qual2prob[i] = Math.pow(10, -i/10.);
+ }
+ private static int set_u(final int b, final int i, final int k) {
+ int x = i - b;
+ x = x > 0? x : 0;
+ return (k + 1 - x) * 3;
+ }
+ public int hmm_glocal(final byte[] _ref, final byte[] _query, final byte[] _iqual, int[] state, byte[] q) {
+ int i, k;
+ /*** initialization ***/
+ // change coordinates
+ int l_ref = _ref.length;
+ byte[] ref = new byte[l_ref+1];
+ for (i = 0; i < l_ref; ++i) ref[i+1] = _ref[i]; // FIXME: this is silly...
+ int l_query = _query.length;
+ byte[] query = new byte[l_query+1];
+ double[] qual = new double[l_query+1];
+ for (i = 0; i < l_query; ++i) {
+ query[i+1] = _query[i];
+ qual[i+1] = qual2prob[_iqual[i]];
+ }
+ // set band width
+ int bw2, bw = l_ref > l_query? l_ref : l_query;
+ if (bw > cb) bw = cb;
+ if (bw < Math.abs(l_ref - l_query)) bw = Math.abs(l_ref - l_query);
+ bw2 = bw * 2 + 1;
+ // allocate the forward and backward matrices f[][] and b[][] and the scaling array s[]
+ double[][] f = new double[l_query+1][bw2*3 + 6];
+ double[][] b = new double[l_query+1][bw2*3 + 6];
+ double[] s = new double[l_query+2];
+ // initialize transition probabilities
+ double sM, sI, bM, bI;
+ sM = sI = 1. / (2 * l_query + 2);
+ bM = (1 - cd) / l_query; bI = cd / l_query; // (bM+bI)*l_query==1
+ double[] m = new double[9];
+ m[0*3+0] = (1 - cd - cd) * (1 - sM); m[0*3+1] = m[0*3+2] = cd * (1 - sM);
+ m[1*3+0] = (1 - ce) * (1 - sI); m[1*3+1] = ce * (1 - sI); m[1*3+2] = 0.;
+ m[2*3+0] = 1 - ce; m[2*3+1] = 0.; m[2*3+2] = ce;
+ /*** forward ***/
+ // f[0]
+ f[0][set_u(bw, 0, 0)] = s[0] = 1.;
+ { // f[1]
+ double[] fi = f[1];
+ double sum;
+ int beg = 1, end = l_ref < bw + 1? l_ref : bw + 1, _beg, _end;
+ for (k = beg, sum = 0.; k <= end; ++k) {
+ int u;
+ double e = (ref[k] > 3 || query[1] > 3)? 1. : ref[k] == query[1]? 1. - qual[1] : qual[1] / 3.;
+ u = set_u(bw, 1, k);
+ fi[u+0] = e * bM; fi[u+1] = .25 * bI;
+ sum += fi[u] + fi[u+1];
+ }
+ // rescale
+ s[1] = sum;
+ _beg = set_u(bw, 1, beg); _end = set_u(bw, 1, end); _end += 2;
+ for (k = _beg; k <= _end; ++k) fi[k] /= sum;
+ }
+ // f[2..l_query]
+ for (i = 2; i <= l_query; ++i) {
+ double[] fi = f[i], fi1 = f[i-1];
+ double sum, qli = qual[i];
+ int beg = 1, end = l_ref, x, _beg, _end;
+ byte qyi = query[i];
+ x = i - bw; beg = beg > x? beg : x; // band start
+ x = i + bw; end = end < x? end : x; // band end
+ for (k = beg, sum = 0.; k <= end; ++k) {
+ int u, v11, v01, v10;
+ double e;
+ e = (ref[k] > 3 || qyi > 3)? 1. : ref[k] == qyi? 1. - qli : qli / 3.;
+ u = set_u(bw, i, k); v11 = set_u(bw, i-1, k-1); v10 = set_u(bw, i-1, k); v01 = set_u(bw, i, k-1);
+ fi[u+0] = e * (m[0] * fi1[v11+0] + m[3] * fi1[v11+1] + m[6] * fi1[v11+2]);
+ fi[u+1] = .25 * (m[1] * fi1[v10+0] + m[4] * fi1[v10+1]);
+ fi[u+2] = m[2] * fi[v01+0] + m[8] * fi[v01+2];
+ sum += fi[u] + fi[u+1] + fi[u+2];
+ //System.out.println("("+i+","+k+";"+u+"): "+fi[u]+","+fi[u+1]+","+fi[u+2]);
+ }
+ // rescale
+ s[i] = sum;
+ _beg = set_u(bw, i, beg); _end = set_u(bw, i, end); _end += 2;
+ for (k = _beg, sum = 1./sum; k <= _end; ++k) fi[k] *= sum;
+ }
+ { // f[l_query+1]
+ double sum;
+ for (k = 1, sum = 0.; k <= l_ref; ++k) {
+ int u = set_u(bw, l_query, k);
+ if (u < 3 || u >= bw2*3+3) continue;
+ sum += f[l_query][u+0] * sM + f[l_query][u+1] * sI;
+ }
+ s[l_query+1] = sum; // the last scaling factor
+ }
+ /*** backward ***/
+ // b[l_query] (b[l_query+1][0]=1 and thus \tilde{b}[][]=1/s[l_query+1]; this is where s[l_query+1] comes from)
+ for (k = 1; k <= l_ref; ++k) {
+ int u = set_u(bw, l_query, k);
+ double[] bi = b[l_query];
+ if (u < 3 || u >= bw2*3+3) continue;
+ bi[u+0] = sM / s[l_query] / s[l_query+1]; bi[u+1] = sI / s[l_query] / s[l_query+1];
+ }
+ // b[l_query-1..1]
+ for (i = l_query - 1; i >= 1; --i) {
+ int beg = 1, end = l_ref, x, _beg, _end;
+ double[] bi = b[i], bi1 = b[i+1];
+ double y = (i > 1)? 1. : 0., qli1 = qual[i+1];
+ byte qyi1 = query[i+1];
+ x = i - bw; beg = beg > x? beg : x;
+ x = i + bw; end = end < x? end : x;
+ for (k = end; k >= beg; --k) {
+ int u, v11, v01, v10;
+ double e;
+ u = set_u(bw, i, k); v11 = set_u(bw, i+1, k+1); v10 = set_u(bw, i+1, k); v01 = set_u(bw, i, k+1);
+ e = (k >= l_ref? 0 : (ref[k+1] > 3 || qyi1 > 3)? 1. : ref[k+1] == qyi1? 1. - qli1 : qli1 / 3.) * bi1[v11];
+ bi[u+0] = e * m[0] + .25 * m[1] * bi1[v10+1] + m[2] * bi[v01+2]; // bi1[v11] has been foled into e.
+ bi[u+1] = e * m[3] + .25 * m[4] * bi1[v10+1];
+ bi[u+2] = (e * m[6] + m[8] * bi[v01+2]) * y;
+ }
+ // rescale
+ _beg = set_u(bw, i, beg); _end = set_u(bw, i, end); _end += 2;
+ for (k = _beg, y = 1./s[i]; k <= _end; ++k) bi[k] *= y;
+ }
+ double pb;
+ { // b[0]
+ int beg = 1, end = l_ref < bw + 1? l_ref : bw + 1;
+ double sum = 0.;
+ for (k = end; k >= beg; --k) {
+ int u = set_u(bw, 1, k);
+ double e = (ref[k] > 3 || query[1] > 3)? 1. : ref[k] == query[1]? 1. - qual[1] : qual[1] / 3.;
+ if (u < 3 || u >= bw2*3+3) continue;
+ sum += e * b[1][u+0] * bM + .25 * b[1][u+1] * bI;
+ }
+ pb = b[0][set_u(bw, 0, 0)] = sum / s[0]; // if everything works as is expected, pb == 1.0
+ }
+ int is_diff = Math.abs(pb - 1.) > 1e-7? 1 : 0;
+ /*** MAP ***/
+ for (i = 1; i <= l_query; ++i) {
+ double sum = 0., max = 0.;
+ double[] fi = f[i], bi = b[i];
+ int beg = 1, end = l_ref, x, max_k = -1;
+ x = i - bw; beg = beg > x? beg : x;
+ x = i + bw; end = end < x? end : x;
+ for (k = beg; k <= end; ++k) {
+ int u = set_u(bw, i, k);
+ double z;
+ sum += (z = fi[u+0] * bi[u+0]); if (z > max) { max = z; max_k = (k-1)<<2 | 0; }
+ sum += (z = fi[u+1] * bi[u+1]); if (z > max) { max = z; max_k = (k-1)<<2 | 1; }
+ }
+ max /= sum; sum *= s[i]; // if everything works as is expected, sum == 1.0
+ if (state != null) state[i-1] = max_k;
+ if (q != null) {
+ k = (int)(-4.343 * Math.log(1. - max) + .499);
+ q[i-1] = (byte)(k > 100? 99 : k);
+ }
+ //System.out.println("("+pb+","+sum+")"+" ("+(i-1)+","+(max_k>>2)+","+(max_k&3)+","+max+")");
+ }
+ return 0;
+ }
+
+ public static void main(String[] args) {
+ byte[] ref = {'\0', '\1', '\3', '\3', '\1'};
+ byte[] query = {'\0', '\3', '\3', '\1'};
+ byte[] qual = new byte[4];
+ qual[0] = qual[1] = qual[2] = qual[3] = (byte)20;
+ HmmGlocal hg = new HmmGlocal(1e-3, 0.1, 7);
+ hg.hmm_glocal(ref, query, qual, null, null);
+ }
+}
\ No newline at end of file
CC= gcc
CXX= g++
-CFLAGS= -g -Wall #-O2 #-m64 #-arch ppc
+CFLAGS= -g -Wall -O2 #-m64 #-arch ppc
CXXFLAGS= $(CFLAGS)
DFLAGS= -D_FILE_OFFSET_BITS=64
OBJS=
my $len=0;
if ($flt == 0) {
if ($t[2] eq '*') { # an indel
-
# If deletion, remember the length of the deletion
my ($a,$b) = split(m{/},$t[3]);
my $alen = length($a) - 1;
buffer = malloc(WINDOW_SIZE);
while((c = read(f_src, buffer, WINDOW_SIZE)) > 0) razf_write(rz, buffer, c);
razf_close(rz); // f_dst will be closed here
- if (argc > optind) unlink(argv[optind]);
+ if (argc > optind && !pstdout) unlink(argv[optind]);
free(buffer);
close(f_src);
return 0;
if (fn_list == 0 && fn_ref) fn_list = samfaipath(fn_ref);
// open file handlers
if ((in = samopen(argv[optind], in_mode, fn_list)) == 0) {
- fprintf(stderr, "[main_samview] fail to open file for reading.\n");
+ fprintf(stderr, "[main_samview] fail to open \"%s\" for reading.\n", argv[optind]);
+ ret = 1;
goto view_end;
}
if (in->header == 0) {
- fprintf(stderr, "[main_samview] fail to read the header.\n");
+ fprintf(stderr, "[main_samview] fail to read the header from \"%s\".\n", argv[optind]);
+ ret = 1;
goto view_end;
}
if ((out = samopen(fn_out? fn_out : "-", out_mode, in->header)) == 0) {
- fprintf(stderr, "[main_samview] fail to open file for writing.\n");
+ fprintf(stderr, "[main_samview] fail to open \"%s\" for writing.\n", fn_out? fn_out : "standard output");
+ ret = 1;
goto view_end;
}
if (is_header_only) goto view_end; // no need to print alignments
samwrite(out, b); // write the alignment to `out'
}
}
- if (r < -1) fprintf(stderr, "[main_samview] truncated file.\n");
+ if (r < -1) {
+ fprintf(stderr, "[main_samview] truncated file.\n");
+ ret = 1;
+ }
bam_destroy1(b);
} else { // retrieve alignments in specified regions
int i;
int tid, beg, end;
bam_parse_region(in->header, argv[i], &tid, &beg, &end); // parse a region in the format like `chr2:100-200'
if (tid < 0) { // reference name is not found
- fprintf(stderr, "[main_samview] fail to get the reference name. Continue anyway.\n");
+ fprintf(stderr, "[main_samview] region \"%s\" specifies an unknown reference name. Continue anyway.\n", argv[i]);
continue;
}
- bam_fetch(in->x.bam, idx, tid, beg, end, out, view_func); // fetch alignments
+ // fetch alignments
+ if (bam_fetch(in->x.bam, idx, tid, beg, end, out, view_func) < 0) {
+ fprintf(stderr, "[main_samview] retrieval of region \"%s\" failed due to truncated file or corrupt BAM index file\n", argv[i]);
+ ret = 1;
+ break;
+ }
}
bam_index_destroy(idx); // destroy the BAM index
}
--- /dev/null
+#include <stdlib.h>
+#include <string.h>
+#include "sample.h"
+#include "khash.h"
+KHASH_MAP_INIT_STR(sm, int)
+
+bam_sample_t *bam_smpl_init(void)
+{
+ bam_sample_t *s;
+ s = calloc(1, sizeof(bam_sample_t));
+ s->rg2smid = kh_init(sm);
+ s->sm2id = kh_init(sm);
+ return s;
+}
+
+void bam_smpl_destroy(bam_sample_t *sm)
+{
+ int i;
+ khint_t k;
+ khash_t(sm) *rg2smid = (khash_t(sm)*)sm->rg2smid;
+ if (sm == 0) return;
+ for (i = 0; i < sm->n; ++i) free(sm->smpl[i]);
+ free(sm->smpl);
+ for (k = kh_begin(rg2smid); k != kh_end(rg2smid); ++k)
+ if (kh_exist(rg2smid, k)) free((char*)kh_key(rg2smid, k));
+ kh_destroy(sm, sm->rg2smid);
+ kh_destroy(sm, sm->sm2id);
+ free(sm);
+}
+
+static void add_pair(bam_sample_t *sm, khash_t(sm) *sm2id, const char *key, const char *val)
+{
+ khint_t k_rg, k_sm;
+ int ret;
+ khash_t(sm) *rg2smid = (khash_t(sm)*)sm->rg2smid;
+ k_rg = kh_get(sm, rg2smid, key);
+ if (k_rg != kh_end(rg2smid)) return; // duplicated @RG-ID
+ k_rg = kh_put(sm, rg2smid, strdup(key), &ret);
+ k_sm = kh_get(sm, sm2id, val);
+ if (k_sm == kh_end(sm2id)) { // absent
+ if (sm->n == sm->m) {
+ sm->m = sm->m? sm->m<<1 : 1;
+ sm->smpl = realloc(sm->smpl, sizeof(void*) * sm->m);
+ }
+ sm->smpl[sm->n] = strdup(val);
+ k_sm = kh_put(sm, sm2id, sm->smpl[sm->n], &ret);
+ kh_val(sm2id, k_sm) = sm->n++;
+ }
+ kh_val(rg2smid, k_rg) = kh_val(sm2id, k_sm);
+}
+
+int bam_smpl_add(bam_sample_t *sm, const char *fn, const char *txt)
+{
+ const char *p = txt, *q, *r;
+ kstring_t buf;
+ int n = 0;
+ khash_t(sm) *sm2id = (khash_t(sm)*)sm->sm2id;
+ memset(&buf, 0, sizeof(kstring_t));
+ while ((q = strstr(p, "@RG")) != 0) {
+ p = q + 3;
+ r = q = 0;
+ if ((q = strstr(p, "\tID:")) != 0) q += 4;
+ if ((r = strstr(p, "\tSM:")) != 0) r += 4;
+ if (r && q) {
+ char *u, *v;
+ int oq, or;
+ for (u = (char*)q; *u && *u != '\t' && *u != '\n'; ++u);
+ for (v = (char*)r; *v && *v != '\t' && *v != '\n'; ++v);
+ oq = *u; or = *v; *u = *v = '\0';
+ buf.l = 0; kputs(fn, &buf); kputc('/', &buf); kputs(q, &buf);
+ add_pair(sm, sm2id, buf.s, r);
+ *u = oq; *v = or;
+ } else break;
+ p = q > r? q : r;
+ ++n;
+ }
+ if (n == 0) add_pair(sm, sm2id, fn, fn);
+ free(buf.s);
+ return 0;
+}
+
+int bam_smpl_rg2smid(const bam_sample_t *sm, const char *fn, const char *rg, kstring_t *str)
+{
+ khint_t k;
+ khash_t(sm) *rg2smid = (khash_t(sm)*)sm->rg2smid;
+ if (rg) {
+ str->l = 0;
+ kputs(fn, str); kputc('/', str); kputs(rg, str);
+ k = kh_get(sm, rg2smid, str->s);
+ } else k = kh_get(sm, rg2smid, fn);
+ return k == kh_end(rg2smid)? -1 : kh_val(rg2smid, k);
+}
--- /dev/null
+#ifndef BAM_SAMPLE_H
+#define BAM_SAMPLE_H
+
+#include "kstring.h"
+
+typedef struct {
+ int n, m;
+ char **smpl;
+ void *rg2smid, *sm2id;
+} bam_sample_t;
+
+bam_sample_t *bam_smpl_init(void);
+int bam_smpl_add(bam_sample_t *sm, const char *abs, const char *txt);
+int bam_smpl_rg2smid(const bam_sample_t *sm, const char *fn, const char *rg, kstring_t *str);
+void bam_smpl_destroy(bam_sample_t *sm);
+
+#endif
-.TH samtools 1 "11 July 2010" "samtools-0.1.8" "Bioinformatics tools"
+.TH samtools 1 "27 October 2010" "samtools-0.1.9" "Bioinformatics tools"
.SH NAME
.PP
samtools - Utilities for the Sequence Alignment/Map (SAM) format
.PP
samtools faidx ref.fasta
.PP
-samtools pileup -f ref.fasta aln.sorted.bam
+samtools pileup -vcf ref.fasta aln.sorted.bam
.PP
-samtools mpileup -f ref.fasta -r chr3:1,000-2,000 in1.bam in2.bam
+samtools mpileup -C50 -agf ref.fasta -r chr3:1,000-2,000 in1.bam in2.bam
.PP
samtools tview aln.sorted.bam ref.fasta
.B -b
Output in the BAM format.
.TP
-.B -u
-Output uncompressed BAM. This option saves time spent on
-compression/decomprssion and is thus preferred when the output is piped
-to another samtools command.
+.BI -f " INT"
+Only output alignments with all bits in INT present in the FLAG
+field. INT can be in hex in the format of /^0x[0-9A-F]+/ [0]
+.TP
+.BI -F " INT"
+Skip alignments with bits present in INT [0]
.TP
.B -h
Include the header in the output.
.B -H
Output the header only.
.TP
+.BI -l " STR"
+Only output reads in library STR [null]
+.TP
+.BI -o " FILE"
+Output file [stdout]
+.TP
+.BI -q " INT"
+Skip alignments with MAPQ smaller than INT [0]
+.TP
+.BI -r " STR"
+Only output reads in read group STR [null]
+.TP
+.BI -R " FILE"
+Output reads in read groups listed in
+.I FILE
+[null]
+.TP
.B -S
Input is in SAM. If @SQ header lines are absent, the
.B `-t'
option is required.
.TP
-.B -t FILE
+.BI -t " FILE"
This file is TAB-delimited. Each line must contain the reference name
and the length of the reference, one line for each distinct reference;
additional fields are ignored. This file also defines the order of the
.I <in.ref_list>
file.
.TP
-.B -o FILE
-Output file [stdout]
-.TP
-.B -f INT
-Only output alignments with all bits in INT present in the FLAG
-field. INT can be in hex in the format of /^0x[0-9A-F]+/ [0]
-.TP
-.B -F INT
-Skip alignments with bits present in INT [0]
-.TP
-.B -q INT
-Skip alignments with MAPQ smaller than INT [0]
-.TP
-.B -l STR
-Only output reads in library STR [null]
-.TP
-.B -r STR
-Only output reads in read group STR [null]
-.TP
-.B -R FILE
-Output reads in read groups listed in
-.I FILE
-[null]
+.B -u
+Output uncompressed BAM. This option saves time spent on
+compression/decomprssion and is thus preferred when the output is piped
+to another samtools command.
.RE
.TP
.TP
.B pileup
-samtools pileup [-f in.ref.fasta] [-t in.ref_list] [-l in.site_list]
-[-iscgS2] [-T theta] [-N nHap] [-r pairDiffRate] <in.bam>|<in.sam>
+samtools pileup [-2sSBicv] [-f in.ref.fasta] [-t in.ref_list] [-l
+in.site_list] [-C capMapQ] [-M maxMapQ] [-T theta] [-N nHap] [-r
+pairDiffRate] [-m mask] [-d maxIndelDepth] [-G indelPrior]
+<in.bam>|<in.sam>
Print the alignment in the pileup format. In the pileup format, each
line represents a genomic position, consisting of chromosome name,
mapping quality and start and end of a read are all encoded at the read
base column. At this column, a dot stands for a match to the reference
base on the forward strand, a comma for a match on the reverse strand,
-`ACGTN' for a mismatch on the forward strand and `acgtn' for a mismatch
-on the reverse strand. A pattern `\\+[0-9]+[ACGTNacgtn]+' indicates
-there is an insertion between this reference position and the next
-reference position. The length of the insertion is given by the integer
-in the pattern, followed by the inserted sequence. Similarly, a pattern
-`-[0-9]+[ACGTNacgtn]+' represents a deletion from the reference. The
-deleted bases will be presented as `*' in the following lines. Also at
-the read base column, a symbol `^' marks the start of a read segment
-which is a contiguous subsequence on the read separated by `N/S/H' CIGAR
-operations. The ASCII of the character following `^' minus 33 gives the
-mapping quality. A symbol `$' marks the end of a read segment.
+a '>' or '<' for a reference skip, `ACGTN' for a mismatch on the forward
+strand and `acgtn' for a mismatch on the reverse strand. A pattern
+`\\+[0-9]+[ACGTNacgtn]+' indicates there is an insertion between this
+reference position and the next reference position. The length of the
+insertion is given by the integer in the pattern, followed by the
+inserted sequence. Similarly, a pattern `-[0-9]+[ACGTNacgtn]+'
+represents a deletion from the reference. The deleted bases will be
+presented as `*' in the following lines. Also at the read base column, a
+symbol `^' marks the start of a read. The ASCII of the character
+following `^' minus 33 gives the mapping quality. A symbol `$' marks the
+end of a read segment.
If option
.B -c
.B OPTIONS:
.RS
.TP 10
-.B -s
-Print the mapping quality as the last column. This option makes the
-output easier to parse, although this format is not space efficient.
+.B -B
+Disable the BAQ computation. See the
+.B mpileup
+command for details.
.TP
-.B -S
-The input file is in SAM.
+.B -c
+Call the consensus sequence using SOAPsnp consensus model. Options
+.BR -T ", " -N ", " -I " and " -r
+are only effective when
+.BR -c " or " -g
+is in use.
.TP
-.B -i
-Only output pileup lines containing indels.
+.BI -C " INT"
+Coefficient for downgrading the mapping quality of poorly mapped
+reads. See the
+.B mpileup
+command for details. [0]
.TP
-.B -f FILE
+.BI -d " INT"
+Use the first
+.I NUM
+reads in the pileup for indel calling for speed up. Zero for unlimited. [1024]
+.TP
+.BI -f " FILE"
The reference sequence in the FASTA format. Index file
.I FILE.fai
will be created if
absent.
.TP
-.B -M INT
-Cap mapping quality at INT [60]
+.B -g
+Generate genotype likelihood in the binary GLFv3 format. This option
+suppresses -c, -i and -s. This option is deprecated by the
+.B mpileup
+command.
.TP
-.B -m INT
+.B -i
+Only output pileup lines containing indels.
+.TP
+.BI -I " INT"
+Phred probability of an indel in sequencing/prep. [40]
+.TP
+.BI -l " FILE"
+List of sites at which pileup is output. This file is space
+delimited. The first two columns are required to be chromosome and
+1-based coordinate. Additional columns are ignored. It is
+recommended to use option
+.TP
+.BI -m " INT"
Filter reads with flag containing bits in
-.I
-INT
+.I INT
[1796]
.TP
-.B -d INT
-Use the first
-.I NUM
-reads in the pileup for indel calling for speed up. Zero for unlimited. [0]
+.BI -M " INT"
+Cap mapping quality at INT [60]
+.TP
+.BI -N " INT"
+Number of haplotypes in the sample (>=2) [2]
.TP
-.B -t FILE
+.BI -r " FLOAT"
+Expected fraction of differences between a pair of haplotypes [0.001]
+.TP
+.B -s
+Print the mapping quality as the last column. This option makes the
+output easier to parse, although this format is not space efficient.
+.TP
+.B -S
+The input file is in SAM.
+.TP
+.BI -t " FILE"
List of reference names ane sequence lengths, in the format described
for the
.B import
command. If this option is present, samtools assumes the input
.I <in.alignment>
is in SAM format; otherwise it assumes in BAM format.
-.TP
-.B -l FILE
-List of sites at which pileup is output. This file is space
-delimited. The first two columns are required to be chromosome and
-1-based coordinate. Additional columns are ignored. It is
-recommended to use option
.B -s
together with
.B -l
as in the default format we may not know the mapping quality.
.TP
-.B -c
-Call the consensus sequence using SOAPsnp consensus model. Options
-.B -T,
-.B -N,
-.B -I
-and
-.B -r
-are only effective when
-.B -c
-or
-.B -g
-is in use.
-.TP
-.B -g
-Generate genotype likelihood in the binary GLFv3 format. This option
-suppresses -c, -i and -s.
-.TP
-.B -T FLOAT
+.BI -T " FLOAT"
The theta parameter (error dependency coefficient) in the maq consensus
calling model [0.85]
-.TP
-.B -N INT
-Number of haplotypes in the sample (>=2) [2]
-.TP
-.B -r FLOAT
-Expected fraction of differences between a pair of haplotypes [0.001]
-.TP
-.B -I INT
-Phred probability of an indel in sequencing/prep. [40]
.RE
.TP
.B mpileup
-samtools mpileup [-r reg] [-f in.fa] in.bam [in2.bam [...]]
+samtools mpileup [-Bug] [-C capQcoef] [-r reg] [-f in.fa] [-l list] [-M capMapQ] [-Q minBaseQ] [-q minMapQ] in.bam [in2.bam [...]]
-Generate pileup for multiple BAM files. Consensus calling is not
-implemented.
+Generate BCF or pileup for one or multiple BAM files. Alignment records
+are grouped by sample identifiers in @RG header lines. If sample
+identifiers are absent, each input file is regarded as one sample.
.B OPTIONS:
.RS
.TP 8
-.B -r STR
+.B -B
+Disable probabilistic realignment for the computation of base alignment
+quality (BAQ). BAQ is the Phred-scaled probability of a read base being
+misaligned. Applying this option greatly helps to reduce false SNPs
+caused by misalignments.
+.TP
+.BI -C " INT"
+Coefficient for downgrading mapping quality for reads containing
+excessive mismatches. Given a read with a phred-scaled probability q of
+being generated from the mapped position, the new mapping quality is
+about sqrt((INT-q)/INT)*INT. A zero value disables this
+functionality; if enabled, the recommended value is 50. [0]
+.TP
+.BI -f " FILE"
+The reference file [null]
+.TP
+.B -g
+Compute genotype likelihoods and output them in the binary call format (BCF).
+.TP
+.B -u
+Similar to
+.B -g
+except that the output is uncompressed BCF, which is preferred for pipeing.
+.TP
+.BI -l " FILE"
+File containing a list of sites where pileup or BCF is outputted [null]
+.TP
+.BI -q " INT"
+Minimum mapping quality for an alignment to be used [0]
+.TP
+.BI -Q " INT"
+Minimum base quality for a base to be considered [13]
+.TP
+.BI -r " STR"
Only generate pileup in region
.I STR
[all sites]
-.TP
-.B -f FILE
-The reference file [null]
.RE
.TP
.TP
.B merge
-samtools merge [-h inh.sam] [-nr] <out.bam> <in1.bam> <in2.bam> [...]
+samtools merge [-nur] [-h inh.sam] [-R reg] <out.bam> <in1.bam> <in2.bam> [...]
Merge multiple sorted alignments.
The header reference lists of all the input BAM files, and the @SQ headers of
.B OPTIONS:
.RS
.TP 8
-.B -h FILE
+.BI -h " FILE"
Use the lines of
.I FILE
as `@' headers to be copied to
is actually in SAM format, though any alignment records it may contain
are ignored.)
.TP
+.BI -R " STR"
+Merge files in the specified region indicated by
+.I STR
+.TP
.B -r
Attach an RG tag to each alignment. The tag value is inferred from file names.
.TP
.B -n
The input alignments are sorted by read names rather than by chromosomal
coordinates
+.TP
+.B -u
+Uncompressed BAM output
.RE
.TP
.TP
.B calmd
-samtools calmd [-eubS] <aln.bam> <ref.fasta>
+samtools calmd [-eubS
r] [-C capQcoef] <aln.bam> <ref.fasta>
Generate the MD tag. If the MD tag is already present, this command will
give a warning if the MD tag generated is different from the existing
.TP
.B -S
The input is SAM with header lines
+.TP
+.BI -C " INT"
+Coefficient to cap mapping quality of poorly mapped reads. See the
+.B pileup
+command for details. [0]
+.TP
+.B -r
+Perform probabilistic realignment to compute BAQ, which will be used to
+cap base quality.
.RE
.SH SAM FORMAT
0x0400 d the read is either a PCR or an optical duplicate
.TE
+.SH EXAMPLES
+.IP o 2
+Import SAM to BAM when
+.B @SQ
+lines are present in the header:
+
+ samtools view -bS aln.sam > aln.bam
+
+If
+.B @SQ
+lines are absent:
+
+ samtools faidx ref.fa
+ samtools view -bt ref.fa.fai aln.sam > aln.bam
+
+where
+.I ref.fa.fai
+is generated automatically by the
+.B faidx
+command.
+
+.IP o 2
+Attach the
+.B RG
+tag while merging sorted alignments:
+
+ perl -e 'print "@RG\\tID:ga\\tSM:hs\\tLB:ga\\tPL:Illumina\\n@RG\\tID:454\\tSM:hs\\tLB:454\\tPL:454\\n"' > rg.txt
+ samtools merge -rh rg.txt merged.bam ga.bam 454.bam
+
+The value in a
+.B RG
+tag is determined by the file name the read is coming from. In this
+example, in the
+.IR merged.bam ,
+reads from
+.I ga.bam
+will be attached
+.IR RG:Z:ga ,
+while reads from
+.I 454.bam
+will be attached
+.IR RG:Z:454 .
+
+.IP o 2
+Call SNPs and short indels for one diploid individual:
+
+ samtools pileup -vcf ref.fa aln.bam > var.raw.plp
+ samtools.pl varFilter -D 100 var.raw.plp > var.flt.plp
+ awk '($3=="*"&&$6>=50)||($3!="*"&&$6>=20)' var.flt.plp > var.final.plp
+
+The
+.B -D
+option of varFilter controls the maximum read depth, which should be
+adjusted to about twice the average read depth. One may consider to add
+.B -C50
+to
+.B pileup
+if mapping quality is overestimated for reads containing excessive
+mismatches. Applying this option usually helps
+.B BWA-short
+but may not other mappers. It also potentially increases reference
+biases.
+
+.IP o 2
+Call SNPs (not short indels) for multiple diploid individuals:
+
+ samtools mpileup -augf ref.fa *.bam | bcftools view -bcv - > snp.raw.bcf
+ bcftools view snp.raw.bcf | vcfutils.pl filter4vcf -D 2000 | bgzip > snp.flt.vcf.gz
+
+Individuals are identified from the
+.B SM
+tags in the
+.B @RG
+header lines. Individuals can be pooled in one alignment file; one
+individual can also be separated into multiple files. Similarly, one may
+consider to apply
+.B -C50
+to
+.BR mpileup .
+SNP calling in this way also works for single sample and has the
+advantage of enabling more powerful filtering. The drawback is the lack
+of short indel calling, which may be implemented in future.
+
+.IP o 2
+Derive the allele frequency spectrum (AFS) on a list of sites from multiple individuals:
+
+ samtools mpileup -gf ref.fa *.bam > all.bcf
+ bcftools view -bl sites.list all.bcf > sites.bcf
+ bcftools view -cGP cond2 sites.bcf > /dev/null 2> sites.1.afs
+ bcftools view -cGP sites.1.afs sites.bcf > /dev/null 2> sites.2.afs
+ bcftools view -cGP sites.2.afs sites.bcf > /dev/null 2> sites.3.afs
+ ......
+
+where
+.I sites.list
+contains the list of sites with each line consisting of the reference
+sequence name and position. The following
+.B bcftools
+commands estimate AFS by EM.
+
+.IP o 2
+Dump BAQ applied alignment for other SNP callers:
+
+ samtools calmd -br aln.bam > aln.baq.bam
+
+It adds and corrects the
+.B NM
+and
+.B MD
+tags at the same time. The
+.B calmd
+command also comes with the
+.B -C
+option, the same as the on in
+.B pileup
+and
+.BR mpileup .
+Apply if it helps.
+
.SH LIMITATIONS
.PP
.IP o 2
samtools faidx ref.fasta
- samtools pileup -f ref.fasta aln.sorted.bam
+ samtools pileup -vcf ref.fasta aln.sorted.bam
- samtools mpileup -f ref.fasta -r chr3:1,000-2,000 in1.bam in2.bam
+ samtools mpileup -C50 -agf ref.fasta -r chr3:1,000-2,000 in1.bam
+ in2.bam
samtools tview aln.sorted.bam ref.fasta
DESCRIPTION
- Samtools is a set of utilities that manipulate alignments in the BAM
+ Samtools is a set of utilities that manipulate alignments in the BAM
format. It imports from and exports to the SAM (Sequence Alignment/Map)
- format, does sorting, merging and indexing, and allows to retrieve
+ format, does sorting, merging and indexing, and allows to retrieve
reads in any regions swiftly.
- Samtools is designed to work on a stream. It regards an input file `-'
- as the standard input (stdin) and an output file `-' as the standard
+ Samtools is designed to work on a stream. It regards an input file `-'
+ as the standard input (stdin) and an output file `-' as the standard
output (stdout). Several commands can thus be combined with Unix pipes.
Samtools always output warning and error messages to the standard error
output (stderr).
- Samtools is also able to open a BAM (not SAM) file on a remote FTP or
- HTTP server if the BAM file name starts with `ftp://' or `http://'.
- Samtools checks the current working directory for the index file and
- will download the index upon absence. Samtools does not retrieve the
+ Samtools is also able to open a BAM (not SAM) file on a remote FTP or
+ HTTP server if the BAM file name starts with `ftp://' or `http://'.
+ Samtools checks the current working directory for the index file and
+ will download the index upon absence. Samtools does not retrieve the
entire alignment file unless it is asked to do so.
COMMANDS AND OPTIONS
view samtools view [-bhuHS] [-t in.refList] [-o output] [-f
- reqFlag] [-F skipFlag] [-q minMapQ] [-l library] [-r read-
+ reqFlag] [-F skipFlag] [-q minMapQ] [-l library] [-r read-
Group] [-R rgFile] <in.bam>|<in.sam> [region1 [...]]
- Extract/print all or sub alignments in SAM or BAM format. If
- no region is specified, all the alignments will be printed;
- otherwise only alignments overlapping the specified regions
- will be output. An alignment may be given multiple times if
+ Extract/print all or sub alignments in SAM or BAM format. If
+ no region is specified, all the alignments will be printed;
+ otherwise only alignments overlapping the specified regions
+ will be output. An alignment may be given multiple times if
it is overlapping several regions. A region can be presented,
- for example, in the following format: `chr2' (the whole
- chr2), `chr2:1000000' (region starting from 1,000,000bp) or
- `chr2:1,000,000-2,000,000' (region between 1,000,000 and
- 2,000,000bp including the end points). The coordinate is
+ for example, in the following format: `chr2' (the whole
+ chr2), `chr2:1000000' (region starting from 1,000,000bp) or
+ `chr2:1,000,000-2,000,000' (region between 1,000,000 and
+ 2,000,000bp including the end points). The coordinate is
1-based.
OPTIONS:
-b Output in the BAM format.
- -u Output uncompressed BAM. This option saves time spent
- on compression/decomprssion and is thus preferred
- when the output is piped to another samtools command.
+ -f INT Only output alignments with all bits in INT present
+ in the FLAG field. INT can be in hex in the format of
+ /^0x[0-9A-F]+/ [0]
+
+ -F INT Skip alignments with bits present in INT [0]
-h Include the header in the output.
-H Output the header only.
+ -l STR Only output reads in library STR [null]
+
+ -o FILE Output file [stdout]
+
+ -q INT Skip alignments with MAPQ smaller than INT [0]
+
+ -r STR Only output reads in read group STR [null]
+
+ -R FILE Output reads in read groups listed in FILE [null]
+
-S Input is in SAM. If @SQ header lines are absent, the
`-t' option is required.
`samtools faidx <ref.fa>', the resultant index file
<ref.fa>.fai can be used as this <in.ref_list> file.
- -o FILE Output file [stdout]
-
- -f INT Only output alignments with all bits in INT present
- in the FLAG field. INT can be in hex in the format of
- /^0x[0-9A-F]+/ [0]
-
- -F INT Skip alignments with bits present in INT [0]
-
- -q INT Skip alignments with MAPQ smaller than INT [0]
-
- -l STR Only output reads in library STR [null]
-
- -r STR Only output reads in read group STR [null]
-
- -R FILE Output reads in read groups listed in FILE [null]
+ -u Output uncompressed BAM. This option saves time spent
+ on compression/decomprssion and is thus preferred
+ when the output is piped to another samtools command.
tview samtools tview <in.sorted.bam> [ref.fasta]
- Text alignment viewer (based on the ncurses library). In the
- viewer, press `?' for help and press `g' to check the align-
- ment start from a region in the format like
- `chr10:10,000,000' or `=10,000,000' when viewing the same
+ Text alignment viewer (based on the ncurses library). In the
+ viewer, press `?' for help and press `g' to check the align-
+ ment start from a region in the format like
+ `chr10:10,000,000' or `=10,000,000' when viewing the same
reference sequence.
- pileup samtools pileup [-f in.ref.fasta] [-t in.ref_list] [-l
- in.site_list] [-iscgS2] [-T theta] [-N nHap] [-r
- pairDiffRate] <in.bam>|<in.sam>
+ pileup samtools pileup [-2sSBicv] [-f in.ref.fasta] [-t in.ref_list]
+ [-l in.site_list] [-C capMapQ] [-M maxMapQ] [-T theta] [-N
+ nHap] [-r pairDiffRate] [-m mask] [-d maxIndelDepth] [-G
+ indelPrior] <in.bam>|<in.sam>
Print the alignment in the pileup format. In the pileup for-
mat, each line represents a genomic position, consisting of
end of a read are all encoded at the read base column. At
this column, a dot stands for a match to the reference base
on the forward strand, a comma for a match on the reverse
- strand, `ACGTN' for a mismatch on the forward strand and
- `acgtn' for a mismatch on the reverse strand. A pattern
- `\+[0-9]+[ACGTNacgtn]+' indicates there is an insertion
- between this reference position and the next reference posi-
- tion. The length of the insertion is given by the integer in
- the pattern, followed by the inserted sequence. Similarly, a
- pattern `-[0-9]+[ACGTNacgtn]+' represents a deletion from the
- reference. The deleted bases will be presented as `*' in the
- following lines. Also at the read base column, a symbol `^'
- marks the start of a read segment which is a contiguous sub-
- sequence on the read separated by `N/S/H' CIGAR operations.
- The ASCII of the character following `^' minus 33 gives the
- mapping quality. A symbol `$' marks the end of a read seg-
- ment.
+ strand, a '>' or '<' for a reference skip, `ACGTN' for a mis-
+ match on the forward strand and `acgtn' for a mismatch on the
+ reverse strand. A pattern `\+[0-9]+[ACGTNacgtn]+' indicates
+ there is an insertion between this reference position and the
+ next reference position. The length of the insertion is given
+ by the integer in the pattern, followed by the inserted
+ sequence. Similarly, a pattern `-[0-9]+[ACGTNacgtn]+' repre-
+ sents a deletion from the reference. The deleted bases will
+ be presented as `*' in the following lines. Also at the read
+ base column, a symbol `^' marks the start of a read. The
+ ASCII of the character following `^' minus 33 gives the map-
+ ping quality. A symbol `$' marks the end of a read segment.
If option -c is applied, the consensus base, Phred-scaled
consensus quality, SNP quality (i.e. the Phred-scaled proba-
OPTIONS:
- -s Print the mapping quality as the last column. This
- option makes the output easier to parse, although
- this format is not space efficient.
+ -B Disable the BAQ computation. See the mpileup com-
+ mand for details.
- -S The input file is in SAM.
+ -c Call the consensus sequence using SOAPsnp consensus
+ model. Options -T, -N, -I and -r are only effective
+ when -c or -g is in use.
- -i Only output pileup lines containing indels.
+ -C INT Coefficient for downgrading the mapping quality of
+ poorly mapped reads. See the mpileup command for
+ details. [0]
+
+ -d INT Use the first NUM reads in the pileup for indel
+ calling for speed up. Zero for unlimited. [1024]
-f FILE The reference sequence in the FASTA format. Index
file FILE.fai will be created if absent.
- -M INT Cap mapping quality at INT [60]
+ -g Generate genotype likelihood in the binary GLFv3
+ format. This option suppresses -c, -i and -s. This
+ option is deprecated by the mpileup command.
- -m INT Filter reads with flag containing bits in INT
- [1796]
-
- -d INT Use the first NUM reads in the pileup for indel
- calling for speed up. Zero for unlimited. [0]
-
- -t FILE List of reference names ane sequence lengths, in
- the format described for the import command. If
- this option is present, samtools assumes the input
- <in.alignment> is in SAM format; otherwise it
- assumes in BAM format.
+ -i Only output pileup lines containing indels.
- -l FILE List of sites at which pileup is output. This file
- is space delimited. The first two columns are
- required to be chromosome and 1-based coordinate.
- Additional columns are ignored. It is recommended
- to use option -s together with -l as in the default
- format we may not know the mapping quality.
+ -I INT Phred probability of an indel in sequencing/prep.
+ [40]
- -c Call the consensus sequence using SOAPsnp consensus
- model. Options -T, -N, -I and -r are only effective
- when -c or -g is in use.
+ -l FILE List of sites at which pileup is output. This file
+ is space delimited. The first two columns are
+ required to be chromosome and 1-based coordinate.
+ Additional columns are ignored. It is recommended
+ to use option
- -g Generate genotype likelihood in the binary GLFv3
- format. This option suppresses -c, -i and -s.
+ -m INT Filter reads with flag containing bits in INT
+ [1796]
- -T FLOAT The theta parameter (error dependency coefficient)
- in the maq consensus calling model [0.85]
+ -M INT Cap mapping quality at INT [60]
-N INT Number of haplotypes in the sample (>=2) [2]
- -r FLOAT Expected fraction of differences between a pair of
+ -r FLOAT Expected fraction of differences between a pair of
haplotypes [0.001]
- -I INT Phred probability of an indel in sequencing/prep.
- [40]
+ -s Print the mapping quality as the last column. This
+ option makes the output easier to parse, although
+ this format is not space efficient.
+
+ -S The input file is in SAM.
+
+ -t FILE List of reference names ane sequence lengths, in
+ the format described for the import command. If
+ this option is present, samtools assumes the input
+ <in.alignment> is in SAM format; otherwise it
+ assumes in BAM format. -s together with -l as in
+ the default format we may not know the mapping
+ quality.
+
+ -T FLOAT The theta parameter (error dependency coefficient)
+ in the maq consensus calling model [0.85]
- mpileup samtools mpileup [-r reg] [-f in.fa] in.bam [in2.bam [...]]
+ mpileup samtools mpileup [-Bug] [-C capQcoef] [-r reg] [-f in.fa] [-l
+ list] [-M capMapQ] [-Q minBaseQ] [-q minMapQ] in.bam [in2.bam
+ [...]]
- Generate pileup for multiple BAM files. Consensus calling is
- not implemented.
+ Generate BCF or pileup for one or multiple BAM files. Align-
+ ment records are grouped by sample identifiers in @RG header
+ lines. If sample identifiers are absent, each input file is
+ regarded as one sample.
OPTIONS:
- -r STR Only generate pileup in region STR [all sites]
+ -B Disable probabilistic realignment for the computation
+ of base alignment quality (BAQ). BAQ is the Phred-
+ scaled probability of a read base being misaligned.
+ Applying this option greatly helps to reduce false
+ SNPs caused by misalignments.
+
+ -C INT Coefficient for downgrading mapping quality for reads
+ containing excessive mismatches. Given a read with a
+ phred-scaled probability q of being generated from
+ the mapped position, the new mapping quality is about
+ sqrt((INT-q)/INT)*INT. A zero value disables this
+ functionality; if enabled, the recommended value is
+ 50. [0]
-f FILE The reference file [null]
+ -g Compute genotype likelihoods and output them in the
+ binary call format (BCF).
+
+ -u Similar to -g except that the output is uncompressed
+ BCF, which is preferred for pipeing.
+
+ -l FILE File containing a list of sites where pileup or BCF
+ is outputted [null]
+
+ -q INT Minimum mapping quality for an alignment to be used
+ [0]
+
+ -Q INT Minimum base quality for a base to be considered [13]
+
+ -r STR Only generate pileup in region STR [all sites]
+
reheader samtools reheader <in.header.sam> <in.bam>
[500000000]
- merge samtools merge [-h inh.sam] [-nr] <out.bam> <in1.bam>
- <in2.bam> [...]
+ merge samtools merge [-nur] [-h inh.sam] [-R reg] <out.bam>
+ <in1.bam> <in2.bam> [...]
Merge multiple sorted alignments. The header reference lists
of all the input BAM files, and the @SQ headers of inh.sam,
SAM format, though any alignment records it may con-
tain are ignored.)
+ -R STR Merge files in the specified region indicated by STR
+
-r Attach an RG tag to each alignment. The tag value is
inferred from file names.
-n The input alignments are sorted by read names rather
than by chromosomal coordinates
+ -u Uncompressed BAM output
+
index samtools index <aln.bam>
-S Treat paired-end reads and single-end reads.
- calmd samtools calmd [-eubS] <aln.bam> <ref.fasta>
+ calmd samtools calmd [-eubS
r] [-C capQcoef] <aln.bam> <ref.fasta>
Generate the MD tag. If the MD tag is already present, this
command will give a warning if the MD tag generated is dif-
-S The input is SAM with header lines
+ -C INT Coefficient to cap mapping quality of poorly mapped
+ reads. See the pileup command for details. [0]
+
+ -r Perform probabilistic realignment to compute BAQ,
+ which will be used to cap base quality.
+
SAM FORMAT
SAM is TAB-delimited. Apart from the header lines, which are started
|0x0400 | d | the read is either a PCR or an optical duplicate |
+-------+-----+--------------------------------------------------+
+EXAMPLES
+ o Import SAM to BAM when @SQ lines are present in the header:
+
+ samtools view -bS aln.sam > aln.bam
+
+ If @SQ lines are absent:
+
+ samtools faidx ref.fa
+ samtools view -bt ref.fa.fai aln.sam > aln.bam
+
+ where ref.fa.fai is generated automatically by the faidx command.
+
+
+ o Attach the RG tag while merging sorted alignments:
+
+ perl -e 'print "@RG\tID:ga\tSM:hs\tLB:ga\tPL:Illu-
+ mina\n@RG\tID:454\tSM:hs\tLB:454\tPL:454\n"' > rg.txt
+ samtools merge -rh rg.txt merged.bam ga.bam 454.bam
+
+ The value in a RG tag is determined by the file name the read is com-
+ ing from. In this example, in the merged.bam, reads from ga.bam will
+ be attached RG:Z:ga, while reads from 454.bam will be attached
+ RG:Z:454.
+
+
+ o Call SNPs and short indels for one diploid individual:
+
+ samtools pileup -vcf ref.fa aln.bam > var.raw.plp
+ samtools.pl varFilter -D 100 var.raw.plp > var.flt.plp
+ awk '($3=="*"&&$6>=50)||($3!="*"&&$6>=20)' var.flt.plp >
+ var.final.plp
+
+ The -D option of varFilter controls the maximum read depth, which
+ should be adjusted to about twice the average read depth. One may
+ consider to add -C50 to pileup if mapping quality is overestimated
+ for reads containing excessive mismatches. Applying this option usu-
+ ally helps BWA-short but may not other mappers. It also potentially
+ increases reference biases.
+
+
+ o Call SNPs (not short indels) for multiple diploid individuals:
+
+ samtools mpileup -augf ref.fa *.bam | bcftools view -bcv - >
+ snp.raw.bcf
+ bcftools view snp.raw.bcf | vcfutils.pl filter4vcf -D 2000 | bgzip
+ > snp.flt.vcf.gz
+
+ Individuals are identified from the SM tags in the @RG header lines.
+ Individuals can be pooled in one alignment file; one individual can
+ also be separated into multiple files. Similarly, one may consider to
+ apply -C50 to mpileup. SNP calling in this way also works for single
+ sample and has the advantage of enabling more powerful filtering. The
+ drawback is the lack of short indel calling, which may be implemented
+ in future.
+
+
+ o Derive the allele frequency spectrum (AFS) on a list of sites from
+ multiple individuals:
+
+ samtools mpileup -gf ref.fa *.bam > all.bcf
+ bcftools view -bl sites.list all.bcf > sites.bcf
+ bcftools view -cGP cond2 sites.bcf > /dev/null 2> sites.1.afs
+ bcftools view -cGP sites.1.afs sites.bcf > /dev/null 2> sites.2.afs
+ bcftools view -cGP sites.2.afs sites.bcf > /dev/null 2> sites.3.afs
+ ......
+
+ where sites.list contains the list of sites with each line consisting
+ of the reference sequence name and position. The following bcftools
+ commands estimate AFS by EM.
+
+
+ o Dump BAQ applied alignment for other SNP callers:
+
+ samtools calmd -br aln.bam > aln.baq.bam
+
+ It adds and corrects the NM and MD tags at the same time. The calmd
+ command also comes with the -C option, the same as the on in pileup
+ and mpileup. Apply if it helps.
+
+
LIMITATIONS
- o Unaligned words used in bam_import.c, bam_endian.h, bam.c and
+ o Unaligned words used in bam_import.c, bam_endian.h, bam.c and
bam_aux.c.
- o In merging, the input files are required to have the same number of
- reference sequences. The requirement can be relaxed. In addition,
- merging does not reconstruct the header dictionaries automatically.
- Endusers have to provide the correct header. Picard is better at
+ o In merging, the input files are required to have the same number of
+ reference sequences. The requirement can be relaxed. In addition,
+ merging does not reconstruct the header dictionaries automatically.
+ Endusers have to provide the correct header. Picard is better at
merging.
- o Samtools paired-end rmdup does not work for unpaired reads (e.g.
- orphan reads or ends mapped to different chromosomes). If this is a
- concern, please use Picard's MarkDuplicate which correctly handles
+ o Samtools paired-end rmdup does not work for unpaired reads (e.g.
+ orphan reads or ends mapped to different chromosomes). If this is a
+ concern, please use Picard's MarkDuplicate which correctly handles
these cases, although a little slower.
AUTHOR
- Heng Li from the Sanger Institute wrote the C version of samtools. Bob
+ Heng Li from the Sanger Institute wrote the C version of samtools. Bob
Handsaker from the Broad Institute implemented the BGZF library and Jue
- Ruan from Beijing Genomics Institute wrote the RAZF library. Various
+ Ruan from Beijing Genomics Institute wrote the RAZF library. Various
people in the 1000 Genomes Project contributed to the SAM format speci-
fication.
-samtools-0.1.8 11 July 2010 samtools(1)
+samtools-0.1.9 27 October 2010 samtools(1)
projects / samtools.git / commitdiff