From 8d2494d1fb7cd0fa7c63be5ffba8dd1a11457522 Mon Sep 17 00:00:00 2001 From: Charles Plessy Date: Mon, 1 Nov 2010 18:45:04 +0900 Subject: [PATCH] Imported Upstream version 0.1.9 --- ChangeLog | 1304 ++++++++++++++++++++++++++++++++++++++++++ Makefile | 17 +- NEWS | 49 ++ bam.c | 28 +- bam.h | 21 +- bam2bcf.c | 186 ++++++ bam2bcf.h | 37 ++ bam_index.c | 28 +- bam_maqcns.c | 82 +-- bam_maqcns.h | 6 +- bam_md.c | 129 ++++- bam_pileup.c | 138 +++-- bam_plcmd.c | 409 ++++++++++--- bam_sort.c | 112 ++-- bam_tview.c | 4 +- bamtk.c | 2 +- bcftools/Makefile | 51 ++ bcftools/README | 36 ++ bcftools/bcf-fix.pl | 101 ++++ bcftools/bcf.c | 306 ++++++++++ bcftools/bcf.h | 161 ++++++ bcftools/bcf.tex | 63 ++ bcftools/bcf2qcall.c | 91 +++ bcftools/bcftools.1 | 115 ++++ bcftools/bcfutils.c | 109 ++++ bcftools/call1.c | 372 ++++++++++++ bcftools/fet.c | 110 ++++ bcftools/index.c | 335 +++++++++++ bcftools/kfunc.c | 162 ++++++ bcftools/ld.c | 100 ++++ bcftools/main.c | 64 +++ bcftools/prob1.c | 394 +++++++++++++ bcftools/prob1.h | 39 ++ bcftools/vcf.c | 212 +++++++ bcftools/vcfutils.pl | 477 +++++++++++++++ bgzf.c | 6 +- bgzip.c | 191 ++++--- errmod.c | 130 +++++ errmod.h | 17 + examples/Makefile | 17 +- examples/toy.fa | 2 + examples/toy.sam | 13 +- kaln.c | 235 +++++++- kaln.h | 16 +- ksort.h | 10 + kstring.c | 89 ++- kstring.h | 32 +- misc/HmmGlocal.java | 178 ++++++ misc/Makefile | 2 +- misc/samtools.pl | 1 - razip.c | 2 +- sam_view.c | 23 +- sample.c | 92 +++ sample.h | 17 + samtools.1 | 384 +++++++++---- samtools.txt | 349 +++++++---- 56 files changed, 7053 insertions(+), 603 deletions(-) create mode 100644 bam2bcf.c create mode 100644 bam2bcf.h create mode 100644 bcftools/Makefile create mode 100644 bcftools/README create mode 100755 bcftools/bcf-fix.pl create mode 100644 bcftools/bcf.c create mode 100644 bcftools/bcf.h create mode 100644 bcftools/bcf.tex create mode 100644 bcftools/bcf2qcall.c create mode 100644 bcftools/bcftools.1 create mode 100644 bcftools/bcfutils.c create mode 100644 bcftools/call1.c create mode 100644 bcftools/fet.c create mode 100644 bcftools/index.c create mode 100644 bcftools/kfunc.c create mode 100644 bcftools/ld.c create mode 100644 bcftools/main.c create mode 100644 bcftools/prob1.c create mode 100644 bcftools/prob1.h create mode 100644 bcftools/vcf.c create mode 100755 bcftools/vcfutils.pl create mode 100644 errmod.c create mode 100644 errmod.h create mode 100644 misc/HmmGlocal.java create mode 100644 sample.c create mode 100644 sample.h diff --git a/ChangeLog b/ChangeLog index 6b0ff6c..12908c9 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,1307 @@ +------------------------------------------------------------------------ +r782 | lh3lh3 | 2010-10-27 19:58:54 -0400 (Wed, 27 Oct 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam_plcmd.c + +fixed a silly bug in pileup + +------------------------------------------------------------------------ +r781 | lh3lh3 | 2010-10-27 14:39:48 -0400 (Wed, 27 Oct 2010) | 5 lines +Changed paths: + M /trunk/samtools/ChangeLog + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bam_sort.c + M /trunk/samtools/bamtk.c + M /trunk/samtools/samtools.1 + + * samtools-0.1.8-22 (r781) + * made BAQ the default behavior of mpileup + * updated manual + * in merge, force to exit given inconsistent header when "-R" is not in use. + +------------------------------------------------------------------------ +r780 | lh3lh3 | 2010-10-27 11:01:11 -0400 (Wed, 27 Oct 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam.h + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bamtk.c + + * samtools-0.1.8-21 (r780) + * minor speedup to pileup + +------------------------------------------------------------------------ +r779 | lh3lh3 | 2010-10-27 09:58:56 -0400 (Wed, 27 Oct 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam_pileup.c + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/examples/toy.sam + +improve pileup a little bit + +------------------------------------------------------------------------ +r778 | lh3lh3 | 2010-10-27 00:14:43 -0400 (Wed, 27 Oct 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam.h + M /trunk/samtools/bam_pileup.c + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bam_tview.c + M /trunk/samtools/bamtk.c + + * samtools-0.1.8-20 (r778) + * speed up pileup, although I do not know how much is the improvement + +------------------------------------------------------------------------ +r777 | lh3lh3 | 2010-10-26 17:26:04 -0400 (Tue, 26 Oct 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam_maqcns.c + M /trunk/samtools/bam_maqcns.h + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bamtk.c + M /trunk/samtools/examples/Makefile + + * samtools-0.1.8-19 (r777) + * integrate mpileup features to pileup: min_baseQ, capQ, prob_realn, paired-only and biased prior + +------------------------------------------------------------------------ +r776 | lh3lh3 | 2010-10-26 15:27:46 -0400 (Tue, 26 Oct 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam_md.c + +remove local realignment (probabilistic realignment is still there) + +------------------------------------------------------------------------ +r774 | jmarshall | 2010-10-21 06:52:38 -0400 (Thu, 21 Oct 2010) | 3 lines +Changed paths: + M /trunk/samtools/sam_view.c + +Add the relevant filename or region to error messages, and cause a failure +exit status where appropriate. Based on a patch provided by Marcel Martin. + +------------------------------------------------------------------------ +r773 | lh3lh3 | 2010-10-19 19:44:31 -0400 (Tue, 19 Oct 2010) | 3 lines +Changed paths: + M /trunk/samtools/examples/toy.sam + M /trunk/samtools/kaln.c + + * Minor code changes. No real effect. + * change quality to 30 in toy.sam + +------------------------------------------------------------------------ +r772 | lh3lh3 | 2010-10-18 23:40:13 -0400 (Mon, 18 Oct 2010) | 2 lines +Changed paths: + M /trunk/samtools/examples/toy.fa + M /trunk/samtools/examples/toy.sam + +added another toy example + +------------------------------------------------------------------------ +r771 | lh3lh3 | 2010-10-13 23:32:12 -0400 (Wed, 13 Oct 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/call1.c + M /trunk/samtools/bcftools/ld.c + M /trunk/samtools/bcftools/vcfutils.pl + +improve the LD statistics + +------------------------------------------------------------------------ +r770 | lh3lh3 | 2010-10-12 23:49:26 -0400 (Tue, 12 Oct 2010) | 3 lines +Changed paths: + M /trunk/samtools/bcftools/call1.c + M /trunk/samtools/bcftools/vcfutils.pl + + * a minor fix to the -L option + * add ldstats to vcfutils.pl + +------------------------------------------------------------------------ +r769 | lh3lh3 | 2010-10-12 15:51:57 -0400 (Tue, 12 Oct 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/bcf.c + +a minor change + +------------------------------------------------------------------------ +r768 | lh3lh3 | 2010-10-12 15:49:06 -0400 (Tue, 12 Oct 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/Makefile + A /trunk/samtools/bcftools/ld.c + +forget to add the key file + +------------------------------------------------------------------------ +r767 | lh3lh3 | 2010-10-12 15:48:46 -0400 (Tue, 12 Oct 2010) | 4 lines +Changed paths: + M /trunk/samtools/bcftools/Makefile + M /trunk/samtools/bcftools/bcf.c + M /trunk/samtools/bcftools/bcf.h + M /trunk/samtools/bcftools/call1.c + M /trunk/samtools/bcftools/prob1.c + M /trunk/samtools/bcftools/vcfutils.pl + + * vcfutils.pl: fixed a typo in help message + * added APIs: bcf_append_info() and bcf_cpy() + * calculate adjacent LD + +------------------------------------------------------------------------ +r766 | lh3lh3 | 2010-10-11 11:06:40 -0400 (Mon, 11 Oct 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/vcfutils.pl + +added filter for samtools/bcftools genetated VCFs + +------------------------------------------------------------------------ +r765 | lh3lh3 | 2010-10-05 14:05:18 -0400 (Tue, 05 Oct 2010) | 3 lines +Changed paths: + M /trunk/samtools/bcftools/vcfutils.pl + M /trunk/samtools/kaln.c + + * removed a comment line in kaln.c + * vcfutils.pl fillac works when GT is not the first field + +------------------------------------------------------------------------ +r764 | petulda | 2010-10-05 08:59:36 -0400 (Tue, 05 Oct 2010) | 1 line +Changed paths: + A /trunk/samtools/bcftools/bcf-fix.pl + +Convert VCF output of "bcftools view -bgcv" to a valid VCF file +------------------------------------------------------------------------ +r763 | lh3lh3 | 2010-10-02 22:51:03 -0400 (Sat, 02 Oct 2010) | 4 lines +Changed paths: + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bamtk.c + A /trunk/samtools/bcftools/bcftools.1 + M /trunk/samtools/bcftools/call1.c + M /trunk/samtools/samtools.1 + + * samtools-0.1.8-18 (r763) + * added bcftools manual page + * minor fix to mpileup and view command lines + +------------------------------------------------------------------------ +r762 | lh3lh3 | 2010-10-02 21:46:25 -0400 (Sat, 02 Oct 2010) | 3 lines +Changed paths: + M /trunk/samtools/bcftools/bcf.c + M /trunk/samtools/bcftools/call1.c + M /trunk/samtools/bcftools/vcfutils.pl + + * vcfutils.pl qstats: calculate marginal ts/tv + * allow to call genotypes at variant sites + +------------------------------------------------------------------------ +r761 | lh3lh3 | 2010-10-01 00:29:55 -0400 (Fri, 01 Oct 2010) | 3 lines +Changed paths: + M /trunk/samtools/kaln.c + M /trunk/samtools/misc/HmmGlocal.java + +I am changing the gap open probability back to 0.001. It seems that +being conservative here is a good thing... + +------------------------------------------------------------------------ +r760 | lh3lh3 | 2010-10-01 00:11:27 -0400 (Fri, 01 Oct 2010) | 5 lines +Changed paths: + M /trunk/samtools/bamtk.c + M /trunk/samtools/kaln.c + A /trunk/samtools/misc/HmmGlocal.java + + * samtools-0.1.8-17 (r760) + * the default gap open penalty is too small (a typo) + * added comments on hmm_realn + * Java implementation + +------------------------------------------------------------------------ +r759 | lh3lh3 | 2010-09-30 10:12:54 -0400 (Thu, 30 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bamtk.c + +mark samtools-0.1.8-16 (r759) + +------------------------------------------------------------------------ +r758 | lh3lh3 | 2010-09-30 10:12:02 -0400 (Thu, 30 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/kaln.c + +round to the nearest integer + +------------------------------------------------------------------------ +r757 | lh3lh3 | 2010-09-28 17:16:43 -0400 (Tue, 28 Sep 2010) | 4 lines +Changed paths: + M /trunk/samtools/kaln.c + +I was trying to accelerate ka_prob_glocal() as this will be the +bottleneck. After an hour, the only gain is to change division to +multiplication. OK. I will stop. + +------------------------------------------------------------------------ +r756 | lh3lh3 | 2010-09-28 16:57:49 -0400 (Tue, 28 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/kaln.c + +this is interesting. multiplication is much faster than division, at least on my Mac + +------------------------------------------------------------------------ +r755 | lh3lh3 | 2010-09-28 16:19:13 -0400 (Tue, 28 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/kaln.c + +minor changes + +------------------------------------------------------------------------ +r754 | lh3lh3 | 2010-09-28 15:44:16 -0400 (Tue, 28 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam_md.c + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/kaln.c + +prob_realn() seems working! + +------------------------------------------------------------------------ +r753 | lh3lh3 | 2010-09-28 12:48:23 -0400 (Tue, 28 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/kaln.c + +minor + +------------------------------------------------------------------------ +r752 | lh3lh3 | 2010-09-28 12:47:41 -0400 (Tue, 28 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/kaln.c + M /trunk/samtools/kaln.h + +Convert phredQ to probabilities + +------------------------------------------------------------------------ +r751 | lh3lh3 | 2010-09-28 12:32:08 -0400 (Tue, 28 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/kaln.c + M /trunk/samtools/kaln.h + +Implement the glocal HMM; discard the extention HMM + +------------------------------------------------------------------------ +r750 | lh3lh3 | 2010-09-28 00:06:11 -0400 (Tue, 28 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/kaln.c + +improve numerical stability + +------------------------------------------------------------------------ +r749 | lh3lh3 | 2010-09-27 23:27:54 -0400 (Mon, 27 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/kaln.c + +more comments + +------------------------------------------------------------------------ +r748 | lh3lh3 | 2010-09-27 23:17:16 -0400 (Mon, 27 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/kaln.c + +fixed a bug in banded DP + +------------------------------------------------------------------------ +r747 | lh3lh3 | 2010-09-27 23:05:12 -0400 (Mon, 27 Sep 2010) | 3 lines +Changed paths: + M /trunk/samtools/kaln.c + + * fixed that weird issue. + * the banded version is NOT working + +------------------------------------------------------------------------ +r746 | lh3lh3 | 2010-09-27 22:57:05 -0400 (Mon, 27 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/kaln.c + +More comments. This version seems working, but something is a little weird... + +------------------------------------------------------------------------ +r745 | lh3lh3 | 2010-09-27 17:21:40 -0400 (Mon, 27 Sep 2010) | 6 lines +Changed paths: + M /trunk/samtools/kaln.c + +A little code cleanup. Now the forward and backback algorithms give +nearly identical P(x), which means both are close to the correct +forms. However, I have only tested on toy examples. Minor errors in +the implementation may not be obvious. + + +------------------------------------------------------------------------ +r744 | lh3lh3 | 2010-09-27 16:55:15 -0400 (Mon, 27 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bam_sort.c + M /trunk/samtools/kaln.c + M /trunk/samtools/kaln.h + +... + +------------------------------------------------------------------------ +r743 | jmarshall | 2010-09-27 08:19:06 -0400 (Mon, 27 Sep 2010) | 6 lines +Changed paths: + M /trunk/samtools/bam_sort.c + +Abort if merge -h's INH.SAM cannot be opened, just as we abort +if any of the IN#.BAM input files cannot be opened. + +Also propagate any error indication returned by bam_merge_core() +to samtools merge's exit status. + +------------------------------------------------------------------------ +r741 | jmarshall | 2010-09-24 11:08:24 -0400 (Fri, 24 Sep 2010) | 5 lines +Changed paths: + M /trunk/samtools/bam_index.c + +Use bam_validate1() to detect garbage records in the event of a corrupt +BAI index file that causes a bam_seek() to an invalid position. At most +one record (namely, the bam_iter_read terminator) is tested per bam_fetch() +call, so the cost is insignificant in the normal case. + +------------------------------------------------------------------------ +r740 | jmarshall | 2010-09-24 11:00:19 -0400 (Fri, 24 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam.c + M /trunk/samtools/bam.h + +Add bam_validate1(). + +------------------------------------------------------------------------ +r739 | lh3lh3 | 2010-09-22 12:07:50 -0400 (Wed, 22 Sep 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam_md.c + M /trunk/samtools/bamtk.c + + * samtools-0.1.8-15 (r379) + * allow to change capQ parameter in calmd + +------------------------------------------------------------------------ +r738 | jmarshall | 2010-09-22 11:15:33 -0400 (Wed, 22 Sep 2010) | 13 lines +Changed paths: + M /trunk/samtools/bam_index.c + M /trunk/samtools/sam_view.c + +When bam_read1() returns an error (return value <= -2), propagate that error +to bam_iter_read()'s own return value. Similarly, also propagate it up to +bam_fetch()'s return value. Previously bam_fetch() always returned 0, and +callers ignored its return value anyway. With this change, 0 continues to +indicate success, while <= -2 (which can be written as < 0, as -1 is never +returned) indicates corrupted input. + +bam_iter_read() ought also to propagate errors returned by bam_seek(). + +main_samview() can now print an error message and fail when bam_fetch() +detects that a .bai index file is corrupted or otherwise does not correspond +to the .bam file it is being used with. + +------------------------------------------------------------------------ +r737 | jmarshall | 2010-09-22 10:47:42 -0400 (Wed, 22 Sep 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam_index.c + +0 is a successful return value from bam_read1(). (In practice, it never +returns 0 anyway; but all the other callers treat 0 as successful.) + +------------------------------------------------------------------------ +r736 | lh3lh3 | 2010-09-20 17:43:08 -0400 (Mon, 20 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam.h + M /trunk/samtools/bam_index.c + M /trunk/samtools/bam_sort.c + + * merge files region-by-region. work on small examples but more tests are needed. + +------------------------------------------------------------------------ +r735 | lh3lh3 | 2010-09-20 16:56:24 -0400 (Mon, 20 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/vcfutils.pl + +improve qstats by checking the alleles as well + +------------------------------------------------------------------------ +r734 | lh3lh3 | 2010-09-17 18:12:13 -0400 (Fri, 17 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/vcfutils.pl + +convert UCSC SNP SQL dump to VCF + +------------------------------------------------------------------------ +r733 | lh3lh3 | 2010-09-17 13:02:11 -0400 (Fri, 17 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/vcfutils.pl + +hapmap2vcf convertor + +------------------------------------------------------------------------ +r732 | lh3lh3 | 2010-09-17 10:11:37 -0400 (Fri, 17 Sep 2010) | 3 lines +Changed paths: + M /trunk/samtools/bcftools/Makefile + M /trunk/samtools/bcftools/bcf.c + M /trunk/samtools/bcftools/bcf.h + M /trunk/samtools/bcftools/vcf.c + + * added comments + * VCF->BCF is not possible without knowing the sequence dictionary before hand... + +------------------------------------------------------------------------ +r731 | lh3lh3 | 2010-09-17 09:15:53 -0400 (Fri, 17 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam2bcf.c + M /trunk/samtools/bcftools/bcf.c + M /trunk/samtools/bcftools/bcf.h + M /trunk/samtools/bcftools/bcfutils.c + M /trunk/samtools/bcftools/call1.c + M /trunk/samtools/bcftools/vcf.c + + * put n_smpl to "bcf1_t" to simplify API a little + +------------------------------------------------------------------------ +r730 | lh3lh3 | 2010-09-16 21:36:01 -0400 (Thu, 16 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/bcf.h + M /trunk/samtools/bcftools/call1.c + M /trunk/samtools/bcftools/index.c + +fixed a bug in indexing + +------------------------------------------------------------------------ +r729 | lh3lh3 | 2010-09-16 16:54:48 -0400 (Thu, 16 Sep 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam.c + M /trunk/samtools/bam_md.c + M /trunk/samtools/bam_pileup.c + + * fixed a bug in capQ + * valgrind identifies a use of uninitialised value, but I have not fixed it. + +------------------------------------------------------------------------ +r728 | lh3lh3 | 2010-09-16 15:03:59 -0400 (Thu, 16 Sep 2010) | 3 lines +Changed paths: + M /trunk/samtools/bgzip.c + M /trunk/samtools/razip.c + + * fixed a bug in razip: -c will delete the input file + * copy tabix/bgzip to here + +------------------------------------------------------------------------ +r727 | lh3lh3 | 2010-09-16 13:45:49 -0400 (Thu, 16 Sep 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam_md.c + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bamtk.c + + * samtools-0.1.8-14 (r727) + * allow to change the capQ parameter at the command line + +------------------------------------------------------------------------ +r726 | lh3lh3 | 2010-09-16 13:38:43 -0400 (Thu, 16 Sep 2010) | 4 lines +Changed paths: + M /trunk/samtools/bam_md.c + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bcftools/vcfutils.pl + M /trunk/samtools/misc/samtools.pl + + * added varFilter to vcfutils.pl + * reimplement realn(). now it performs a local alignment + * added cap_mapQ() to cap mapping quality when there are many substitutions + +------------------------------------------------------------------------ +r724 | lh3lh3 | 2010-09-15 00:18:31 -0400 (Wed, 15 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/Makefile + A /trunk/samtools/bcftools/bcf2qcall.c + M /trunk/samtools/bcftools/call1.c + + * convert BCF to QCALL input + +------------------------------------------------------------------------ +r723 | lh3lh3 | 2010-09-14 22:41:50 -0400 (Tue, 14 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam_md.c + +dynamic band width in realignment + +------------------------------------------------------------------------ +r722 | lh3lh3 | 2010-09-14 22:05:32 -0400 (Tue, 14 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam_md.c + M /trunk/samtools/bam_plcmd.c + +fixed a bug in realignment + +------------------------------------------------------------------------ +r721 | lh3lh3 | 2010-09-14 20:54:09 -0400 (Tue, 14 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/prob1.c + +fixed a minor issue + +------------------------------------------------------------------------ +r720 | lh3lh3 | 2010-09-14 19:25:10 -0400 (Tue, 14 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/Makefile + M /trunk/samtools/bam_maqcns.c + M /trunk/samtools/bam_md.c + +fixed a bug in realignment + +------------------------------------------------------------------------ +r719 | lh3lh3 | 2010-09-14 19:18:24 -0400 (Tue, 14 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam_plcmd.c + +minor changes. It is BUGGY now! + +------------------------------------------------------------------------ +r718 | lh3lh3 | 2010-09-14 16:32:33 -0400 (Tue, 14 Sep 2010) | 4 lines +Changed paths: + M /trunk/samtools/bam_md.c + M /trunk/samtools/bam_pileup.c + M /trunk/samtools/kaln.c + M /trunk/samtools/kaln.h + + * aggressive gapped aligner is implemented in calmd. + * distinguish gap_open and gap_end_open in banded alignment + * make tview accepts alignment with heading and tailing D + +------------------------------------------------------------------------ +r717 | jmarshall | 2010-09-14 09:04:28 -0400 (Tue, 14 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools + +Add svn:ignore properties for generated files that don't appear in "make all". + +------------------------------------------------------------------------ +r716 | jmarshall | 2010-09-13 08:37:53 -0400 (Mon, 13 Sep 2010) | 3 lines +Changed paths: + M /trunk/samtools + M /trunk/samtools/bcftools + M /trunk/samtools/misc + +Add svn:ignore properties listing the generated files. +(Except for *.o, which we'll assume is in global-ignores.) + +------------------------------------------------------------------------ +r715 | lh3lh3 | 2010-09-08 12:53:55 -0400 (Wed, 08 Sep 2010) | 5 lines +Changed paths: + M /trunk/samtools/bamtk.c + M /trunk/samtools/bcftools/call1.c + M /trunk/samtools/bcftools/prob1.c + M /trunk/samtools/sample.c + M /trunk/samtools/sample.h + + * samtools-0.1.8-13 (r715) + * fixed a bug in identifying SM across files + * bcftools: estimate heterozygosity + * bcftools: allow to skip sites without reference bases + +------------------------------------------------------------------------ +r713 | lh3lh3 | 2010-09-03 17:19:12 -0400 (Fri, 03 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/Makefile + M /trunk/samtools/bcftools/call1.c + M /trunk/samtools/bcftools/prob1.c + M /trunk/samtools/bcftools/prob1.h + +quite a lot changes to the contrast caller, but I still feel something is missing... + +------------------------------------------------------------------------ +r711 | lh3lh3 | 2010-09-03 00:30:48 -0400 (Fri, 03 Sep 2010) | 4 lines +Changed paths: + M /trunk/samtools/bcftools/Makefile + M /trunk/samtools/bcftools/call1.c + M /trunk/samtools/bcftools/prob1.c + M /trunk/samtools/bcftools/vcfutils.pl + + * changed 3.434 to 4.343 (typo!) + * fixed a bug in the contrast caller + * calculate heterozygosity + +------------------------------------------------------------------------ +r710 | lh3lh3 | 2010-09-01 23:24:47 -0400 (Wed, 01 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/bcf.h + M /trunk/samtools/bcftools/bcfutils.c + M /trunk/samtools/bcftools/call1.c + +SNP calling from the GL field + +------------------------------------------------------------------------ +r709 | lh3lh3 | 2010-09-01 18:52:30 -0400 (Wed, 01 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/vcf.c + +fixed another problem + +------------------------------------------------------------------------ +r708 | lh3lh3 | 2010-09-01 18:31:17 -0400 (Wed, 01 Sep 2010) | 3 lines +Changed paths: + M /trunk/samtools/bcftools/bcf.c + M /trunk/samtools/bcftools/vcf.c + + * fixed bugs in parsing VCF + * parser now works with GT/GQ/DP/PL/GL + +------------------------------------------------------------------------ +r707 | lh3lh3 | 2010-09-01 15:28:29 -0400 (Wed, 01 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/Makefile + M /trunk/samtools/bcftools/prob1.c + +Do not compile _BCF_QUAD by default + +------------------------------------------------------------------------ +r706 | lh3lh3 | 2010-09-01 15:21:41 -0400 (Wed, 01 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/bcf.c + M /trunk/samtools/bcftools/bcf.h + M /trunk/samtools/bcftools/bcfutils.c + M /trunk/samtools/bcftools/call1.c + +Write the correct ALT and PL in the SNP calling mode. + +------------------------------------------------------------------------ +r705 | lh3lh3 | 2010-09-01 12:50:33 -0400 (Wed, 01 Sep 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/vcfutils.pl + +more commands for my own uses + +------------------------------------------------------------------------ +r704 | lh3lh3 | 2010-09-01 09:26:10 -0400 (Wed, 01 Sep 2010) | 2 lines +Changed paths: + A /trunk/samtools/bcftools/vcfutils.pl + +Utilities for processing VCF + +------------------------------------------------------------------------ +r703 | lh3lh3 | 2010-08-31 16:44:57 -0400 (Tue, 31 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/Makefile + M /trunk/samtools/bcftools/call1.c + M /trunk/samtools/bcftools/prob1.c + M /trunk/samtools/bcftools/prob1.h + +preliminary contrast variant caller + +------------------------------------------------------------------------ +r702 | lh3lh3 | 2010-08-31 12:28:39 -0400 (Tue, 31 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/call1.c + M /trunk/samtools/bcftools/prob1.c + M /trunk/samtools/bcftools/prob1.h + +z' and z'' can be calculated + +------------------------------------------------------------------------ +r701 | lh3lh3 | 2010-08-31 10:20:57 -0400 (Tue, 31 Aug 2010) | 3 lines +Changed paths: + M /trunk/samtools/bcftools/Makefile + A /trunk/samtools/bcftools/call1.c (from /trunk/samtools/bcftools/vcfout.c:699) + M /trunk/samtools/bcftools/prob1.c + D /trunk/samtools/bcftools/vcfout.c + + * rename vcfout.c as call1.c + * prepare to add two-sample comparison + +------------------------------------------------------------------------ +r699 | lh3lh3 | 2010-08-24 15:28:16 -0400 (Tue, 24 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/vcfout.c + +fixed a bug in calculating the t statistics + +------------------------------------------------------------------------ +r698 | lh3lh3 | 2010-08-24 14:05:50 -0400 (Tue, 24 Aug 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam2bcf.c + M /trunk/samtools/bam2bcf.h + M /trunk/samtools/bamtk.c + M /trunk/samtools/bcftools/kfunc.c + M /trunk/samtools/bcftools/vcfout.c + + * samtools-0.1.8-13 (r698) + * perform one-tailed t-test for baseQ, mapQ and endDist + +------------------------------------------------------------------------ +r697 | lh3lh3 | 2010-08-24 12:30:13 -0400 (Tue, 24 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/kfunc.c + +added regularized incomplete beta function + +------------------------------------------------------------------------ +r695 | lh3lh3 | 2010-08-23 17:36:17 -0400 (Mon, 23 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam_maqcns.c + M /trunk/samtools/bam_plcmd.c + +change the default correlation coefficient + +------------------------------------------------------------------------ +r694 | lh3lh3 | 2010-08-23 14:46:52 -0400 (Mon, 23 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/bcf.c + M /trunk/samtools/bcftools/vcfout.c + +print QUAL as floating numbers + +------------------------------------------------------------------------ +r693 | lh3lh3 | 2010-08-23 14:06:07 -0400 (Mon, 23 Aug 2010) | 3 lines +Changed paths: + M /trunk/samtools/Makefile + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bamtk.c + M /trunk/samtools/examples/Makefile + A /trunk/samtools/sample.c + A /trunk/samtools/sample.h + + * samtools-0.1.8-12 (r692) + * group data by samples in "mpileup -g" + +------------------------------------------------------------------------ +r692 | lh3lh3 | 2010-08-23 10:58:53 -0400 (Mon, 23 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/Makefile + D /trunk/samtools/bam_mcns.c + D /trunk/samtools/bam_mcns.h + M /trunk/samtools/bam_plcmd.c + +remove VCF output in mpileup + +------------------------------------------------------------------------ +r691 | lh3lh3 | 2010-08-23 10:48:20 -0400 (Mon, 23 Aug 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam2bcf.c + M /trunk/samtools/bam2bcf.h + + * use the revised MAQ error model for mpileup + * prepare to remove the independent model from mpileup + +------------------------------------------------------------------------ +r690 | lh3lh3 | 2010-08-20 15:46:40 -0400 (Fri, 20 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/Makefile + M /trunk/samtools/bam_maqcns.c + M /trunk/samtools/bam_maqcns.h + M /trunk/samtools/bam_plcmd.c + A /trunk/samtools/errmod.c + A /trunk/samtools/errmod.h + M /trunk/samtools/ksort.h + +added revised MAQ error model + +------------------------------------------------------------------------ +r689 | lh3lh3 | 2010-08-18 09:55:20 -0400 (Wed, 18 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/prob1.c + M /trunk/samtools/bcftools/prob1.h + M /trunk/samtools/bcftools/vcfout.c + +allow to read the prior from the error output. EM iteration is working. + +------------------------------------------------------------------------ +r688 | lh3lh3 | 2010-08-17 12:12:20 -0400 (Tue, 17 Aug 2010) | 3 lines +Changed paths: + M /trunk/samtools/bcftools/main.c + M /trunk/samtools/bcftools/vcf.c + + * write a little more VCF header + * concatenate BCFs + +------------------------------------------------------------------------ +r687 | lh3lh3 | 2010-08-16 20:53:16 -0400 (Mon, 16 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/bcf.c + M /trunk/samtools/bcftools/bcf.h + M /trunk/samtools/bcftools/bcf.tex + +use float for QUAL + +------------------------------------------------------------------------ +r686 | lh3lh3 | 2010-08-14 00:11:13 -0400 (Sat, 14 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/bcf.c + M /trunk/samtools/bcftools/prob1.c + +faster for large sample size (in principle) + +------------------------------------------------------------------------ +r685 | lh3lh3 | 2010-08-13 23:28:31 -0400 (Fri, 13 Aug 2010) | 4 lines +Changed paths: + M /trunk/samtools/bcftools/prob1.c + + * a numerically stable method to calculate z_{jk} + * currently slower than the old method but will be important for large sample size + * in principle, we can speed up for large n, but have not tried + +------------------------------------------------------------------------ +r684 | lh3lh3 | 2010-08-11 21:58:31 -0400 (Wed, 11 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/vcfout.c + +fixed an issue in parsing integer + +------------------------------------------------------------------------ +r683 | lh3lh3 | 2010-08-09 13:05:07 -0400 (Mon, 09 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/bcf.c + +do not print refname if file is converted from VCF + +------------------------------------------------------------------------ +r682 | lh3lh3 | 2010-08-09 12:59:47 -0400 (Mon, 09 Aug 2010) | 3 lines +Changed paths: + M /trunk/samtools/bcftools/vcf.c + + * parse PL + * fixed a bug in parsing VCF + +------------------------------------------------------------------------ +r681 | lh3lh3 | 2010-08-09 12:49:23 -0400 (Mon, 09 Aug 2010) | 4 lines +Changed paths: + M /trunk/samtools/bcftools/bcf.c + M /trunk/samtools/bcftools/bcf.h + M /trunk/samtools/bcftools/bcfutils.c + M /trunk/samtools/bcftools/main.c + M /trunk/samtools/bcftools/vcf.c + M /trunk/samtools/bcftools/vcfout.c + M /trunk/samtools/bgzf.c + M /trunk/samtools/kstring.c + + * fixed a bug in kstrtok@kstring.c + * preliminary VCF parser (not parse everything for now) + * improved view interface + +------------------------------------------------------------------------ +r680 | lh3lh3 | 2010-08-09 10:43:13 -0400 (Mon, 09 Aug 2010) | 4 lines +Changed paths: + M /trunk/samtools/bcftools/bcf.c + M /trunk/samtools/bcftools/bcf.h + M /trunk/samtools/bcftools/vcfout.c + M /trunk/samtools/kstring.c + M /trunk/samtools/kstring.h + + * improved kstring (added kstrtok) + * removed the limit on the format string length in bcftools + * use kstrtok to parse format which fixed a bug in the old code + +------------------------------------------------------------------------ +r679 | lh3lh3 | 2010-08-09 01:12:05 -0400 (Mon, 09 Aug 2010) | 2 lines +Changed paths: + A /trunk/samtools/bcftools/README + M /trunk/samtools/bcftools/vcfout.c + +help messages + +------------------------------------------------------------------------ +r678 | lh3lh3 | 2010-08-09 00:01:52 -0400 (Mon, 09 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/vcfout.c + +perform single-tail test for ED4 + +------------------------------------------------------------------------ +r677 | lh3lh3 | 2010-08-08 23:48:35 -0400 (Sun, 08 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/Makefile + M /trunk/samtools/bcftools/kfunc.c + M /trunk/samtools/bcftools/vcfout.c + + * test depth, end distance and HWE + +------------------------------------------------------------------------ +r676 | lh3lh3 | 2010-08-08 02:04:15 -0400 (Sun, 08 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/kfunc.c + +reimplement incomplete gamma functions. no copy-paste + +------------------------------------------------------------------------ +r675 | lh3lh3 | 2010-08-06 22:42:54 -0400 (Fri, 06 Aug 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam2bcf.c + M /trunk/samtools/bam2bcf.h + M /trunk/samtools/bcftools/fet.c + M /trunk/samtools/bcftools/prob1.c + M /trunk/samtools/bcftools/prob1.h + M /trunk/samtools/bcftools/vcfout.c + + * bcftools: add HWE (no testing for now) + * record end dist in a 2x2 table, not avg, std any more + +------------------------------------------------------------------------ +r674 | lh3lh3 | 2010-08-06 17:30:16 -0400 (Fri, 06 Aug 2010) | 3 lines +Changed paths: + A /trunk/samtools/bcftools/kfunc.c + + * Special functions: log(gamma()), erfc(), P(a,x) (incomplete gamma) + * Not using Numerical Recipe due to licensing issues + +------------------------------------------------------------------------ +r673 | lh3lh3 | 2010-08-05 23:46:53 -0400 (Thu, 05 Aug 2010) | 2 lines +Changed paths: + A /trunk/samtools/bcftools/fet.c + +Fisher's exact test + +------------------------------------------------------------------------ +r672 | lh3lh3 | 2010-08-05 21:48:33 -0400 (Thu, 05 Aug 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam2bcf.c + M /trunk/samtools/bam2bcf.h + M /trunk/samtools/bamtk.c + M /trunk/samtools/examples/Makefile + + * samtools-0.1.8-11 (r672) + * collect more stats for allele balance test in bcftools (not yet) + +------------------------------------------------------------------------ +r671 | lh3lh3 | 2010-08-05 16:17:58 -0400 (Thu, 05 Aug 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bcftools/bcf.c + M /trunk/samtools/bcftools/main.c + + * the code base is stablized again. + * I will delay the vcf parser, which is quite complicated but with little value for now + +------------------------------------------------------------------------ +r670 | lh3lh3 | 2010-08-05 16:03:23 -0400 (Thu, 05 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/examples/Makefile + +minor + +------------------------------------------------------------------------ +r669 | lh3lh3 | 2010-08-05 16:03:08 -0400 (Thu, 05 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcftools/vcf.c + +unfinished vcf parser + +------------------------------------------------------------------------ +r668 | lh3lh3 | 2010-08-05 15:46:40 -0400 (Thu, 05 Aug 2010) | 3 lines +Changed paths: + M /trunk/samtools/bcftools/Makefile + M /trunk/samtools/bcftools/bcf.c + M /trunk/samtools/bcftools/bcf.h + M /trunk/samtools/bcftools/bcfutils.c + M /trunk/samtools/bcftools/index.c + M /trunk/samtools/bcftools/main.c + A /trunk/samtools/bcftools/vcf.c + M /trunk/samtools/bcftools/vcfout.c + + * added prelimiary VCF parser (not finished) + * change struct a bit + +------------------------------------------------------------------------ +r667 | lh3lh3 | 2010-08-03 22:35:27 -0400 (Tue, 03 Aug 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam2bcf.c + M /trunk/samtools/bam2bcf.h + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bcftools/bcf.c + + * allow to set min base q + * fixed a bug in mpileup -u + +------------------------------------------------------------------------ +r666 | lh3lh3 | 2010-08-03 22:08:44 -0400 (Tue, 03 Aug 2010) | 2 lines +Changed paths: + A /trunk/samtools/bcftools/bcf.tex + +spec + +------------------------------------------------------------------------ +r665 | lh3lh3 | 2010-08-03 21:18:57 -0400 (Tue, 03 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/examples/Makefile + +added more examples + +------------------------------------------------------------------------ +r664 | lh3lh3 | 2010-08-03 21:13:00 -0400 (Tue, 03 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/Makefile + M /trunk/samtools/bam2bcf.c + M /trunk/samtools/bam2bcf.h + M /trunk/samtools/bcftools/Makefile + +fixed compilation error + +------------------------------------------------------------------------ +r662 | lh3lh3 | 2010-08-03 21:04:00 -0400 (Tue, 03 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/Makefile + D /trunk/samtools/bcf.c + D /trunk/samtools/bcf.h + A /trunk/samtools/bcftools + A /trunk/samtools/bcftools/Makefile + A /trunk/samtools/bcftools/bcf.c + A /trunk/samtools/bcftools/bcf.h + A /trunk/samtools/bcftools/bcfutils.c + A /trunk/samtools/bcftools/index.c + A /trunk/samtools/bcftools/main.c + A /trunk/samtools/bcftools/prob1.c + A /trunk/samtools/bcftools/prob1.h + A /trunk/samtools/bcftools/vcfout.c + +move bcftools to samtools + +------------------------------------------------------------------------ +r660 | lh3lh3 | 2010-08-03 15:58:32 -0400 (Tue, 03 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam2bcf.c + +fixed another minor bug + +------------------------------------------------------------------------ +r658 | lh3lh3 | 2010-08-03 15:06:45 -0400 (Tue, 03 Aug 2010) | 3 lines +Changed paths: + M /trunk/samtools/bamtk.c + M /trunk/samtools/bcf.c + + * samtools-0.1.8-10 (r658) + * fixed a bug in bam2bcf when the reference is N + +------------------------------------------------------------------------ +r657 | lh3lh3 | 2010-08-03 14:50:23 -0400 (Tue, 03 Aug 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam2bcf.c + M /trunk/samtools/bam2bcf.h + + * fixed a bug + * treat ambiguous ref base as the fifth base + +------------------------------------------------------------------------ +r654 | lh3lh3 | 2010-08-02 17:38:27 -0400 (Mon, 02 Aug 2010) | 2 lines +Changed paths: + M /trunk/bcftools/bcf.c + M /trunk/samtools/bcf.c + +missing a column in VCF output... + +------------------------------------------------------------------------ +r653 | lh3lh3 | 2010-08-02 17:31:33 -0400 (Mon, 02 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcf.c + +fixed a memory leak + +------------------------------------------------------------------------ +r651 | lh3lh3 | 2010-08-02 17:27:31 -0400 (Mon, 02 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bcf.c + +fixed a bug in bcf reader + +------------------------------------------------------------------------ +r650 | lh3lh3 | 2010-08-02 17:00:41 -0400 (Mon, 02 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam2bcf.c + +fixed a bug + +------------------------------------------------------------------------ +r649 | lh3lh3 | 2010-08-02 16:49:35 -0400 (Mon, 02 Aug 2010) | 3 lines +Changed paths: + M /trunk/samtools/Makefile + M /trunk/samtools/bam2bcf.c + M /trunk/samtools/bam2bcf.h + M /trunk/samtools/bamtk.c + + * samtools-0.1.8-9 (r649) + * lossless representation of PL in BCF output + +------------------------------------------------------------------------ +r648 | lh3lh3 | 2010-08-02 16:07:25 -0400 (Mon, 02 Aug 2010) | 2 lines +Changed paths: + M /trunk/samtools/Makefile + A /trunk/samtools/bam2bcf.c + A /trunk/samtools/bam2bcf.h + M /trunk/samtools/bam_plcmd.c + A /trunk/samtools/bcf.c + A /trunk/samtools/bcf.h + +Generate binary VCF + +------------------------------------------------------------------------ +r644 | lh3lh3 | 2010-07-28 11:59:19 -0400 (Wed, 28 Jul 2010) | 5 lines +Changed paths: + M /trunk/samtools/bam_mcns.c + M /trunk/samtools/bamtk.c + + * samtools-0.1.8-8 (r644) + * mpileup becomes a little stable again + * the method is slightly different, but is more theoretically correct + * snp calling is O(n^2) instead of O(n^3) + +------------------------------------------------------------------------ +r643 | lh3lh3 | 2010-07-28 11:54:15 -0400 (Wed, 28 Jul 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam_mcns.c + + * fixed a STUPID bug, which cost me a lot of time. + * I am going to clean up mcns a little bit + +------------------------------------------------------------------------ +r642 | lh3lh3 | 2010-07-27 23:23:07 -0400 (Tue, 27 Jul 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam_mcns.c + M /trunk/samtools/bam_mcns.h + M /trunk/samtools/bam_plcmd.c + +supposedly this is THE correct implementation, but more testing is needed + +------------------------------------------------------------------------ +r641 | lh3lh3 | 2010-07-27 22:43:39 -0400 (Tue, 27 Jul 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam_mcns.c + +NOT ready yet. Going to make further changes... + +------------------------------------------------------------------------ +r639 | lh3lh3 | 2010-07-25 22:18:38 -0400 (Sun, 25 Jul 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam_mcns.c + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bamtk.c + + * samtools-0.1.8-7 (r639) + * fixed the reference allele assignment + +------------------------------------------------------------------------ +r638 | lh3lh3 | 2010-07-25 12:01:26 -0400 (Sun, 25 Jul 2010) | 5 lines +Changed paths: + M /trunk/samtools/bam_mcns.c + M /trunk/samtools/bam_mcns.h + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bamtk.c + + * samtools-0.1.8-6 (r638) + * skip isnan/isinf in case of float underflow + * added the flat prior + * fixed an issue where there are no reads supporting the reference + +------------------------------------------------------------------------ +r637 | lh3lh3 | 2010-07-24 14:16:27 -0400 (Sat, 24 Jul 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam_plcmd.c + +minor changes + +------------------------------------------------------------------------ +r636 | lh3lh3 | 2010-07-24 14:07:27 -0400 (Sat, 24 Jul 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam_mcns.c + M /trunk/samtools/bam_mcns.h + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bamtk.c + +minor tweaks + +------------------------------------------------------------------------ +r635 | lh3lh3 | 2010-07-24 01:49:49 -0400 (Sat, 24 Jul 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam_mcns.c + M /trunk/samtools/bam_mcns.h + M /trunk/samtools/bam_plcmd.c + +posterior expectation FINALLY working. I am so tired... + +------------------------------------------------------------------------ +r633 | lh3lh3 | 2010-07-23 13:50:48 -0400 (Fri, 23 Jul 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam_plcmd.c + +another minor fix to mpileup + +------------------------------------------------------------------------ +r632 | lh3lh3 | 2010-07-23 13:43:31 -0400 (Fri, 23 Jul 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam_plcmd.c + +added the format column + +------------------------------------------------------------------------ +r631 | lh3lh3 | 2010-07-23 13:25:44 -0400 (Fri, 23 Jul 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam_mcns.c + M /trunk/samtools/bam_mcns.h + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bamtk.c + +added an alternative prior + +------------------------------------------------------------------------ +r628 | lh3lh3 | 2010-07-23 11:48:51 -0400 (Fri, 23 Jul 2010) | 2 lines +Changed paths: + M /trunk/samtools/bam_mcns.c + M /trunk/samtools/bam_mcns.h + M /trunk/samtools/bam_plcmd.c + +calculate posterior allele frequency + +------------------------------------------------------------------------ +r627 | lh3lh3 | 2010-07-22 21:39:13 -0400 (Thu, 22 Jul 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam_mcns.c + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bamtk.c + + * samtools-0.1.8-3 (r627) + * multi-sample snp calling appears to work. More tests needed. + +------------------------------------------------------------------------ +r626 | lh3lh3 | 2010-07-22 16:37:56 -0400 (Thu, 22 Jul 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam_mcns.c + M /trunk/samtools/bam_mcns.h + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bam_tview.c + + * preliminary multisample SNP caller. + * something looks not so right, but it largely works + +------------------------------------------------------------------------ +r617 | lh3lh3 | 2010-07-14 16:26:27 -0400 (Wed, 14 Jul 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam_mcns.c + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bamtk.c + + * samtools-0.1.8-2 (r617) + * allele frequency calculation apparently works... + +------------------------------------------------------------------------ +r616 | lh3lh3 | 2010-07-14 13:33:51 -0400 (Wed, 14 Jul 2010) | 3 lines +Changed paths: + M /trunk/samtools/Makefile + A /trunk/samtools/bam_mcns.c + A /trunk/samtools/bam_mcns.h + M /trunk/samtools/bam_plcmd.c + + * added mutli-sample framework. It is not working, yet. + * improved the mpileup interface + +------------------------------------------------------------------------ +r615 | lh3lh3 | 2010-07-13 14:50:12 -0400 (Tue, 13 Jul 2010) | 3 lines +Changed paths: + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bamtk.c + M /trunk/samtools/misc/Makefile + + * samtools-0.1.8-1 (r615) + * allow to get mpileup at required sites + +------------------------------------------------------------------------ +r613 | lh3lh3 | 2010-07-11 22:40:56 -0400 (Sun, 11 Jul 2010) | 2 lines +Changed paths: + M /trunk/samtools/ChangeLog + M /trunk/samtools/NEWS + M /trunk/samtools/bam_plcmd.c + M /trunk/samtools/bamtk.c + M /trunk/samtools/samtools.1 + +Release samtools-0.1.8 + ------------------------------------------------------------------------ r612 | lh3lh3 | 2010-07-11 21:08:56 -0400 (Sun, 11 Jul 2010) | 2 lines Changed paths: diff --git a/Makefile b/Makefile index 35d578f..77cda86 100644 --- a/Makefile +++ b/Makefile @@ -7,10 +7,10 @@ LOBJS= bgzf.o kstring.o bam_aux.o bam.o bam_import.o sam.o bam_index.o \ $(KNETFILE_O) bam_sort.o sam_header.o bam_reheader.o AOBJS= bam_tview.o bam_maqcns.o bam_plcmd.o sam_view.o \ bam_rmdup.o bam_rmdupse.o bam_mate.o bam_stat.o bam_color.o \ - bamtk.o kaln.o + bamtk.o kaln.o bam2bcf.o errmod.o sample.o PROG= samtools -INCLUDES= -SUBDIRS= . misc +INCLUDES= -I. +SUBDIRS= . bcftools misc LIBPATH= LIBCURSES= -lcurses # -lXCurses @@ -39,8 +39,8 @@ lib:libbam.a libbam.a:$(LOBJS) $(AR) -cru $@ $(LOBJS) -samtools:$(AOBJS) libbam.a - $(CC) $(CFLAGS) -o $@ $(AOBJS) libbam.a -lm $(LIBPATH) $(LIBCURSES) -lz +samtools:lib-recur $(AOBJS) + $(CC) $(CFLAGS) -o $@ $(AOBJS) libbam.a -lm $(LIBPATH) $(LIBCURSES) -lz -Lbcftools -lbcf razip:razip.o razf.o $(KNETFILE_O) $(CC) $(CFLAGS) -o $@ razf.o razip.o $(KNETFILE_O) -lz @@ -53,15 +53,18 @@ bam.o:bam.h razf.h bam_endian.h kstring.h sam_header.h sam.o:sam.h bam.h bam_import.o:bam.h kseq.h khash.h razf.h bam_pileup.o:bam.h razf.h ksort.h -bam_plcmd.o:bam.h faidx.h bam_maqcns.h glf.h +bam_plcmd.o:bam.h faidx.h bam_maqcns.h glf.h bcftools/bcf.h bam2bcf.h bam_index.o:bam.h khash.h ksort.h razf.h bam_endian.h bam_lpileup.o:bam.h ksort.h bam_tview.o:bam.h faidx.h bam_maqcns.h -bam_maqcns.o:bam.h ksort.h bam_maqcns.h +bam_maqcns.o:bam.h ksort.h bam_maqcns.h kaln.h bam_sort.o:bam.h ksort.h razf.h bam_md.o:bam.h faidx.h glf.o:glf.h sam_header.o:sam_header.h khash.h +bcf.o:bcftools/bcf.h +bam2bcf.o:bam2bcf.h errmod.h bcftools/bcf.h +errmod.o:errmod.h faidx.o:faidx.h razf.h khash.h faidx_main.o:faidx.h razf.h diff --git a/NEWS b/NEWS index 28d6aaa..82646ba 100644 --- a/NEWS +++ b/NEWS @@ -1,3 +1,52 @@ +Beta Release 0.1.9 (27 October, 2010) +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This release is featured as the first major improvement to the samtools' +SNP caller. It comes with a revised MAQ error model, the support of +multi-sample SNP calling and the computation of base alignment quality +(BAQ). + +The revised MAQ error model is based on the original model. It solves an +issue of miscalling SNPs in repetitive regions. Althought such SNPs can +usually be filtered at a later step, they mess up unfiltered calls. This +is a theoretical flaw in the original model. The revised MAQ model +deprecates the orginal MAQ model and the simplified SOAPsnp model. + +Multi-sample SNP calling is separated in two steps. The first is done by +samtools mpileup and the second by a new program, bcftools, which is +included in the samtools source code tree. Multi-sample SNP calling also +works for single sample and has the advantage of enabling more powerful +filtration. It is likely to deprecate pileup in future once a proper +indel calling method is implemented. + +BAQ is the Phred-scaled probability of a read base being wrongly +aligned. Capping base quality by BAQ has been shown to be very effective +in suppressing false SNPs caused by misalignments around indels or in +low-complexity regions with acceptable compromise on computation +time. This strategy is highly recommended and can be used with other SNP +callers as well. + +In addition to the three major improvements, other notable changes are: + + * Changes to the pileup format. A reference skip (the N CIGAR operator) + is shown as '<' or '>' depending on the strand. Tview is also changed + accordingly. + + * Accelerated pileup. The plain pileup is about 50% faster. + + * Regional merge. The merge command now accepts a new option to merge + files in a specified region. + + * Fixed a bug in bgzip and razip which causes source files to be + deleted even if option -c is applied. + + * In APIs, propogate errors to downstream callers and make samtools + return non-zero values once errors occur. + +(0.1.9: 27 October 2010, r783) + + + Beta Release 0.1.8 (11 July, 2010) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/bam.c b/bam.c index 94b0aa8..521c1dd 100644 --- a/bam.c +++ b/bam.c @@ -245,7 +245,11 @@ char *bam_format1_core(const bam_header_t *header, const bam1_t *b, int of) kputc('\t', &str); } if (c->tid < 0) kputsn("*\t", 2, &str); - else { kputs(header->target_name[c->tid], &str); kputc('\t', &str); } + else { + if (header) kputs(header->target_name[c->tid] , &str); + else kputw(c->tid, &str); + kputc('\t', &str); + } kputw(c->pos + 1, &str); kputc('\t', &str); kputw(c->qual, &str); kputc('\t', &str); if (c->n_cigar == 0) kputc('*', &str); else { @@ -257,7 +261,11 @@ char *bam_format1_core(const bam_header_t *header, const bam1_t *b, int of) kputc('\t', &str); if (c->mtid < 0) kputsn("*\t", 2, &str); else if (c->mtid == c->tid) kputsn("=\t", 2, &str); - else { kputs(header->target_name[c->mtid], &str); kputc('\t', &str); } + else { + if (header) kputs(header->target_name[c->mtid], &str); + else kputw(c->mtid, &str); + kputc('\t', &str); + } kputw(c->mpos + 1, &str); kputc('\t', &str); kputw(c->isize, &str); kputc('\t', &str); if (c->l_qseq) { for (i = 0; i < c->l_qseq; ++i) kputc(bam_nt16_rev_table[bam1_seqi(s, i)], &str); @@ -297,6 +305,22 @@ void bam_view1(const bam_header_t *header, const bam1_t *b) free(s); } +int bam_validate1(const bam_header_t *header, const bam1_t *b) +{ + char *s; + + if (b->core.tid < -1 || b->core.mtid < -1) return 0; + if (header && (b->core.tid >= header->n_targets || b->core.mtid >= header->n_targets)) return 0; + + if (b->data_len < b->core.l_qname) return 0; + s = memchr(bam1_qname(b), '\0', b->core.l_qname); + if (s != &bam1_qname(b)[b->core.l_qname-1]) return 0; + + // FIXME: Other fields could also be checked, especially the auxiliary data + + return 1; +} + // FIXME: we should also check the LB tag associated with each alignment const char *bam_get_library(bam_header_t *h, const bam1_t *b) { diff --git a/bam.h b/bam.h index 8e26ea6..4c8536f 100644 --- a/bam.h +++ b/bam.h @@ -1,6 +1,6 @@ /* The MIT License - Copyright (c) 2008 Genome Research Ltd (GRL). + Copyright (c) 2008-2010 Genome Research Ltd (GRL). Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the @@ -230,7 +230,7 @@ typedef struct __bam_iter_t *bam_iter_t; @param b pointer to an alignment @return pointer to quality string */ -#define bam1_qual(b) ((b)->data + (b)->core.n_cigar*4 + (b)->core.l_qname + ((b)->core.l_qseq + 1)/2) +#define bam1_qual(b) ((b)->data + (b)->core.n_cigar*4 + (b)->core.l_qname + (((b)->core.l_qseq + 1)>>1)) /*! @function @abstract Get a base on read @@ -455,6 +455,21 @@ extern "C" { char *bam_format1_core(const bam_header_t *header, const bam1_t *b, int of); + /*! + @abstract Check whether a BAM record is plausibly valid + @param header associated header structure, or NULL if unavailable + @param b alignment to validate + @return 0 if the alignment is invalid; non-zero otherwise + + @discussion Simple consistency check of some of the fields of the + alignment record. If the header is provided, several additional checks + are made. Not all fields are checked, so a non-zero result is not a + guarantee that the record is valid. However it is usually good enough + to detect when bam_seek() has been called with a virtual file offset + that is not the offset of an alignment record. + */ + int bam_validate1(const bam_header_t *header, const bam1_t *b); + const char *bam_get_library(bam_header_t *header, const bam1_t *b); @@ -480,7 +495,7 @@ extern "C" { bam1_t *b; int32_t qpos; int indel, level; - uint32_t is_del:1, is_head:1, is_tail:1; + uint32_t is_del:1, is_head:1, is_tail:1, is_refskip:1; } bam_pileup1_t; typedef int (*bam_plp_auto_f)(void *data, bam1_t *b); diff --git a/bam2bcf.c b/bam2bcf.c new file mode 100644 index 0000000..e55212c --- /dev/null +++ b/bam2bcf.c @@ -0,0 +1,186 @@ +#include +#include +#include "bam.h" +#include "kstring.h" +#include "bam2bcf.h" +#include "errmod.h" +#include "bcftools/bcf.h" + +extern void ks_introsort_uint32_t(size_t n, uint32_t a[]); + +#define CALL_ETA 0.03f +#define CALL_MAX 256 +#define CALL_DEFTHETA 0.83f + +#define CAP_DIST 25 + +struct __bcf_callaux_t { + int max_bases, capQ, min_baseQ; + uint16_t *bases; + errmod_t *e; +}; + +bcf_callaux_t *bcf_call_init(double theta, int min_baseQ) +{ + bcf_callaux_t *bca; + if (theta <= 0.) theta = CALL_DEFTHETA; + bca = calloc(1, sizeof(bcf_callaux_t)); + bca->capQ = 60; + bca->min_baseQ = min_baseQ; + bca->e = errmod_init(1. - theta); + return bca; +} + +void bcf_call_destroy(bcf_callaux_t *bca) +{ + if (bca == 0) return; + errmod_destroy(bca->e); + free(bca->bases); free(bca); +} + +int bcf_call_glfgen(int _n, const bam_pileup1_t *pl, int ref_base /*4-bit*/, bcf_callaux_t *bca, bcf_callret1_t *r) +{ + int i, n, ref4; + memset(r, 0, sizeof(bcf_callret1_t)); + ref4 = bam_nt16_nt4_table[ref_base]; + if (_n == 0) return -1; + + // enlarge the bases array if necessary + if (bca->max_bases < _n) { + bca->max_bases = _n; + kroundup32(bca->max_bases); + bca->bases = (uint16_t*)realloc(bca->bases, 2 * bca->max_bases); + } + // fill the bases array + memset(r, 0, sizeof(bcf_callret1_t)); + for (i = n = 0; i < _n; ++i) { + const bam_pileup1_t *p = pl + i; + int q, b, mapQ, baseQ, is_diff, min_dist; + // set base + if (p->is_del || (p->b->core.flag&BAM_FUNMAP)) continue; // skip unmapped reads and deleted bases + baseQ = q = (int)bam1_qual(p->b)[p->qpos]; // base quality + if (q < bca->min_baseQ) continue; + mapQ = p->b->core.qual < bca->capQ? p->b->core.qual : bca->capQ; + if (q > mapQ) q = mapQ; + if (q > 63) q = 63; + if (q < 4) q = 4; + b = bam1_seqi(bam1_seq(p->b), p->qpos); // base + b = bam_nt16_nt4_table[b? b : ref_base]; // b is the 2-bit base + bca->bases[n++] = q<<5 | (int)bam1_strand(p->b)<<4 | b; + // collect annotations + r->qsum[b] += q; + is_diff = (ref4 < 4 && b == ref4)? 0 : 1; + ++r->anno[0<<2|is_diff<<1|bam1_strand(p->b)]; + min_dist = p->b->core.l_qseq - 1 - p->qpos; + if (min_dist > p->qpos) min_dist = p->qpos; + if (min_dist > CAP_DIST) min_dist = CAP_DIST; + r->anno[1<<2|is_diff<<1|0] += baseQ; + r->anno[1<<2|is_diff<<1|1] += baseQ * baseQ; + r->anno[2<<2|is_diff<<1|0] += mapQ; + r->anno[2<<2|is_diff<<1|1] += mapQ * mapQ; + r->anno[3<<2|is_diff<<1|0] += min_dist; + r->anno[3<<2|is_diff<<1|1] += min_dist * min_dist; + } + r->depth = n; + // glfgen + errmod_cal(bca->e, n, 5, bca->bases, r->p); + return r->depth; +} + +int bcf_call_combine(int n, const bcf_callret1_t *calls, int ref_base /*4-bit*/, bcf_call_t *call) +{ + int ref4, i, j, qsum[4]; + int64_t tmp; + call->ori_ref = ref4 = bam_nt16_nt4_table[ref_base]; + if (ref4 > 4) ref4 = 4; + // calculate qsum + memset(qsum, 0, 4 * sizeof(int)); + for (i = 0; i < n; ++i) + for (j = 0; j < 4; ++j) + qsum[j] += calls[i].qsum[j]; + for (j = 0; j < 4; ++j) qsum[j] = qsum[j] << 2 | j; + // find the top 2 alleles + for (i = 1; i < 4; ++i) // insertion sort + for (j = i; j > 0 && qsum[j] < qsum[j-1]; --j) + tmp = qsum[j], qsum[j] = qsum[j-1], qsum[j-1] = tmp; + // set the reference allele and alternative allele(s) + for (i = 0; i < 5; ++i) call->a[i] = -1; + call->unseen = -1; + call->a[0] = ref4; + for (i = 3, j = 1; i >= 0; --i) { + if ((qsum[i]&3) != ref4) { + if (qsum[i]>>2 != 0) call->a[j++] = qsum[i]&3; + else break; + } + } + if (((ref4 < 4 && j < 4) || (ref4 == 4 && j < 5)) && i >= 0) + call->unseen = j, call->a[j++] = qsum[i]&3; + call->n_alleles = j; + // set the PL array + if (call->n < n) { + call->n = n; + call->PL = realloc(call->PL, 15 * n); + } + { + int x, g[15], z; + double sum_min = 0.; + x = call->n_alleles * (call->n_alleles + 1) / 2; + // get the possible genotypes + for (i = z = 0; i < call->n_alleles; ++i) + for (j = i; j < call->n_alleles; ++j) + g[z++] = call->a[i] * 5 + call->a[j]; + for (i = 0; i < n; ++i) { + uint8_t *PL = call->PL + x * i; + const bcf_callret1_t *r = calls + i; + float min = 1e37; + for (j = 0; j < x; ++j) + if (min > r->p[g[j]]) min = r->p[g[j]]; + sum_min += min; + for (j = 0; j < x; ++j) { + int y; + y = (int)(r->p[g[j]] - min + .499); + if (y > 255) y = 255; + PL[j] = y; + } + } + call->shift = (int)(sum_min + .499); + } + // combine annotations + memset(call->anno, 0, 16 * sizeof(int)); + for (i = call->depth = 0, tmp = 0; i < n; ++i) { + call->depth += calls[i].depth; + for (j = 0; j < 16; ++j) call->anno[j] += calls[i].anno[j]; + } + return 0; +} + +int bcf_call2bcf(int tid, int pos, bcf_call_t *bc, bcf1_t *b) +{ + kstring_t s; + int i; + b->n_smpl = bc->n; + b->tid = tid; b->pos = pos; b->qual = 0; + s.s = b->str; s.m = b->m_str; s.l = 0; + kputc('\0', &s); + kputc("ACGTN"[bc->ori_ref], &s); kputc('\0', &s); + for (i = 1; i < 5; ++i) { + if (bc->a[i] < 0) break; + if (i > 1) kputc(',', &s); + kputc(bc->unseen == i? 'X' : "ACGT"[bc->a[i]], &s); + } + kputc('\0', &s); + kputc('\0', &s); + // INFO + kputs("I16=", &s); + for (i = 0; i < 16; ++i) { + if (i) kputc(',', &s); + kputw(bc->anno[i], &s); + } + kputc('\0', &s); + // FMT + kputs("PL", &s); kputc('\0', &s); + b->m_str = s.m; b->str = s.s; b->l_str = s.l; + bcf_sync(b); + memcpy(b->gi[0].data, bc->PL, b->gi[0].len * bc->n); + return 0; +} diff --git a/bam2bcf.h b/bam2bcf.h new file mode 100644 index 0000000..0dddb92 --- /dev/null +++ b/bam2bcf.h @@ -0,0 +1,37 @@ +#ifndef BAM2BCF_H +#define BAM2BCF_H + +#include +#include "bcftools/bcf.h" + +struct __bcf_callaux_t; +typedef struct __bcf_callaux_t bcf_callaux_t; + +typedef struct { + int depth, qsum[4]; + int anno[16]; + float p[25]; +} bcf_callret1_t; + +typedef struct { + int a[5]; // alleles: ref, alt, alt2, alt3 + int n, n_alleles, shift, ori_ref, unseen; + int anno[16], depth; + uint8_t *PL; +} bcf_call_t; + +#ifdef __cplusplus +extern "C" { +#endif + + bcf_callaux_t *bcf_call_init(double theta, int min_baseQ); + void bcf_call_destroy(bcf_callaux_t *bca); + int bcf_call_glfgen(int _n, const bam_pileup1_t *pl, int ref_base /*4-bit*/, bcf_callaux_t *bca, bcf_callret1_t *r); + int bcf_call_combine(int n, const bcf_callret1_t *calls, int ref_base /*4-bit*/, bcf_call_t *call); + int bcf_call2bcf(int tid, int pos, bcf_call_t *bc, bcf1_t *b); + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/bam_index.c b/bam_index.c index 4152f20..1ad2e93 100644 --- a/bam_index.c +++ b/bam_index.c @@ -656,17 +656,17 @@ void bam_iter_destroy(bam_iter_t iter) int bam_iter_read(bamFile fp, bam_iter_t iter, bam1_t *b) { - if (iter->finished) return -1; - if (iter->from_first) { - int ret = bam_read1(fp, b); - if (ret < 0) iter->finished = 1; + int ret; + if (iter && iter->finished) return -1; + if (iter == 0 || iter->from_first) { + ret = bam_read1(fp, b); + if (ret < 0 && iter) iter->finished = 1; return ret; } if (iter->off == 0) return -1; for (;;) { - int ret; if (iter->curr_off == 0 || iter->curr_off >= iter->off[iter->i].v) { // then jump to the next chunk - if (iter->i == iter->n_off - 1) break; // no more chunks + if (iter->i == iter->n_off - 1) { ret = -1; break; } // no more chunks if (iter->i >= 0) assert(iter->curr_off == iter->off[iter->i].v); // otherwise bug if (iter->i < 0 || iter->off[iter->i].v != iter->off[iter->i+1].u) { // not adjacent chunks; then seek bam_seek(fp, iter->off[iter->i+1].u, SEEK_SET); @@ -674,23 +674,27 @@ int bam_iter_read(bamFile fp, bam_iter_t iter, bam1_t *b) } ++iter->i; } - if ((ret = bam_read1(fp, b)) > 0) { + if ((ret = bam_read1(fp, b)) >= 0) { iter->curr_off = bam_tell(fp); - if (b->core.tid != iter->tid || b->core.pos >= iter->end) break; // no need to proceed + if (b->core.tid != iter->tid || b->core.pos >= iter->end) { // no need to proceed + ret = bam_validate1(NULL, b)? -1 : -5; // determine whether end of region or error + break; + } else if (is_overlap(iter->beg, iter->end, b)) return ret; - } else break; // end of file + } else break; // end of file or error } iter->finished = 1; - return -1; + return ret; } int bam_fetch(bamFile fp, const bam_index_t *idx, int tid, int beg, int end, void *data, bam_fetch_f func) { + int ret; bam_iter_t iter; bam1_t *b; b = bam_init1(); iter = bam_iter_query(idx, tid, beg, end); - while (bam_iter_read(fp, iter, b) >= 0) func(b, data); + while ((ret = bam_iter_read(fp, iter, b)) >= 0) func(b, data); bam_destroy1(b); - return 0; + return (ret == -1)? 0 : ret; } diff --git a/bam_maqcns.c b/bam_maqcns.c index cad63d7..2f3fc08 100644 --- a/bam_maqcns.c +++ b/bam_maqcns.c @@ -3,6 +3,7 @@ #include "bam.h" #include "bam_maqcns.h" #include "ksort.h" +#include "errmod.h" #include "kaln.h" KSORT_INIT_GENERIC(uint32_t) @@ -12,12 +13,13 @@ KSORT_INIT_GENERIC(uint32_t) typedef struct __bmc_aux_t { int max; uint32_t *info; + uint16_t *info16; + errmod_t *em; } bmc_aux_t; typedef struct { float esum[4], fsum[4]; uint32_t c[4]; - uint32_t rms_mapQ; } glf_call_aux_t; char bam_nt16_nt4_table[] = { 4, 0, 1, 4, 2, 4, 4, 4, 3, 4, 4, 4, 4, 4, 4, 4 }; @@ -41,7 +43,7 @@ static void cal_het(bam_maqcns_t *aa) for (n1 = 0; n1 < 256; ++n1) { for (n2 = 0; n2 < 256; ++n2) { long double sum = 0.0; - double lC = aa->is_soap? 0 : lgamma(n1+n2+1) - lgamma(n1+1) - lgamma(n2+1); // \binom{n1+n2}{n1} + double lC = aa->errmod == BAM_ERRMOD_SOAP? 0 : lgamma(n1+n2+1) - lgamma(n1+1) - lgamma(n2+1); for (k = 1; k <= aa->n_hap - 1; ++k) { double pk = 1.0 / k / sum_harmo; double log1 = log((double)k/aa->n_hap); @@ -62,6 +64,7 @@ static void cal_coef(bam_maqcns_t *aa) long double sum_a[257], b[256], q_c[256], tmp[256], fk2[256]; double *lC; + if (aa->errmod == BAM_ERRMOD_MAQ2) return; // no need to do the following // aa->lhet will be allocated and initialized free(aa->fk); free(aa->coef); aa->coef = 0; @@ -71,7 +74,7 @@ static void cal_coef(bam_maqcns_t *aa) aa->fk[n] = pow(aa->theta, n) * (1.0 - aa->eta) + aa->eta; fk2[n] = aa->fk[n>>1]; // this is an approximation, assuming reads equally likely come from both strands } - if (aa->is_soap) return; + if (aa->errmod == BAM_ERRMOD_SOAP) return; aa->coef = (double*)calloc(256*256*64, sizeof(double)); lC = (double*)calloc(256 * 256, sizeof(double)); for (n = 1; n != 256; ++n) @@ -107,28 +110,31 @@ bam_maqcns_t *bam_maqcns_init() bm = (bam_maqcns_t*)calloc(1, sizeof(bam_maqcns_t)); bm->aux = (bmc_aux_t*)calloc(1, sizeof(bmc_aux_t)); bm->het_rate = 0.001; - bm->theta = 0.85; + bm->theta = 0.83f; bm->n_hap = 2; bm->eta = 0.03; bm->cap_mapQ = 60; + bm->min_baseQ = 13; return bm; } void bam_maqcns_prepare(bam_maqcns_t *bm) { + if (bm->errmod == BAM_ERRMOD_MAQ2) bm->aux->em = errmod_init(1. - bm->theta); cal_coef(bm); cal_het(bm); } void bam_maqcns_destroy(bam_maqcns_t *bm) { if (bm == 0) return; - free(bm->lhet); free(bm->fk); free(bm->coef); free(bm->aux->info); + free(bm->lhet); free(bm->fk); free(bm->coef); free(bm->aux->info); free(bm->aux->info16); + if (bm->aux->em) errmod_destroy(bm->aux->em); free(bm->aux); free(bm); } glf1_t *bam_maqcns_glfgen(int _n, const bam_pileup1_t *pl, uint8_t ref_base, bam_maqcns_t *bm) { - glf_call_aux_t *b; + glf_call_aux_t *b = 0; int i, j, k, w[8], c, n; glf1_t *g = (glf1_t*)calloc(1, sizeof(glf1_t)); float p[16], min_p = 1e30; @@ -142,28 +148,39 @@ glf1_t *bam_maqcns_glfgen(int _n, const bam_pileup1_t *pl, uint8_t ref_base, bam bm->aux->max = _n; kroundup32(bm->aux->max); bm->aux->info = (uint32_t*)realloc(bm->aux->info, 4 * bm->aux->max); + bm->aux->info16 = (uint16_t*)realloc(bm->aux->info16, 2 * bm->aux->max); } - for (i = n = 0; i < _n; ++i) { + for (i = n = 0, rms = 0; i < _n; ++i) { const bam_pileup1_t *p = pl + i; uint32_t q, x = 0, qq; + uint16_t y = 0; if (p->is_del || (p->b->core.flag&BAM_FUNMAP)) continue; q = (uint32_t)bam1_qual(p->b)[p->qpos]; + if (q < bm->min_baseQ) continue; x |= (uint32_t)bam1_strand(p->b) << 18 | q << 8 | p->b->core.qual; + y |= bam1_strand(p->b)<<4; if (p->b->core.qual < q) q = p->b->core.qual; + c = p->b->core.qual < bm->cap_mapQ? p->b->core.qual : bm->cap_mapQ; + rms += c * c; x |= q << 24; + y |= q << 5; qq = bam1_seqi(bam1_seq(p->b), p->qpos); q = bam_nt16_nt4_table[qq? qq : ref_base]; - if (!p->is_del && q < 4) x |= 1 << 21 | q << 16; + if (!p->is_del && q < 4) x |= 1 << 21 | q << 16, y |= q; + bm->aux->info16[n] = y; bm->aux->info[n++] = x; } + rms = (uint8_t)(sqrt((double)rms / n) + .499); + if (bm->errmod == BAM_ERRMOD_MAQ2) { + errmod_cal(bm->aux->em, n, 4, bm->aux->info16, p); + goto goto_glf; + } ks_introsort(uint32_t, n, bm->aux->info); // generate esum and fsum b = (glf_call_aux_t*)calloc(1, sizeof(glf_call_aux_t)); for (k = 0; k != 8; ++k) w[k] = 0; - rms = 0; for (j = n - 1; j >= 0; --j) { // calculate esum and fsum uint32_t info = bm->aux->info[j]; - int tmp; if (info>>24 < 4 && (info>>8&0x3f) != 0) info = 4<<24 | (info&0xffffff); k = info>>16&7; if (info>>24 > 0) { @@ -172,17 +189,14 @@ glf1_t *bam_maqcns_glfgen(int _n, const bam_pileup1_t *pl, uint8_t ref_base, bam if (w[k] < 0xff) ++w[k]; ++b->c[k&3]; } - tmp = (int)(info&0xff) < bm->cap_mapQ? (int)(info&0xff) : bm->cap_mapQ; - rms += tmp * tmp; } - b->rms_mapQ = (uint8_t)(sqrt((double)rms / n) + .499); // rescale ->c[] for (j = c = 0; j != 4; ++j) c += b->c[j]; if (c > 255) { for (j = 0; j != 4; ++j) b->c[j] = (int)(254.0 * b->c[j] / c + 0.5); for (j = c = 0; j != 4; ++j) c += b->c[j]; } - if (!bm->is_soap) { + if (bm->errmod == BAM_ERRMOD_MAQ) { // generate likelihood for (j = 0; j != 4; ++j) { // homozygous @@ -234,7 +248,7 @@ glf1_t *bam_maqcns_glfgen(int _n, const bam_pileup1_t *pl, uint8_t ref_base, bam if (max1 > max2 && (min_k != max_k || min1 + 1.0 > min2)) p[max_k<<2|max_k] = min1 > 1.0? min1 - 1.0 : 0.0; } - } else { // apply the SOAP model + } else if (bm->errmod == BAM_ERRMOD_SOAP) { // apply the SOAP model // generate likelihood for (j = 0; j != 4; ++j) { float tmp; @@ -251,8 +265,9 @@ glf1_t *bam_maqcns_glfgen(int _n, const bam_pileup1_t *pl, uint8_t ref_base, bam } } +goto_glf: // convert necessary information to glf1_t - g->ref_base = ref_base; g->max_mapQ = b->rms_mapQ; + g->ref_base = ref_base; g->max_mapQ = rms; g->depth = n > 16777215? 16777215 : n; for (j = 0; j != 4; ++j) for (k = j; k < 4; ++k) @@ -268,26 +283,25 @@ glf1_t *bam_maqcns_glfgen(int _n, const bam_pileup1_t *pl, uint8_t ref_base, bam uint32_t glf2cns(const glf1_t *g, int q_r) { - int i, j, k, tmp[16], min = 10000, min2 = 10000, min3 = 10000, min_g = -1, min_g2 = -1; + int i, j, k, p[10], ref4; uint32_t x = 0; + ref4 = bam_nt16_nt4_table[g->ref_base]; for (i = k = 0; i < 4; ++i) for (j = i; j < 4; ++j) { - tmp[j<<2|i] = -1; - tmp[i<<2|j] = g->lk[k++] + (i == j? 0 : q_r); + int prior = (i == ref4 && j == ref4? 0 : i == ref4 || j == ref4? q_r : q_r + 3); + p[k] = (g->lk[k] + prior)<<4 | i<<2 | j; + ++k; } - for (i = 0; i < 16; ++i) { - if (tmp[i] < 0) continue; - if (tmp[i] < min) { - min3 = min2; min2 = min; min = tmp[i]; min_g2 = min_g; min_g = i; - } else if (tmp[i] < min2) { - min3 = min2; min2 = tmp[i]; min_g2 = i; - } else if (tmp[i] < min3) min3 = tmp[i]; - } - x = min_g >= 0? (1U<<(min_g>>2&3) | 1U<<(min_g&3)) << 28 : 0xf << 28; - x |= min_g2 >= 0? (1U<<(min_g2>>2&3) | 1U<<(min_g2&3)) << 24 : 0xf << 24; - x |= (uint32_t)g->max_mapQ << 16; - x |= min2 < 10000? (min2 - min < 256? min2 - min : 255) << 8 : 0xff << 8; - x |= min2 < 10000 && min3 < 10000? (min3 - min2 < 256? min3 - min2 : 255) : 0xff; + for (i = 1; i < 10; ++i) // insertion sort + for (j = i; j > 0 && p[j] < p[j-1]; --j) + k = p[j], p[j] = p[j-1], p[j-1] = k; + x = (1u<<(p[0]&3) | 1u<<(p[0]>>2&3)) << 28; // the best genotype + x |= (uint32_t)g->max_mapQ << 16; // rms mapQ + x |= ((p[1]>>4) - (p[0]>>4) < 256? (p[1]>>4) - (p[0]>>4) : 255) << 8; // consensus Q + for (k = 0; k < 10; ++k) + if ((p[k]&0xf) == (ref4<<2|ref4)) break; + if (k == 10) k = 9; + x |= (p[k]>>4) - (p[0]>>4) < 256? (p[k]>>4) - (p[0]>>4) : 255; // snp Q return x; } @@ -297,7 +311,7 @@ uint32_t bam_maqcns_call(int n, const bam_pileup1_t *pl, bam_maqcns_t *bm) uint32_t x; if (n) { g = bam_maqcns_glfgen(n, pl, 0xf, bm); - x = glf2cns(g, (int)(bm->q_r + 0.5)); + x = g->depth == 0? (0xfU<<28 | 0xfU<<24) : glf2cns(g, (int)(bm->q_r + 0.5)); free(g); } else x = 0xfU<<28 | 0xfU<<24; return x; @@ -448,7 +462,7 @@ bam_maqindel_ret_t *bam_maqindel(int n, int pos, const bam_maqindel_opt_t *mi, c for (i = 0; i < n_types; ++i) { ka_param_t ap = ka_param_blast; ap.band_width = 2 * types[n_types - 1] + 2; - ap.gap_end = 0; + ap.gap_end_ext = 0; // write ref2 for (k = 0, j = left; j <= pos; ++j) ref2[k++] = bam_nt16_nt4_table[bam_nt16_table[(int)ref[j]]]; diff --git a/bam_maqcns.h b/bam_maqcns.h index 6cc5355..291ae53 100644 --- a/bam_maqcns.h +++ b/bam_maqcns.h @@ -3,11 +3,15 @@ #include "glf.h" +#define BAM_ERRMOD_MAQ2 0 +#define BAM_ERRMOD_MAQ 1 +#define BAM_ERRMOD_SOAP 2 + struct __bmc_aux_t; typedef struct { float het_rate, theta; - int n_hap, cap_mapQ, is_soap; + int n_hap, cap_mapQ, errmod, min_baseQ; float eta, q_r; double *fk, *coef; diff --git a/bam_md.c b/bam_md.c index 17b0a4a..f65b845 100644 --- a/bam_md.c +++ b/bam_md.c @@ -2,9 +2,11 @@ #include #include #include +#include #include "faidx.h" #include "sam.h" #include "kstring.h" +#include "kaln.h" void bam_fillmd1_core(bam1_t *b, char *ref, int is_equal, int max_nm) { @@ -108,24 +110,134 @@ void bam_fillmd1(bam1_t *b, char *ref, int is_equal) bam_fillmd1_core(b, ref, is_equal, 0); } +int bam_cap_mapQ(bam1_t *b, char *ref, int thres) +{ + uint8_t *seq = bam1_seq(b), *qual = bam1_qual(b); + uint32_t *cigar = bam1_cigar(b); + bam1_core_t *c = &b->core; + int i, x, y, mm, q, len, clip_l, clip_q; + double t; + if (thres < 0) thres = 40; // set the default + mm = q = len = clip_l = clip_q = 0; + for (i = y = 0, x = c->pos; i < c->n_cigar; ++i) { + int j, l = cigar[i]>>4, op = cigar[i]&0xf; + if (op == BAM_CMATCH) { + for (j = 0; j < l; ++j) { + int z = y + j; + int c1 = bam1_seqi(seq, z), c2 = bam_nt16_table[(int)ref[x+j]]; + if (ref[x+j] == 0) break; // out of boundary + if (c2 != 15 && c1 != 15 && qual[z] >= 13) { // not ambiguous + ++len; + if (c1 && c1 != c2 && qual[z] >= 13) { // mismatch + ++mm; + q += qual[z] > 33? 33 : qual[z]; + } + } + } + if (j < l) break; + x += l; y += l; len += l; + } else if (op == BAM_CDEL) { + for (j = 0; j < l; ++j) + if (ref[x+j] == 0) break; + if (j < l) break; + x += l; + } else if (op == BAM_CSOFT_CLIP) { + for (j = 0; j < l; ++j) clip_q += qual[y+j]; + clip_l += l; + y += l; + } else if (op == BAM_CHARD_CLIP) { + clip_q += 13 * l; + clip_l += l; + } else if (op == BAM_CINS) y += l; + else if (op == BAM_CREF_SKIP) x += l; + } + for (i = 0, t = 1; i < mm; ++i) + t *= (double)len / (i+1); + t = q - 4.343 * log(t) + clip_q / 5.; + if (t > thres) return -1; + if (t < 0) t = 0; + t = sqrt((thres - t) / thres) * thres; +// fprintf(stderr, "%s %lf %d\n", bam1_qname(b), t, q); + return (int)(t + .499); +} + +int bam_prob_realn(bam1_t *b, const char *ref) +{ + int k, i, bw, x, y, yb, ye, xb, xe; + uint32_t *cigar = bam1_cigar(b); + bam1_core_t *c = &b->core; + ka_probpar_t conf = ka_probpar_def; + // find the start and end of the alignment + if (c->flag & BAM_FUNMAP) return -1; + x = c->pos, y = 0, yb = ye = xb = xe = -1; + for (k = 0; k < c->n_cigar; ++k) { + int op, l; + op = cigar[k]&0xf; l = cigar[k]>>4; + if (op == BAM_CMATCH) { + if (yb < 0) yb = y; + if (xb < 0) xb = x; + ye = y + l; xe = x + l; + x += l; y += l; + } else if (op == BAM_CSOFT_CLIP || op == BAM_CINS) y += l; + else if (op == BAM_CDEL) x += l; + else if (op == BAM_CREF_SKIP) return -1; + } + // set bandwidth and the start and the end + bw = 7; + if (abs((xe - xb) - (ye - yb)) > bw) + bw = abs((xe - xb) - (ye - yb)) + 3; + conf.bw = bw; + xb -= yb + bw/2; if (xb < 0) xb = 0; + xe += c->l_qseq - ye + bw/2; + if (xe - xb - c->l_qseq > bw) + xb += (xe - xb - c->l_qseq - bw) / 2, xe -= (xe - xb - c->l_qseq - bw) / 2; + { // glocal + uint8_t *s, *r, *q, *seq = bam1_seq(b), *qual = bam1_qual(b); + int *state; + s = calloc(c->l_qseq, 1); + for (i = 0; i < c->l_qseq; ++i) s[i] = bam_nt16_nt4_table[bam1_seqi(seq, i)]; + r = calloc(xe - xb, 1); + for (i = xb; i < xe; ++i) + r[i-xb] = bam_nt16_nt4_table[bam_nt16_table[(int)ref[i]]]; + state = calloc(c->l_qseq, sizeof(int)); + q = calloc(c->l_qseq, 1); + ka_prob_glocal(r, xe-xb, s, c->l_qseq, qual, &conf, state, q); + for (k = 0, x = c->pos, y = 0; k < c->n_cigar; ++k) { + int op = cigar[k]&0xf, l = cigar[k]>>4; + if (op == BAM_CMATCH) { + for (i = y; i < y + l; ++i) { + if ((state[i]&3) != 0 || state[i]>>2 != x - xb + (i - y)) qual[i] = 0; + else qual[i] = qual[i] < q[i]? qual[i] : q[i]; + } + x += l; y += l; + } else if (op == BAM_CSOFT_CLIP || op == BAM_CINS) y += l; + else if (op == BAM_CDEL) x += l; + } + free(s); free(r); free(q); free(state); + } + return 0; +} + int bam_fillmd(int argc, char *argv[]) { - int c, is_equal = 0, tid = -2, ret, len, is_bam_out, is_sam_in, is_uncompressed, max_nm = 0; + int c, is_equal = 0, tid = -2, ret, len, is_bam_out, is_sam_in, is_uncompressed, max_nm = 0, is_realn, capQ = 0; samfile_t *fp, *fpout = 0; faidx_t *fai; char *ref = 0, mode_w[8], mode_r[8]; bam1_t *b; - is_bam_out = is_sam_in = is_uncompressed = 0; + is_bam_out = is_sam_in = is_uncompressed = is_realn = 0; mode_w[0] = mode_r[0] = 0; strcpy(mode_r, "r"); strcpy(mode_w, "w"); - while ((c = getopt(argc, argv, "eubSn:")) >= 0) { + while ((c = getopt(argc, argv, "reubSC:n:")) >= 0) { switch (c) { + case 'r': is_realn = 1; break; case 'e': is_equal = 1; break; case 'b': is_bam_out = 1; break; case 'u': is_uncompressed = is_bam_out = 1; break; case 'S': is_sam_in = 1; break; case 'n': max_nm = atoi(optarg); break; + case 'C': capQ = atoi(optarg); break; default: fprintf(stderr, "[bam_fillmd] unrecognized option '-%c'\n", c); return 1; } } @@ -135,11 +247,12 @@ int bam_fillmd(int argc, char *argv[]) if (is_uncompressed) strcat(mode_w, "u"); if (optind + 1 >= argc) { fprintf(stderr, "\n"); - fprintf(stderr, "Usage: samtools fillmd [-eubS] \n\n"); + fprintf(stderr, "Usage: samtools fillmd [-eubrS] \n\n"); fprintf(stderr, "Options: -e change identical bases to '='\n"); fprintf(stderr, " -u uncompressed BAM output (for piping)\n"); fprintf(stderr, " -b compressed BAM output\n"); - fprintf(stderr, " -S the input is SAM with header\n\n"); + fprintf(stderr, " -S the input is SAM with header\n"); + fprintf(stderr, " -r read-independent local realignment\n\n"); return 1; } fp = samopen(argv[optind], mode_r, 0); @@ -162,6 +275,12 @@ int bam_fillmd(int argc, char *argv[]) fprintf(stderr, "[bam_fillmd] fail to find sequence '%s' in the reference.\n", fp->header->target_name[tid]); } +// if (is_realn) bam_realn(b, ref); + if (is_realn) bam_prob_realn(b, ref); + if (capQ > 10) { + int q = bam_cap_mapQ(b, ref, capQ); + if (b->core.qual > q) b->core.qual = q; + } if (ref) bam_fillmd1_core(b, ref, is_equal, max_nm); } samwrite(fpout, b); diff --git a/bam_pileup.c b/bam_pileup.c index 3c41a16..a49f795 100644 --- a/bam_pileup.c +++ b/bam_pileup.c @@ -4,9 +4,16 @@ #include #include "sam.h" +typedef struct { + int k, x, y, end; +} cstate_t; + +static cstate_t g_cstate_null = { -1, 0, 0, 0 }; + typedef struct __linkbuf_t { bam1_t b; uint32_t beg, end; + cstate_t s; struct __linkbuf_t *next; } lbnode_t; @@ -53,68 +60,86 @@ static inline void mp_free(mempool_t *mp, lbnode_t *p) /* --- BEGIN: Auxiliary functions */ -static inline int resolve_cigar(bam_pileup1_t *p, uint32_t pos) +/* s->k: the index of the CIGAR operator that has just been processed. + s->x: the reference coordinate of the start of s->k + s->y: the query coordiante of the start of s->k + */ +static inline int resolve_cigar2(bam_pileup1_t *p, uint32_t pos, cstate_t *s) { - unsigned k; +#define _cop(c) ((c)&BAM_CIGAR_MASK) +#define _cln(c) ((c)>>BAM_CIGAR_SHIFT) + bam1_t *b = p->b; bam1_core_t *c = &b->core; - uint32_t x = c->pos, y = 0; - int ret = 1, is_restart = 1; - - if (c->flag&BAM_FUNMAP) return 0; // unmapped read - assert(x <= pos); // otherwise a bug - p->qpos = -1; p->indel = 0; p->is_del = p->is_head = p->is_tail = 0; - for (k = 0; k < c->n_cigar; ++k) { - int op = bam1_cigar(b)[k] & BAM_CIGAR_MASK; // operation - int l = bam1_cigar(b)[k] >> BAM_CIGAR_SHIFT; // length - if (op == BAM_CMATCH) { // NOTE: this assumes the first and the last operation MUST BE a match or a clip - if (x + l > pos) { // overlap with pos - p->indel = p->is_del = 0; - p->qpos = y + (pos - x); - if (x == pos && is_restart) p->is_head = 1; - if (x + l - 1 == pos) { // come to the end of a match - int has_next_match = 0; - unsigned i; - for (i = k + 1; i < c->n_cigar; ++i) { - uint32_t cigar = bam1_cigar(b)[i]; - int opi = cigar&BAM_CIGAR_MASK; - if (opi == BAM_CMATCH) { - has_next_match = 1; - break; - } else if (opi == BAM_CSOFT_CLIP || opi == BAM_CREF_SKIP || opi == BAM_CHARD_CLIP) break; - } - if (!has_next_match) p->is_tail = 1; - if (k < c->n_cigar - 1 && has_next_match) { // there are additional operation(s) - uint32_t cigar = bam1_cigar(b)[k+1]; // next CIGAR - int op_next = cigar&BAM_CIGAR_MASK; // next CIGAR operation - if (op_next == BAM_CDEL) p->indel = -(int32_t)(cigar>>BAM_CIGAR_SHIFT); // del - else if (op_next == BAM_CINS) p->indel = cigar>>BAM_CIGAR_SHIFT; // ins - else if (op_next == BAM_CPAD && k + 2 < c->n_cigar) { // no working for adjacent padding - cigar = bam1_cigar(b)[k+2]; op_next = cigar&BAM_CIGAR_MASK; - if (op_next == BAM_CDEL) p->indel = -(int32_t)(cigar>>BAM_CIGAR_SHIFT); // del - else if (op_next == BAM_CINS) p->indel = cigar>>BAM_CIGAR_SHIFT; // ins - } - } + uint32_t *cigar = bam1_cigar(b); + int k, is_head = 0; + // determine the current CIGAR operation +// fprintf(stderr, "%s\tpos=%d\tend=%d\t(%d,%d,%d)\n", bam1_qname(b), pos, s->end, s->k, s->x, s->y); + if (s->k == -1) { // never processed + is_head = 1; + if (c->n_cigar == 1) { // just one operation, save a loop + if (_cop(cigar[0]) == BAM_CMATCH) s->k = 0, s->x = c->pos, s->y = 0; + } else { // find the first match + for (k = 0, s->x = c->pos, s->y = 0; k < c->n_cigar; ++k) { + int op = _cop(cigar[k]); + int l = _cln(cigar[k]); + if (op == BAM_CMATCH) break; + else if (op == BAM_CDEL || op == BAM_CREF_SKIP) s->x += l; + else if (op == BAM_CINS || op == BAM_CSOFT_CLIP) s->y += l; + } + assert(k < c->n_cigar); + s->k = k; + } + } else { // the read has been processed before + int op, l = _cln(cigar[s->k]); + if (pos - s->x >= l) { // jump to the next operation + assert(s->k < c->n_cigar); // otherwise a bug: this function should not be called in this case + op = _cop(cigar[s->k+1]); + if (op == BAM_CMATCH || op == BAM_CDEL || op == BAM_CREF_SKIP) { // jump to the next without a loop + if (_cop(cigar[s->k]) == BAM_CMATCH) s->y += l; + s->x += l; + ++s->k; + } else { // find the next M/D/N + if (_cop(cigar[s->k]) == BAM_CMATCH) s->y += l; + s->x += l; + for (k = s->k + 1; k < c->n_cigar; ++k) { + op = _cop(cigar[k]), l = _cln(cigar[k]); + if (op == BAM_CMATCH || op == BAM_CDEL || op == BAM_CREF_SKIP) break; + else if (op == BAM_CINS || op == BAM_CSOFT_CLIP) s->y += l; } + s->k = k; } - x += l; y += l; - } else if (op == BAM_CDEL) { // then set ->is_del - if (x + l > pos) { - p->indel = 0; p->is_del = 1; - p->qpos = y + (pos - x); + assert(s->k < c->n_cigar); // otherwise a bug + } // else, do nothing + } + { // collect pileup information + int op, l; + op = _cop(cigar[s->k]); l = _cln(cigar[s->k]); + p->is_del = p->indel = p->is_refskip = 0; + if (s->x + l - 1 == pos && s->k + 1 < c->n_cigar) { // peek the next operation + int op2 = _cop(cigar[s->k+1]); + int l2 = _cln(cigar[s->k+1]); + if (op2 == BAM_CDEL) p->indel = -(int)l2; + else if (op2 == BAM_CINS) p->indel = l2; + else if (op2 == BAM_CPAD && s->k + 2 < c->n_cigar) { // no working for adjacent padding + int l3 = 0; + for (k = s->k + 2; k < c->n_cigar; ++k) { + op2 = _cop(cigar[k]); l2 = _cln(cigar[k]); + if (op2 == BAM_CINS) l3 += l2; + else if (op2 == BAM_CDEL || op2 == BAM_CMATCH || op2 == BAM_CREF_SKIP) break; + } + if (l3 > 0) p->indel = l3; } - x += l; - } else if (op == BAM_CREF_SKIP) x += l; - else if (op == BAM_CINS || op == BAM_CSOFT_CLIP) y += l; - if (is_restart) is_restart ^= (op == BAM_CMATCH); - else is_restart ^= (op == BAM_CREF_SKIP || op == BAM_CSOFT_CLIP || op == BAM_CHARD_CLIP); - if (x > pos) { - if (op == BAM_CREF_SKIP) ret = 0; // then do not put it into pileup at all - break; } + if (op == BAM_CMATCH) { + p->qpos = s->y + (pos - s->x); + } else if (op == BAM_CDEL || op == BAM_CREF_SKIP) { + p->is_del = 1; p->qpos = s->y; // FIXME: distinguish D and N!!!!! + p->is_refskip = (op == BAM_CREF_SKIP); + } // cannot be other operations; otherwise a bug + p->is_head = (pos == c->pos); p->is_tail = (pos == s->end); } - assert(x > pos); // otherwise a bug - return ret; + return 1; } /* --- END: Auxiliary functions */ @@ -183,7 +208,7 @@ const bam_pileup1_t *bam_plp_next(bam_plp_t iter, int *_tid, int *_pos, int *_n_ iter->plp = (bam_pileup1_t*)realloc(iter->plp, sizeof(bam_pileup1_t) * iter->max_plp); } iter->plp[n_plp].b = &p->b; - if (resolve_cigar(iter->plp + n_plp, iter->pos)) ++n_plp; // skip the read if we are looking at ref-skip + if (resolve_cigar2(iter->plp + n_plp, iter->pos, &p->s)) ++n_plp; // actually always true... } } iter->head = iter->dummy->next; // dummy->next may be changed @@ -217,6 +242,7 @@ int bam_plp_push(bam_plp_t iter, const bam1_t *b) if (b->core.flag & iter->flag_mask) return 0; bam_copy1(&iter->tail->b, b); iter->tail->beg = b->core.pos; iter->tail->end = bam_calend(&b->core, bam1_cigar(b)); + iter->tail->s = g_cstate_null; iter->tail->s.end = iter->tail->end - 1; // initialize cstate_t if (b->core.tid < iter->max_tid) { fprintf(stderr, "[bam_pileup_core] the input is not sorted (chromosomes out of order)\n"); iter->error = 1; @@ -387,7 +413,7 @@ int bam_mplp_auto(bam_mplp_t iter, int *_tid, int *_pos, int *n_plp, const bam_p if (new_min == (uint64_t)-1) return 0; *_tid = new_min>>32; *_pos = (uint32_t)new_min; for (i = 0; i < iter->n; ++i) { - if (iter->pos[i] == iter->min) { + if (iter->pos[i] == iter->min) { // FIXME: valgrind reports "uninitialised value(s) at this line" n_plp[i] = iter->n_plp[i], plp[i] = iter->plp[i]; ++ret; } else n_plp[i] = 0, plp[i] = 0; diff --git a/bam_plcmd.c b/bam_plcmd.c index 6804795..dca8518 100644 --- a/bam_plcmd.c +++ b/bam_plcmd.c @@ -22,6 +22,7 @@ KHASH_MAP_INIT_INT64(64, indel_list_t) #define BAM_PLF_1STBASE 0x80 #define BAM_PLF_ALLBASE 0x100 #define BAM_PLF_READPOS 0x200 +#define BAM_PLF_NOBAQ 0x400 typedef struct { bam_header_t *h; @@ -32,6 +33,7 @@ typedef struct { uint32_t format; int tid, len, last_pos; int mask; + int capQ_thres, min_baseQ; int max_depth; // for indel calling, ignore reads with the depth too high. 0 for unlimited char *ref; glfFile fp_glf; // for glf output only @@ -89,6 +91,21 @@ static khash_t(64) *load_pos(const char *fn, bam_header_t *h) return hash; } +static inline int printw(int c, FILE *fp) +{ + char buf[16]; + int l, x; + if (c == 0) return fputc('0', fp); + for (l = 0, x = c < 0? -c : c; x > 0; x /= 10) buf[l++] = x%10 + '0'; + if (c < 0) buf[l++] = '-'; + buf[l] = 0; + for (x = 0; x < l/2; ++x) { + int y = buf[x]; buf[x] = buf[l-1-x]; buf[l-1-x] = y; + } + fputs(buf, fp); + return 0; +} + // an analogy to pileup_func() below static int glt3_func(uint32_t tid, uint32_t pos, int n, const bam_pileup1_t *pu, void *data) { @@ -158,29 +175,38 @@ static int glt3_func(uint32_t tid, uint32_t pos, int n, const bam_pileup1_t *pu, return 0; } -static void pileup_seq(const bam_pileup1_t *p, int pos, int ref_len, const char *ref) +static inline void pileup_seq(const bam_pileup1_t *p, int pos, int ref_len, const char *ref) { - if (p->is_head) printf("^%c", p->b->core.qual > 93? 126 : p->b->core.qual + 33); + int j; + if (p->is_head) { + putchar('^'); + putchar(p->b->core.qual > 93? 126 : p->b->core.qual + 33); + } if (!p->is_del) { - int j, rb, c = bam_nt16_rev_table[bam1_seqi(bam1_seq(p->b), p->qpos)]; - rb = (ref && pos < ref_len)? ref[pos] : 'N'; - if (c == '=' || toupper(c) == toupper(rb)) c = bam1_strand(p->b)? ',' : '.'; - else c = bam1_strand(p->b)? tolower(c) : toupper(c); + int c = bam_nt16_rev_table[bam1_seqi(bam1_seq(p->b), p->qpos)]; + if (ref) { + int rb = pos < ref_len? ref[pos] : 'N'; + if (c == '=' || bam_nt16_table[c] == bam_nt16_table[rb]) c = bam1_strand(p->b)? ',' : '.'; + else c = bam1_strand(p->b)? tolower(c) : toupper(c); + } else { + if (c == '=') c = bam1_strand(p->b)? ',' : '.'; + else c = bam1_strand(p->b)? tolower(c) : toupper(c); + } putchar(c); - if (p->indel > 0) { - printf("+%d", p->indel); - for (j = 1; j <= p->indel; ++j) { - c = bam_nt16_rev_table[bam1_seqi(bam1_seq(p->b), p->qpos + j)]; - putchar(bam1_strand(p->b)? tolower(c) : toupper(c)); - } - } else if (p->indel < 0) { - printf("%d", p->indel); - for (j = 1; j <= -p->indel; ++j) { - c = (ref && (int)pos+j < ref_len)? ref[pos+j] : 'N'; - putchar(bam1_strand(p->b)? tolower(c) : toupper(c)); - } + } else putchar(p->is_refskip? (bam1_strand(p->b)? '<' : '>') : '*'); + if (p->indel > 0) { + putchar('+'); printw(p->indel, stdout); + for (j = 1; j <= p->indel; ++j) { + int c = bam_nt16_rev_table[bam1_seqi(bam1_seq(p->b), p->qpos + j)]; + putchar(bam1_strand(p->b)? tolower(c) : toupper(c)); + } + } else if (p->indel < 0) { + printw(p->indel, stdout); + for (j = 1; j <= -p->indel; ++j) { + int c = (ref && (int)pos+j < ref_len)? ref[pos+j] : 'N'; + putchar(bam1_strand(p->b)? tolower(c) : toupper(c)); } - } else putchar('*'); + } if (p->is_tail) putchar('$'); } @@ -232,7 +258,11 @@ static int pileup_func(uint32_t tid, uint32_t pos, int n, const bam_pileup1_t *p } rmsQ = (uint64_t)(sqrt((double)rmsQ / n) + .499); cns = b<<28 | 0xf<<24 | rmsQ<<16 | 60<<8; - } else cns = bam_maqcns_call(n, pu, d->c); + } else { + glf1_t *g = bam_maqcns_glfgen(n, pu, bam_nt16_table[rb], d->c); + cns = g->depth == 0? (0xfu<<28 | 0xf<<24) : glf2cns(g, (int)(d->c->q_r + .499)); + free(g); + } } if ((d->format & (BAM_PLF_CNS|BAM_PLF_INDEL_ONLY)) && d->ref && pos < d->len) { // call indels int m = (!d->max_depth || d->max_depth>n) ? n : d->max_depth; @@ -242,7 +272,7 @@ static int pileup_func(uint32_t tid, uint32_t pos, int n, const bam_pileup1_t *p } // when only variant sites are asked for, test if the site is a variant if ((d->format & BAM_PLF_CNS) && (d->format & BAM_PLF_VAR_ONLY)) { - if (!(bam_nt16_table[rb] != 15 && cns>>28 != bam_nt16_table[rb])) { // not a SNP + if (!(bam_nt16_table[rb] != 15 && cns>>28 != 15 && cns>>28 != bam_nt16_table[rb])) { // not a SNP if (!(r && (r->gt == 2 || strcmp(r->s[r->gt], "*")))) { // not an indel if (r) bam_maqindel_ret_destroy(r); return 0; @@ -250,30 +280,29 @@ static int pileup_func(uint32_t tid, uint32_t pos, int n, const bam_pileup1_t *p } } // print the first 3 columns - printf("%s\t%d\t%c\t", d->h->target_name[tid], pos + 1, rb); + fputs(d->h->target_name[tid], stdout); putchar('\t'); + printw(pos+1, stdout); putchar('\t'); putchar(rb); putchar('\t'); // print consensus information if required if (d->format & BAM_PLF_CNS) { - int ref_q, rb4 = bam_nt16_table[rb]; - ref_q = 0; - if (rb4 != 15 && cns>>28 != 15 && cns>>28 != rb4) { // a SNP - ref_q = ((cns>>24&0xf) == rb4)? cns>>8&0xff : (cns>>8&0xff) + (cns&0xff); - if (ref_q > 255) ref_q = 255; - } - rms_mapq = cns>>16&0xff; - printf("%c\t%d\t%d\t%d\t", bam_nt16_rev_table[cns>>28], cns>>8&0xff, ref_q, rms_mapq); + putchar(bam_nt16_rev_table[cns>>28]); putchar('\t'); + printw(cns>>8&0xff, stdout); putchar('\t'); + printw(cns&0xff, stdout); putchar('\t'); + printw(cns>>16&0xff, stdout); putchar('\t'); } // print pileup sequences - printf("%d\t", n); - rms_aux = 0; // we need to recalculate rms_mapq when -c is not flagged on the command line - for (i = 0; i < n; ++i) { - const bam_pileup1_t *p = pu + i; - int tmp = p->b->core.qual < d->c->cap_mapQ? p->b->core.qual : d->c->cap_mapQ; - rms_aux += tmp * tmp; - pileup_seq(p, pos, d->len, d->ref); - } + printw(n, stdout); putchar('\t'); + for (i = 0; i < n; ++i) + pileup_seq(pu + i, pos, d->len, d->ref); // finalize rms_mapq - rms_aux = (uint64_t)(sqrt((double)rms_aux / n) + .499); - if (rms_mapq < 0) rms_mapq = rms_aux; + if (d->format & BAM_PLF_CNS) { + for (i = rms_aux = 0; i < n; ++i) { + const bam_pileup1_t *p = pu + i; + int tmp = p->b->core.qual < d->c->cap_mapQ? p->b->core.qual : d->c->cap_mapQ; + rms_aux += tmp * tmp; + } + rms_aux = (uint64_t)(sqrt((double)rms_aux / n) + .499); + if (rms_mapq < 0) rms_mapq = rms_aux; + } putchar('\t'); // print quality for (i = 0; i < n; ++i) { @@ -312,7 +341,7 @@ static int pileup_func(uint32_t tid, uint32_t pos, int n, const bam_pileup1_t *p for (i = 0; i < n; ++i) { int x = pu[i].qpos; int l = pu[i].b->core.l_qseq; - printf("%d,", x < l/2? x+1 : -((l-1)-x+1)); + printw(x < l/2? x+1 : -((l-1)-x+1), stdout); putchar(','); } } putchar('\n'); @@ -338,15 +367,17 @@ int bam_pileup(int argc, char *argv[]) int c, is_SAM = 0; char *fn_list = 0, *fn_fa = 0, *fn_pos = 0; pu_data_t *d = (pu_data_t*)calloc(1, sizeof(pu_data_t)); - d->max_depth = 0; - d->tid = -1; d->mask = BAM_DEF_MASK; + d->max_depth = 1024; d->tid = -1; d->mask = BAM_DEF_MASK; d->min_baseQ = 13; d->c = bam_maqcns_init(); - d->c->is_soap = 1; // change the default model + d->c->errmod = BAM_ERRMOD_MAQ2; // change the default model d->ido = bam_maqindel_opt_init(); - while ((c = getopt(argc, argv, "st:f:cT:N:r:l:d:im:gI:G:vM:S2aR:PA")) >= 0) { + while ((c = getopt(argc, argv, "st:f:cT:N:r:l:d:im:gI:G:vM:S2aR:PAQ:C:B")) >= 0) { switch (c) { - case 'a': d->c->is_soap = 1; break; - case 'A': d->c->is_soap = 0; break; + case 'Q': d->c->min_baseQ = atoi(optarg); break; + case 'C': d->capQ_thres = atoi(optarg); break; + case 'B': d->format |= BAM_PLF_NOBAQ; break; + case 'a': d->c->errmod = BAM_ERRMOD_SOAP; break; + case 'A': d->c->errmod = BAM_ERRMOD_MAQ; break; case 's': d->format |= BAM_PLF_SIMPLE; break; case 't': fn_list = strdup(optarg); break; case 'l': fn_pos = strdup(optarg); break; @@ -375,29 +406,34 @@ int bam_pileup(int argc, char *argv[]) default: fprintf(stderr, "Unrecognizd option '-%c'.\n", c); return 1; } } + if (d->c->errmod != BAM_ERRMOD_MAQ2) d->c->theta += 0.02; + if (d->c->theta > 1.0) d->c->theta = 1.0; if (fn_list) is_SAM = 1; if (optind == argc) { fprintf(stderr, "\n"); fprintf(stderr, "Usage: samtools pileup [options] |\n\n"); fprintf(stderr, "Option: -s simple (yet incomplete) pileup format\n"); fprintf(stderr, " -S the input is in SAM\n"); - fprintf(stderr, " -A use the MAQ model for SNP calling\n"); + fprintf(stderr, " -B disable BAQ computation\n"); + fprintf(stderr, " -A use the original MAQ model for SNP calling (DEPRECATED)\n"); fprintf(stderr, " -2 output the 2nd best call and quality\n"); fprintf(stderr, " -i only show lines/consensus with indels\n"); - fprintf(stderr, " -m INT filtering reads with bits in INT [%d]\n", d->mask); + fprintf(stderr, " -Q INT min base quality (possibly capped by BAQ) [%d]\n", d->c->min_baseQ); + fprintf(stderr, " -C INT coefficient for adjusting mapQ of poor mappings [%d]\n", d->capQ_thres); + fprintf(stderr, " -m INT filtering reads with bits in INT [0x%x]\n", d->mask); fprintf(stderr, " -M INT cap mapping quality at INT [%d]\n", d->c->cap_mapQ); - fprintf(stderr, " -d INT limit maximum depth for indels [unlimited]\n"); + fprintf(stderr, " -d INT limit maximum depth for indels [%d]\n", d->max_depth); fprintf(stderr, " -t FILE list of reference sequences (force -S)\n"); fprintf(stderr, " -l FILE list of sites at which pileup is output\n"); fprintf(stderr, " -f FILE reference sequence in the FASTA format\n\n"); - fprintf(stderr, " -c output the SOAPsnp consensus sequence\n"); + fprintf(stderr, " -c compute the consensus sequence\n"); fprintf(stderr, " -v print variants only (for -c)\n"); - fprintf(stderr, " -g output in the GLFv3 format (suppressing -c/-i/-s)\n"); - fprintf(stderr, " -T FLOAT theta in maq consensus calling model (for -c/-g) [%f]\n", d->c->theta); - fprintf(stderr, " -N INT number of haplotypes in the sample (for -c/-g) [%d]\n", d->c->n_hap); - fprintf(stderr, " -r FLOAT prior of a difference between two haplotypes (for -c/-g) [%f]\n", d->c->het_rate); - fprintf(stderr, " -G FLOAT prior of an indel between two haplotypes (for -c/-g) [%f]\n", d->ido->r_indel); - fprintf(stderr, " -I INT phred prob. of an indel in sequencing/prep. (for -c/-g) [%d]\n", d->ido->q_indel); + fprintf(stderr, " -g output in the GLFv3 format (DEPRECATED)\n"); + fprintf(stderr, " -T FLOAT theta in maq consensus calling model (for -c) [%.4g]\n", d->c->theta); + fprintf(stderr, " -N INT number of haplotypes in the sample (for -c) [%d]\n", d->c->n_hap); + fprintf(stderr, " -r FLOAT prior of a difference between two haplotypes (for -c) [%.4g]\n", d->c->het_rate); + fprintf(stderr, " -G FLOAT prior of an indel between two haplotypes (for -c) [%.4g]\n", d->ido->r_indel); + fprintf(stderr, " -I INT phred prob. of an indel in sequencing/prep. (for -c) [%d]\n", d->ido->q_indel); fprintf(stderr, "\n"); free(fn_list); free(fn_fa); free(d); return 1; @@ -425,7 +461,43 @@ int bam_pileup(int argc, char *argv[]) } d->h = fp->header; if (fn_pos) d->hash = load_pos(fn_pos, d->h); - sampileup(fp, d->mask, pileup_func, d); + { // run pileup + extern int bam_prob_realn(bam1_t *b, const char *ref); + extern int bam_cap_mapQ(bam1_t *b, char *ref, int thres); + bam1_t *b; + int ret, tid, pos, n_plp; + bam_plp_t iter; + const bam_pileup1_t *plp; + b = bam_init1(); + iter = bam_plp_init(0, 0); + bam_plp_set_mask(iter, d->mask); + while ((ret = samread(fp, b)) >= 0) { + int skip = 0; + if ((int)b->core.tid < 0) break; + // update d->ref if necessary + if (d->fai && (int)b->core.tid != d->tid) { + free(d->ref); + d->ref = faidx_fetch_seq(d->fai, d->h->target_name[b->core.tid], 0, 0x7fffffff, &d->len); + d->tid = b->core.tid; + } + if (d->ref && (d->format&BAM_PLF_CNS) && !(d->format&BAM_PLF_NOBAQ)) bam_prob_realn(b, d->ref); + if (d->ref && (d->format&BAM_PLF_CNS) && d->capQ_thres > 10) { + int q = bam_cap_mapQ(b, d->ref, d->capQ_thres); + if (q < 0) skip = 1; + else if (b->core.qual > q) b->core.qual = q; + } else if (b->core.flag&BAM_FUNMAP) skip = 1; + else if ((d->format&BAM_PLF_CNS) && (b->core.flag&1) && !(b->core.flag&2)) skip = 1; + if (skip) continue; + bam_plp_push(iter, b); + while ((plp = bam_plp_next(iter, &tid, &pos, &n_plp)) != 0) + pileup_func(tid, pos, n_plp, plp, d); + } + bam_plp_push(iter, 0); + while ((plp = bam_plp_next(iter, &tid, &pos, &n_plp)) != 0) + pileup_func(tid, pos, n_plp, plp, d); + bam_plp_destroy(iter); + bam_destroy1(b); + } samclose(fp); // d->h will be destroyed here } @@ -448,21 +520,80 @@ int bam_pileup(int argc, char *argv[]) * mpileup * ***********/ +#include +#include "bam2bcf.h" +#include "sample.h" + +#define MPLP_GLF 0x10 +#define MPLP_NO_COMP 0x20 +#define MPLP_NO_ORPHAN 0x40 +#define MPLP_REALN 0x80 + typedef struct { - char *reg; + int max_mq, min_mq, flag, min_baseQ, capQ_thres; + char *reg, *fn_pos; faidx_t *fai; + kh_64_t *hash; } mplp_conf_t; typedef struct { bamFile fp; bam_iter_t iter; + int min_mq, flag, ref_id, capQ_thres; + char *ref; } mplp_aux_t; +typedef struct { + int n; + int *n_plp, *m_plp; + bam_pileup1_t **plp; +} mplp_pileup_t; + static int mplp_func(void *data, bam1_t *b) { + extern int bam_realn(bam1_t *b, const char *ref); + extern int bam_prob_realn(bam1_t *b, const char *ref); + extern int bam_cap_mapQ(bam1_t *b, char *ref, int thres); mplp_aux_t *ma = (mplp_aux_t*)data; - if (ma->iter) return bam_iter_read(ma->fp, ma->iter, b); - return bam_read1(ma->fp, b); + int ret, skip = 0; + do { + int has_ref = (ma->ref && ma->ref_id == b->core.tid)? 1 : 0; + ret = ma->iter? bam_iter_read(ma->fp, ma->iter, b) : bam_read1(ma->fp, b); + if (ret < 0) break; + skip = 0; + if (has_ref && (ma->flag&MPLP_REALN)) bam_prob_realn(b, ma->ref); + if (has_ref && ma->capQ_thres > 10) { + int q = bam_cap_mapQ(b, ma->ref, ma->capQ_thres); + if (q < 0) skip = 1; + else if (b->core.qual > q) b->core.qual = q; + } else if (b->core.flag&BAM_FUNMAP) skip = 1; + else if (b->core.qual < ma->min_mq) skip = 1; + else if ((ma->flag&MPLP_NO_ORPHAN) && (b->core.flag&1) && !(b->core.flag&2)) skip = 1; + } while (skip); + return ret; +} + +static void group_smpl(mplp_pileup_t *m, bam_sample_t *sm, kstring_t *buf, + int n, char *const*fn, int *n_plp, const bam_pileup1_t **plp) +{ + int i, j; + memset(m->n_plp, 0, m->n * sizeof(int)); + for (i = 0; i < n; ++i) { + for (j = 0; j < n_plp[i]; ++j) { + const bam_pileup1_t *p = plp[i] + j; + uint8_t *q; + int id = -1; + q = bam_aux_get(p->b, "RG"); + if (q) id = bam_smpl_rg2smid(sm, fn[i], (char*)q+1, buf); + if (id < 0) id = bam_smpl_rg2smid(sm, fn[i], 0, buf); + assert(id >= 0 && id < m->n); + if (m->n_plp[id] == m->m_plp[id]) { + m->m_plp[id] = m->m_plp[id]? m->m_plp[id]<<1 : 8; + m->plp[id] = realloc(m->plp[id], sizeof(bam_pileup1_t) * m->m_plp[id]); + } + m->plp[id][m->n_plp[id]++] = *p; + } + } } static int mpileup(mplp_conf_t *conf, int n, char **fn) @@ -473,16 +604,36 @@ static int mpileup(mplp_conf_t *conf, int n, char **fn) bam_mplp_t iter; bam_header_t *h = 0; char *ref; - // allocate + khash_t(64) *hash = 0; + + bcf_callaux_t *bca = 0; + bcf_callret1_t *bcr = 0; + bcf_call_t bc; + bcf_t *bp = 0; + bcf_hdr_t *bh = 0; + + bam_sample_t *sm = 0; + kstring_t buf; + mplp_pileup_t gplp; + + memset(&gplp, 0, sizeof(mplp_pileup_t)); + memset(&buf, 0, sizeof(kstring_t)); + memset(&bc, 0, sizeof(bcf_call_t)); data = calloc(n, sizeof(void*)); plp = calloc(n, sizeof(void*)); n_plp = calloc(n, sizeof(int*)); + sm = bam_smpl_init(); + // read the header and initialize data for (i = 0; i < n; ++i) { bam_header_t *h_tmp; data[i] = calloc(1, sizeof(mplp_aux_t)); - data[i]->fp = bam_open(fn[i], "r"); + data[i]->min_mq = conf->min_mq; + data[i]->flag = conf->flag; + data[i]->capQ_thres = conf->capQ_thres; + data[i]->fp = strcmp(fn[i], "-") == 0? bam_dopen(fileno(stdin), "r") : bam_open(fn[i], "r"); h_tmp = bam_header_read(data[i]->fp); + bam_smpl_add(sm, fn[i], h_tmp->text); if (conf->reg) { int beg, end; bam_index_t *idx; @@ -505,35 +656,100 @@ static int mpileup(mplp_conf_t *conf, int n, char **fn) bam_header_destroy(h_tmp); } } - // mpileup + gplp.n = sm->n; + gplp.n_plp = calloc(sm->n, sizeof(int)); + gplp.m_plp = calloc(sm->n, sizeof(int)); + gplp.plp = calloc(sm->n, sizeof(void*)); + + fprintf(stderr, "[%s] %d samples in %d input files\n", __func__, sm->n, n); + if (conf->fn_pos) hash = load_pos(conf->fn_pos, h); + // write the VCF header + if (conf->flag & MPLP_GLF) { + kstring_t s; + bh = calloc(1, sizeof(bcf_hdr_t)); + s.l = s.m = 0; s.s = 0; + bp = bcf_open("-", (conf->flag&MPLP_NO_COMP)? "wu" : "w"); + for (i = 0; i < h->n_targets; ++i) { + kputs(h->target_name[i], &s); + kputc('\0', &s); + } + bh->l_nm = s.l; + bh->name = malloc(s.l); + memcpy(bh->name, s.s, s.l); + s.l = 0; + for (i = 0; i < sm->n; ++i) { + kputs(sm->smpl[i], &s); kputc('\0', &s); + } + bh->l_smpl = s.l; + bh->sname = malloc(s.l); + memcpy(bh->sname, s.s, s.l); + bh->l_txt = 0; + free(s.s); + bcf_hdr_sync(bh); + bcf_hdr_write(bp, bh); + bca = bcf_call_init(-1., conf->min_baseQ); + bcr = calloc(sm->n, sizeof(bcf_callret1_t)); + } ref_tid = -1; ref = 0; iter = bam_mplp_init(n, mplp_func, (void**)data); while (bam_mplp_auto(iter, &tid, &pos, n_plp, plp) > 0) { if (conf->reg && (pos < beg0 || pos >= end0)) continue; // out of the region requested + if (hash) { + khint_t k; + k = kh_get(64, hash, (uint64_t)tid<<32 | pos); + if (k == kh_end(hash)) continue; + } if (tid != ref_tid) { - free(ref); + free(ref); ref = 0; if (conf->fai) ref = fai_fetch(conf->fai, h->target_name[tid], &ref_len); + for (i = 0; i < n; ++i) data[i]->ref = ref, data[i]->ref_id = tid; ref_tid = tid; } - printf("%s\t%d\t%c", h->target_name[tid], pos + 1, (ref && pos < ref_len)? ref[pos] : 'N'); - for (i = 0; i < n; ++i) { - int j; - printf("\t%d\t", n_plp[i]); - if (n_plp[i] == 0) printf("*\t*"); - else { - for (j = 0; j < n_plp[i]; ++j) - pileup_seq(plp[i] + j, pos, ref_len, ref); - putchar('\t'); - for (j = 0; j < n_plp[i]; ++j) { - const bam_pileup1_t *p = plp[i] + j; - int c = bam1_qual(p->b)[p->qpos] + 33; - if (c > 126) c = 126; - putchar(c); + if (conf->flag & MPLP_GLF) { + int _ref0, ref16; + bcf1_t *b = calloc(1, sizeof(bcf1_t)); + group_smpl(&gplp, sm, &buf, n, fn, n_plp, plp); + _ref0 = (ref && pos < ref_len)? ref[pos] : 'N'; + ref16 = bam_nt16_table[_ref0]; + for (i = 0; i < gplp.n; ++i) + bcf_call_glfgen(gplp.n_plp[i], gplp.plp[i], ref16, bca, bcr + i); + bcf_call_combine(gplp.n, bcr, ref16, &bc); + bcf_call2bcf(tid, pos, &bc, b); + bcf_write(bp, bh, b); + bcf_destroy(b); + } else { + printf("%s\t%d\t%c", h->target_name[tid], pos + 1, (ref && pos < ref_len)? ref[pos] : 'N'); + for (i = 0; i < n; ++i) { + int j; + printf("\t%d\t", n_plp[i]); + if (n_plp[i] == 0) printf("*\t*"); + else { + for (j = 0; j < n_plp[i]; ++j) + pileup_seq(plp[i] + j, pos, ref_len, ref); + putchar('\t'); + for (j = 0; j < n_plp[i]; ++j) { + const bam_pileup1_t *p = plp[i] + j; + int c = bam1_qual(p->b)[p->qpos] + 33; + if (c > 126) c = 126; + putchar(c); + } } } + putchar('\n'); } - putchar('\n'); } + + bcf_close(bp); + bam_smpl_destroy(sm); free(buf.s); + for (i = 0; i < gplp.n; ++i) free(gplp.plp[i]); + free(gplp.plp); free(gplp.n_plp); free(gplp.m_plp); + if (hash) { // free the hash table + khint_t k; + for (k = kh_begin(hash); k < kh_end(hash); ++k) + if (kh_exist(hash, k)) free(kh_val(hash, k)); + kh_destroy(64, hash); + } + bcf_hdr_destroy(bh); bcf_call_destroy(bca); free(bc.PL); free(bcr); bam_mplp_destroy(iter); bam_header_destroy(h); for (i = 0; i < n; ++i) { @@ -550,17 +766,44 @@ int bam_mpileup(int argc, char *argv[]) int c; mplp_conf_t mplp; memset(&mplp, 0, sizeof(mplp_conf_t)); - while ((c = getopt(argc, argv, "f:r:")) >= 0) { + mplp.max_mq = 60; + mplp.min_baseQ = 13; + mplp.capQ_thres = 0; + mplp.flag = MPLP_NO_ORPHAN | MPLP_REALN; + while ((c = getopt(argc, argv, "gf:r:l:M:q:Q:uaORC:B")) >= 0) { switch (c) { case 'f': mplp.fai = fai_load(optarg); if (mplp.fai == 0) return 1; break; - case 'r': mplp.reg = strdup(optarg); + case 'r': mplp.reg = strdup(optarg); break; + case 'l': mplp.fn_pos = strdup(optarg); break; + case 'g': mplp.flag |= MPLP_GLF; break; + case 'u': mplp.flag |= MPLP_NO_COMP | MPLP_GLF; break; + case 'a': mplp.flag |= MPLP_NO_ORPHAN | MPLP_REALN; break; + case 'B': mplp.flag &= ~MPLP_REALN & ~MPLP_NO_ORPHAN; break; + case 'O': mplp.flag |= MPLP_NO_ORPHAN; break; + case 'R': mplp.flag |= MPLP_REALN; break; + case 'C': mplp.capQ_thres = atoi(optarg); break; + case 'M': mplp.max_mq = atoi(optarg); break; + case 'q': mplp.min_mq = atoi(optarg); break; + case 'Q': mplp.min_baseQ = atoi(optarg); break; } } if (argc == 1) { - fprintf(stderr, "Usage: samtools mpileup [-r reg] [-f in.fa] in1.bam [in2.bam [...]]\n"); + fprintf(stderr, "\n"); + fprintf(stderr, "Usage: samtools mpileup [options] in1.bam [in2.bam [...]]\n\n"); + fprintf(stderr, "Options: -f FILE reference sequence file [null]\n"); + fprintf(stderr, " -r STR region in which pileup is generated [null]\n"); + fprintf(stderr, " -l FILE list of positions (format: chr pos) [null]\n"); + fprintf(stderr, " -M INT cap mapping quality at INT [%d]\n", mplp.max_mq); + fprintf(stderr, " -Q INT min base quality [%d]\n", mplp.min_baseQ); + fprintf(stderr, " -q INT filter out alignment with MQ smaller than INT [%d]\n", mplp.min_mq); + fprintf(stderr, " -g generate BCF output\n"); + fprintf(stderr, " -u do not compress BCF output\n"); + fprintf(stderr, " -B disable BAQ computation\n"); + fprintf(stderr, "\n"); + fprintf(stderr, "Notes: Assuming diploid individuals.\n\n"); return 1; } mpileup(&mplp, argc - optind, argv + optind); diff --git a/bam_sort.c b/bam_sort.c index 12b1b54..76ab793 100644 --- a/bam_sort.c +++ b/bam_sort.c @@ -1,6 +1,7 @@ #include #include #include +#include #include #include #include @@ -59,6 +60,9 @@ static void swap_header_text(bam_header_t *h1, bam_header_t *h2) temps = h1->text, h1->text = h2->text, h2->text = temps; } +#define MERGE_RG 1 +#define MERGE_UNCOMP 2 + /*! @abstract Merge multiple sorted BAM. @param is_by_qname whether to sort by query name @@ -71,7 +75,8 @@ static void swap_header_text(bam_header_t *h1, bam_header_t *h2) @discussion Padding information may NOT correctly maintained. This function is NOT thread safe. */ -void bam_merge_core(int by_qname, const char *out, const char *headers, int n, char * const *fn, int add_RG) +int bam_merge_core(int by_qname, const char *out, const char *headers, int n, char * const *fn, + int flag, const char *reg) { bamFile fpout, *fp; heap1_t *heap; @@ -80,22 +85,25 @@ void bam_merge_core(int by_qname, const char *out, const char *headers, int n, c int i, j, *RG_len = 0; uint64_t idx = 0; char **RG = 0; + bam_iter_t *iter = 0; if (headers) { tamFile fpheaders = sam_open(headers); if (fpheaders == 0) { - fprintf(stderr, "[bam_merge_core] Cannot open file `%s'. Continue anyway.\n", headers); - } else { - hheaders = sam_header_read(fpheaders); - sam_close(fpheaders); + const char *message = strerror(errno); + fprintf(stderr, "[bam_merge_core] cannot open '%s': %s\n", headers, message); + return -1; } + hheaders = sam_header_read(fpheaders); + sam_close(fpheaders); } g_is_by_qname = by_qname; fp = (bamFile*)calloc(n, sizeof(bamFile)); heap = (heap1_t*)calloc(n, sizeof(heap1_t)); + iter = (bam_iter_t*)calloc(n, sizeof(bam_iter_t)); // prepare RG tag - if (add_RG) { + if (flag & MERGE_RG) { RG = (char**)calloc(n, sizeof(void*)); RG_len = (int*)calloc(n, sizeof(int)); for (i = 0; i != n; ++i) { @@ -111,7 +119,6 @@ void bam_merge_core(int by_qname, const char *out, const char *headers, int n, c } // read the first for (i = 0; i != n; ++i) { - heap1_t *h; bam_header_t *hin; fp[i] = bam_open(fn[i], "r"); if (fp[i] == 0) { @@ -120,7 +127,7 @@ void bam_merge_core(int by_qname, const char *out, const char *headers, int n, c for (j = 0; j < i; ++j) bam_close(fp[j]); free(fp); free(heap); // FIXME: possible memory leak - return; + return -1; } hin = bam_header_read(fp[i]); if (i == 0) { // the first SAM @@ -130,11 +137,15 @@ void bam_merge_core(int by_qname, const char *out, const char *headers, int n, c // check that they are consistent with the existing binary list // of reference information. if (hheaders->n_targets > 0) { - if (hout->n_targets != hheaders->n_targets) + if (hout->n_targets != hheaders->n_targets) { fprintf(stderr, "[bam_merge_core] number of @SQ headers in `%s' differs from number of target sequences", headers); + if (!reg) return -1; + } for (j = 0; j < hout->n_targets; ++j) - if (strcmp(hout->target_name[j], hheaders->target_name[j]) != 0) + if (strcmp(hout->target_name[j], hheaders->target_name[j]) != 0) { fprintf(stderr, "[bam_merge_core] @SQ header '%s' in '%s' differs from target sequence", hheaders->target_name[j], headers); + if (!reg) return -1; + } } swap_header_text(hout, hheaders); bam_header_destroy(hheaders); @@ -142,39 +153,62 @@ void bam_merge_core(int by_qname, const char *out, const char *headers, int n, c } } else { // validate multiple baf if (hout->n_targets != hin->n_targets) { - fprintf(stderr, "[bam_merge_core] file '%s' has different number of target sequences. Abort!\n", fn[i]); - exit(1); - } - for (j = 0; j < hout->n_targets; ++j) { - if (strcmp(hout->target_name[j], hin->target_name[j])) { - fprintf(stderr, "[bam_merge_core] different target sequence name: '%s' != '%s' in file '%s'. Abort!\n", - hout->target_name[j], hin->target_name[j], fn[i]); - exit(1); + fprintf(stderr, "[bam_merge_core] file '%s' has different number of target sequences. Continue anyway!\n", fn[i]); + } else { + for (j = 0; j < hout->n_targets; ++j) { + if (strcmp(hout->target_name[j], hin->target_name[j])) { + fprintf(stderr, "[bam_merge_core] different target sequence name: '%s' != '%s' in file '%s'. Continue anyway!\n", + hout->target_name[j], hin->target_name[j], fn[i]); + } } } bam_header_destroy(hin); } - h = heap + i; + } + + if (reg) { + int tid, beg, end; + if (bam_parse_region(hout, reg, &tid, &beg, &end) < 0) { + fprintf(stderr, "[%s] Malformated region string or undefined reference name\n", __func__); + return -1; + } + for (i = 0; i < n; ++i) { + bam_index_t *idx; + idx = bam_index_load(fn[i]); + iter[i] = bam_iter_query(idx, tid, beg, end); + bam_index_destroy(idx); + } + } + + for (i = 0; i < n; ++i) { + heap1_t *h = heap + i; h->i = i; h->b = (bam1_t*)calloc(1, sizeof(bam1_t)); - if (bam_read1(fp[i], h->b) >= 0) { + if (bam_iter_read(fp[i], iter[i], h->b) >= 0) { h->pos = ((uint64_t)h->b->core.tid<<32) | (uint32_t)h->b->core.pos<<1 | bam1_strand(h->b); h->idx = idx++; } else h->pos = HEAP_EMPTY; } - fpout = strcmp(out, "-")? bam_open(out, "w") : bam_dopen(fileno(stdout), "w"); - assert(fpout); + if (flag & MERGE_UNCOMP) { + fpout = strcmp(out, "-")? bam_open(out, "wu") : bam_dopen(fileno(stdout), "wu"); + } else { + fpout = strcmp(out, "-")? bam_open(out, "w") : bam_dopen(fileno(stdout), "w"); + } + if (fpout == 0) { + fprintf(stderr, "[%s] fail to create the output file.\n", __func__); + return -1; + } bam_header_write(fpout, hout); bam_header_destroy(hout); ks_heapmake(heap, n, heap); while (heap->pos != HEAP_EMPTY) { bam1_t *b = heap->b; - if (add_RG && bam_aux_get(b, "RG") == 0) + if ((flag & MERGE_RG) && bam_aux_get(b, "RG") == 0) bam_aux_append(b, "RG", 'Z', RG_len[heap->i] + 1, (uint8_t*)RG[heap->i]); bam_write1_core(fpout, &b->core, b->data_len, b->data); - if ((j = bam_read1(fp[heap->i], b)) >= 0) { + if ((j = bam_iter_read(fp[heap->i], iter[heap->i], b)) >= 0) { heap->pos = ((uint64_t)b->core.tid<<32) | (uint32_t)b->core.pos<<1 | bam1_strand(b); heap->idx = idx++; } else if (j == -1) { @@ -185,24 +219,31 @@ void bam_merge_core(int by_qname, const char *out, const char *headers, int n, c ks_heapadjust(heap, 0, n, heap); } - if (add_RG) { + if (flag & MERGE_RG) { for (i = 0; i != n; ++i) free(RG[i]); free(RG); free(RG_len); } - for (i = 0; i != n; ++i) bam_close(fp[i]); + for (i = 0; i != n; ++i) { + bam_iter_destroy(iter[i]); + bam_close(fp[i]); + } bam_close(fpout); - free(fp); free(heap); + free(fp); free(heap); free(iter); + return 0; } + int bam_merge(int argc, char *argv[]) { - int c, is_by_qname = 0, add_RG = 0; - char *fn_headers = NULL; + int c, is_by_qname = 0, flag = 0, ret = 0; + char *fn_headers = NULL, *reg = 0; - while ((c = getopt(argc, argv, "h:nr")) >= 0) { + while ((c = getopt(argc, argv, "h:nruR:")) >= 0) { switch (c) { - case 'r': add_RG = 1; break; + case 'r': flag |= MERGE_RG; break; case 'h': fn_headers = strdup(optarg); break; case 'n': is_by_qname = 1; break; + case 'u': flag |= MERGE_UNCOMP; break; + case 'R': reg = strdup(optarg); break; } } if (optind + 2 >= argc) { @@ -210,15 +251,18 @@ int bam_merge(int argc, char *argv[]) fprintf(stderr, "Usage: samtools merge [-nr] [-h inh.sam] [...]\n\n"); fprintf(stderr, "Options: -n sort by read names\n"); fprintf(stderr, " -r attach RG tag (inferred from file names)\n"); + fprintf(stderr, " -u uncompressed BAM output\n"); + fprintf(stderr, " -R STR merge file in the specified region STR [all]\n"); fprintf(stderr, " -h FILE copy the header in FILE to [in1.bam]\n\n"); fprintf(stderr, "Note: Samtools' merge does not reconstruct the @RG dictionary in the header. Users\n"); fprintf(stderr, " must provide the correct header with -h, or uses Picard which properly maintains\n"); fprintf(stderr, " the header dictionary in merging.\n\n"); return 1; } - bam_merge_core(is_by_qname, argv[optind], fn_headers, argc - optind - 1, argv + optind + 1, add_RG); + if (bam_merge_core(is_by_qname, argv[optind], fn_headers, argc - optind - 1, argv + optind + 1, flag, reg) < 0) ret = 1; + free(reg); free(fn_headers); - return 0; + return ret; } typedef bam1_t *bam1_p; @@ -313,7 +357,7 @@ void bam_sort_core_ext(int is_by_qname, const char *fn, const char *prefix, size fns[i] = (char*)calloc(strlen(prefix) + 20, 1); sprintf(fns[i], "%s.%.4d.bam", prefix, i); } - bam_merge_core(is_by_qname, fnout, 0, n, fns, 0); + bam_merge_core(is_by_qname, fnout, 0, n, fns, 0, 0); free(fnout); for (i = 0; i < n; ++i) { unlink(fns[i]); diff --git a/bam_tview.c b/bam_tview.c index 7b326fc..e48afa7 100644 --- a/bam_tview.c +++ b/bam_tview.c @@ -109,7 +109,7 @@ int tv_pl_func(uint32_t tid, uint32_t pos, int n, const bam_pileup1_t *pl, void if (tv->is_dot && toupper(c) == toupper(rb)) c = bam1_strand(p->b)? ',' : '.'; } } - } else c = '*'; + } else c = p->is_refskip? (bam1_strand(p->b)? '<' : '>') : '*'; } else { // padding if (j > p->indel) c = '*'; else { // insertion @@ -292,7 +292,7 @@ static void tv_win_goto(tview_t *tv, int *tid, int *pos) } else if (c == KEY_ENTER || c == '\012' || c == '\015') { int _tid = -1, _beg, _end; if (str[0] == '=') { - _beg = strtol(str+1, &p, 10); + _beg = strtol(str+1, &p, 10) - 1; if (_beg > 0) { *pos = _beg; return; diff --git a/bamtk.c b/bamtk.c index 94c4d3f..082767c 100644 --- a/bamtk.c +++ b/bamtk.c @@ -9,7 +9,7 @@ #endif #ifndef PACKAGE_VERSION -#define PACKAGE_VERSION "0.1.8 (r613)" +#define PACKAGE_VERSION "0.1.9 (r783)" #endif int bam_taf2baf(int argc, char *argv[]); diff --git a/bcftools/Makefile b/bcftools/Makefile new file mode 100644 index 0000000..8b890ba --- /dev/null +++ b/bcftools/Makefile @@ -0,0 +1,51 @@ +CC= gcc +CFLAGS= -g -Wall -O2 #-m64 #-arch ppc +DFLAGS= -D_FILE_OFFSET_BITS=64 -D_USE_KNETFILE +LOBJS= bcf.o vcf.o bcfutils.o prob1.o ld.o kfunc.o index.o fet.o bcf2qcall.o +OMISC= .. +AOBJS= call1.o main.o $(OMISC)/kstring.o $(OMISC)/bgzf.o $(OMISC)/knetfile.o +PROG= bcftools +INCLUDES= +SUBDIRS= . + +.SUFFIXES:.c .o + +.c.o: + $(CC) -c $(CFLAGS) $(DFLAGS) -I.. $(INCLUDES) $< -o $@ + +all-recur lib-recur clean-recur cleanlocal-recur install-recur: + @target=`echo $@ | sed s/-recur//`; \ + wdir=`pwd`; \ + list='$(SUBDIRS)'; for subdir in $$list; do \ + cd $$subdir; \ + $(MAKE) CC="$(CC)" DFLAGS="$(DFLAGS)" CFLAGS="$(CFLAGS)" \ + INCLUDES="$(INCLUDES)" LIBPATH="$(LIBPATH)" $$target || exit 1; \ + cd $$wdir; \ + done; + +all:$(PROG) + +lib:libbcf.a + +libbcf.a:$(LOBJS) + $(AR) -cru $@ $(LOBJS) + +bcftools:lib $(AOBJS) + $(CC) $(CFLAGS) -o $@ $(AOBJS) -lm $(LIBPATH) -lz -L. -lbcf + +bcf.o:bcf.h +vcf.o:bcf.h +index.o:bcf.h +bcfutils.o:bcf.h +prob1.o:prob1.h bcf.h +call1.o:prob1.h bcf.h +bcf2qcall.o:bcf.h +main.o:bcf.h + +bcf.pdf:bcf.tex + pdflatex bcf + +cleanlocal: + rm -fr gmon.out *.o a.out *.dSYM $(PROG) *~ *.a bcf.aux bcf.log bcf.pdf *.class libbcf.*.dylib libbcf.so* + +clean:cleanlocal-recur diff --git a/bcftools/README b/bcftools/README new file mode 100644 index 0000000..1d7159d --- /dev/null +++ b/bcftools/README @@ -0,0 +1,36 @@ +The view command of bcftools calls variants, tests Hardy-Weinberg +equilibrium (HWE), tests allele balances and estimates allele frequency. + +This command calls a site as a potential variant if P(ref|D,F) is below +0.9 (controlled by the -p option), where D is data and F is the prior +allele frequency spectrum (AFS). + +The view command performs two types of allele balance tests, both based +on Fisher's exact test for 2x2 contingency tables with the row variable +being reference allele or not. In the first table, the column variable +is strand. Two-tail P-value is taken. We test if variant bases tend to +come from one strand. In the second table, the column variable is +whether a base appears in the first or the last 11bp of the read. +One-tail P-value is taken. We test if variant bases tend to occur +towards the end of reads, which is usually an indication of +misalignment. + +Site allele frequency is estimated in two ways. In the first way, the +frequency is esimated as \argmax_f P(D|f) under the assumption of +HWE. Prior AFS is not used. In the second way, the frequency is +estimated as the posterior expectation of allele counts \sum_k +kP(k|D,F), dividied by the total number of haplotypes. HWE is not +assumed, but the estimate depends on the prior AFS. The two estimates +largely agree when the signal is strong, but may differ greatly on weak +sites as in this case, the prior plays an important role. + +To test HWE, we calculate the posterior distribution of genotypes +(ref-hom, het and alt-hom). Chi-square test is performed. It is worth +noting that the model used here is prior dependent and assumes HWE, +which is different from both models for allele frequency estimate. The +new model actually yields a third estimate of site allele frequency. + +The estimate allele frequency spectrum is printed to stderr per 64k +sites. The estimate is in fact only the first round of a EM +procedure. The second model (not the model for HWE testing) is used to +estimate the AFS. \ No newline at end of file diff --git a/bcftools/bcf-fix.pl b/bcftools/bcf-fix.pl new file mode 100755 index 0000000..61c6136 --- /dev/null +++ b/bcftools/bcf-fix.pl @@ -0,0 +1,101 @@ +#!/usr/bin/perl -w + +use strict; +use warnings; +use Carp; + +my $opts = parse_params(); +bcf_fix(); + +exit; + +#-------------------------------- + +sub error +{ + my (@msg) = @_; + if ( scalar @msg ) { confess @msg; } + die + "Usage: bcftools view test.bcf | bcf-fix.pl > test.vcf\n", + "Options:\n", + " -h, -?, --help This help message.\n", + "\n"; +} + + +sub parse_params +{ + my $opts = {}; + while (my $arg=shift(@ARGV)) + { + if ( $arg eq '-?' || $arg eq '-h' || $arg eq '--help' ) { error(); } + error("Unknown parameter \"$arg\". Run -h for help.\n"); + } + return $opts; +} + +sub bcf_fix +{ + while (my $line=) + { + if ( $line=~/^#CHROM/ ) + { + print +qq[##INFO= +##INFO= +##INFO= +##INFO= +##INFO= +##INFO= +##FORMAT= +##FORMAT= +##FORMAT= +]; + print $line; + } + elsif ( $line=~/^#/ ) + { + print $line; + } + else + { + my @items = split(/\t/,$line); + my @tags = split(/:/,$items[8]); # FORMAT tags + + my $nidx=2; + my @idxs; # Mapping which defines new ordering: $idxs[$inew]=$iold; GT comes first, PL second + for (my $i=0; $i<@tags; $i++) + { + if ( $tags[$i] eq 'GT' ) { $idxs[0]=$i; } + elsif ( $tags[$i] eq 'PL' ) { $idxs[1]=$i; } + else { $idxs[$nidx++]=$i; } + } + if ( !exists($tags[0]) or !exists($tags[1]) ) { error("FIXME: expected GT and PL in the format field.\n"); } + + # First fix the FORMAT column + $items[8] = 'GT:GL'; + for (my $i=2; $i<@tags; $i++) + { + $items[8] .= ':'.$tags[$idxs[$i]]; + } + + # Now all the genotype columns + for (my $iitem=9; $iitem<@items; $iitem++) + { + @tags = split(/:/,$items[$iitem]); + $items[$iitem] = $tags[$idxs[0]] .':'; + + # GL=-PL/10 + my ($a,$b,$c) = split(/,/,$tags[$idxs[1]]); + $items[$iitem] .= sprintf "%.2f,%.2f,%.2f",-$a/10.,-$b/10.,-$c/10.; + + for (my $itag=2; $itag<@tags; $itag++) + { + $items[$iitem] .= ':'.$tags[$idxs[$itag]]; + } + } + print join("\t",@items); + } + } +} + diff --git a/bcftools/bcf.c b/bcftools/bcf.c new file mode 100644 index 0000000..9044b93 --- /dev/null +++ b/bcftools/bcf.c @@ -0,0 +1,306 @@ +#include +#include +#include +#include "kstring.h" +#include "bcf.h" + +bcf_t *bcf_open(const char *fn, const char *mode) +{ + bcf_t *b; + b = calloc(1, sizeof(bcf_t)); + if (strchr(mode, 'w')) { + b->fp = strcmp(fn, "-")? bgzf_open(fn, mode) : bgzf_fdopen(fileno(stdout), mode); + } else { + b->fp = strcmp(fn, "-")? bgzf_open(fn, mode) : bgzf_fdopen(fileno(stdin), mode); + } + b->fp->owned_file = 1; + return b; +} + +int bcf_close(bcf_t *b) +{ + int ret; + if (b == 0) return 0; + ret = bgzf_close(b->fp); + free(b); + return ret; +} + +int bcf_hdr_write(bcf_t *b, const bcf_hdr_t *h) +{ + if (b == 0 || h == 0) return -1; + bgzf_write(b->fp, "BCF\4", 4); + bgzf_write(b->fp, &h->l_nm, 4); + bgzf_write(b->fp, h->name, h->l_nm); + bgzf_write(b->fp, &h->l_smpl, 4); + bgzf_write(b->fp, h->sname, h->l_smpl); + bgzf_write(b->fp, &h->l_txt, 4); + bgzf_write(b->fp, h->txt, h->l_txt); + bgzf_flush(b->fp); + return 16 + h->l_nm + h->l_smpl + h->l_txt; +} + +bcf_hdr_t *bcf_hdr_read(bcf_t *b) +{ + uint8_t magic[4]; + bcf_hdr_t *h; + if (b == 0) return 0; + h = calloc(1, sizeof(bcf_hdr_t)); + bgzf_read(b->fp, magic, 4); + bgzf_read(b->fp, &h->l_nm, 4); + h->name = malloc(h->l_nm); + bgzf_read(b->fp, h->name, h->l_nm); + bgzf_read(b->fp, &h->l_smpl, 4); + h->sname = malloc(h->l_smpl); + bgzf_read(b->fp, h->sname, h->l_smpl); + bgzf_read(b->fp, &h->l_txt, 4); + h->txt = malloc(h->l_txt); + bgzf_read(b->fp, h->txt, h->l_txt); + bcf_hdr_sync(h); + return h; +} + +void bcf_hdr_destroy(bcf_hdr_t *h) +{ + if (h == 0) return; + free(h->name); free(h->sname); free(h->txt); free(h->ns); free(h->sns); + free(h); +} + +static inline char **cnt_null(int l, char *str, int *_n) +{ + int n = 0; + char *p, **list; + *_n = 0; + if (l == 0 || str == 0) return 0; + for (p = str; p != str + l; ++p) + if (*p == 0) ++n; + *_n = n; + list = calloc(n, sizeof(void*)); + list[0] = str; + for (p = str, n = 1; p < str + l - 1; ++p) + if (*p == 0) list[n++] = p + 1; + return list; +} + +int bcf_hdr_sync(bcf_hdr_t *b) +{ + if (b == 0) return -1; + if (b->ns) free(b->ns); + if (b->sns) free(b->sns); + if (b->l_nm) b->ns = cnt_null(b->l_nm, b->name, &b->n_ref); + else b->ns = 0, b->n_ref = 0; + b->sns = cnt_null(b->l_smpl, b->sname, &b->n_smpl); + return 0; +} + +int bcf_sync(bcf1_t *b) +{ + char *p, *tmp[5]; + int i, n, n_smpl = b->n_smpl; + ks_tokaux_t aux; + // set ref, alt, flt, info, fmt + b->ref = b->alt = b->flt = b->info = b->fmt = 0; + for (p = b->str, n = 0; p < b->str + b->l_str; ++p) + if (*p == 0 && p+1 != b->str + b->l_str) tmp[n++] = p + 1; + if (n != 5) { + fprintf(stderr, "[%s] incorrect number of fields (%d != 5). Corrupted file?\n", __func__, n); + return -1; + } + b->ref = tmp[0]; b->alt = tmp[1]; b->flt = tmp[2]; b->info = tmp[3]; b->fmt = tmp[4]; + // set n_alleles + if (*b->alt == 0) b->n_alleles = 1; + else { + for (p = b->alt, n = 1; *p; ++p) + if (*p == ',') ++n; + b->n_alleles = n + 1; + } + // set n_gi and gi[i].fmt + for (p = b->fmt, n = 1; *p; ++p) + if (*p == ':') ++n; + if (n > b->m_gi) { + int old_m = b->m_gi; + b->m_gi = n; + kroundup32(b->m_gi); + b->gi = realloc(b->gi, b->m_gi * sizeof(bcf_ginfo_t)); + memset(b->gi + old_m, 0, (b->m_gi - old_m) * sizeof(bcf_ginfo_t)); + } + b->n_gi = n; + for (p = kstrtok(b->fmt, ":", &aux), n = 0; p; p = kstrtok(0, 0, &aux)) + b->gi[n++].fmt = bcf_str2int(p, aux.p - p); + // set gi[i].len + for (i = 0; i < b->n_gi; ++i) { + if (b->gi[i].fmt == bcf_str2int("PL", 2)) { + b->gi[i].len = b->n_alleles * (b->n_alleles + 1) / 2; + } else if (b->gi[i].fmt == bcf_str2int("DP", 2) || b->gi[i].fmt == bcf_str2int("HQ", 2)) { + b->gi[i].len = 2; + } else if (b->gi[i].fmt == bcf_str2int("GQ", 2) || b->gi[i].fmt == bcf_str2int("GT", 2)) { + b->gi[i].len = 1; + } else if (b->gi[i].fmt == bcf_str2int("GL", 2)) { + b->gi[i].len = b->n_alleles * (b->n_alleles + 1) / 2 * 4; + } + b->gi[i].data = realloc(b->gi[i].data, n_smpl * b->gi[i].len); + } + return 0; +} + +int bcf_write(bcf_t *bp, const bcf_hdr_t *h, const bcf1_t *b) +{ + int i, l = 0; + if (b == 0) return -1; + bgzf_write(bp->fp, &b->tid, 4); + bgzf_write(bp->fp, &b->pos, 4); + bgzf_write(bp->fp, &b->qual, 4); + bgzf_write(bp->fp, &b->l_str, 4); + bgzf_write(bp->fp, b->str, b->l_str); + l = 12 + b->l_str; + for (i = 0; i < b->n_gi; ++i) { + bgzf_write(bp->fp, b->gi[i].data, b->gi[i].len * h->n_smpl); + l += b->gi[i].len * h->n_smpl; + } + return l; +} + +int bcf_read(bcf_t *bp, const bcf_hdr_t *h, bcf1_t *b) +{ + int i, l = 0; + if (b == 0) return -1; + if (bgzf_read(bp->fp, &b->tid, 4) == 0) return -1; + b->n_smpl = h->n_smpl; + bgzf_read(bp->fp, &b->pos, 4); + bgzf_read(bp->fp, &b->qual, 4); + bgzf_read(bp->fp, &b->l_str, 4); + if (b->l_str > b->m_str) { + b->m_str = b->l_str; + kroundup32(b->m_str); + b->str = realloc(b->str, b->m_str); + } + bgzf_read(bp->fp, b->str, b->l_str); + l = 12 + b->l_str; + bcf_sync(b); + for (i = 0; i < b->n_gi; ++i) { + bgzf_read(bp->fp, b->gi[i].data, b->gi[i].len * h->n_smpl); + l += b->gi[i].len * h->n_smpl; + } + return l; +} + +int bcf_destroy(bcf1_t *b) +{ + int i; + if (b == 0) return -1; + free(b->str); + for (i = 0; i < b->m_gi; ++i) + free(b->gi[i].data); + free(b->gi); + free(b); + return 0; +} + +static inline void fmt_str(const char *p, kstring_t *s) +{ + if (*p == 0) kputc('.', s); + else kputs(p, s); +} + +void bcf_fmt_core(const bcf_hdr_t *h, bcf1_t *b, kstring_t *s) +{ + int i, j, x; + s->l = 0; + if (h->n_ref) kputs(h->ns[b->tid], s); + else kputw(b->tid, s); + kputc('\t', s); + kputw(b->pos + 1, s); kputc('\t', s); + fmt_str(b->str, s); kputc('\t', s); + fmt_str(b->ref, s); kputc('\t', s); + fmt_str(b->alt, s); kputc('\t', s); + ksprintf(s, "%.3g", b->qual); kputc('\t', s); + fmt_str(b->flt, s); kputc('\t', s); + fmt_str(b->info, s); + if (b->fmt[0]) { + kputc('\t', s); + fmt_str(b->fmt, s); + } + x = b->n_alleles * (b->n_alleles + 1) / 2; + if (b->n_gi == 0) return; + for (j = 0; j < h->n_smpl; ++j) { + kputc('\t', s); + for (i = 0; i < b->n_gi; ++i) { + if (i) kputc(':', s); + if (b->gi[i].fmt == bcf_str2int("PL", 2)) { + uint8_t *d = (uint8_t*)b->gi[i].data + j * x; + int k; + for (k = 0; k < x; ++k) { + if (k > 0) kputc(',', s); + kputw(d[k], s); + } + } else if (b->gi[i].fmt == bcf_str2int("DP", 2)) { + kputw(((uint16_t*)b->gi[i].data)[j], s); + } else if (b->gi[i].fmt == bcf_str2int("GQ", 2)) { + kputw(((uint8_t*)b->gi[i].data)[j], s); + } else if (b->gi[i].fmt == bcf_str2int("GT", 2)) { + int y = ((uint8_t*)b->gi[i].data)[j]; + if (y>>7&1) { + kputsn("./.", 3, s); + } else { + kputc('0' + (y>>3&7), s); + kputc("/|"[y>>6&1], s); + kputc('0' + (y&7), s); + } + } else if (b->gi[i].fmt == bcf_str2int("GL", 2)) { + float *d = (float*)b->gi[i].data + j * x; + int k; + //printf("- %lx\n", d); + for (k = 0; k < x; ++k) { + if (k > 0) kputc(',', s); + ksprintf(s, "%.2f", d[k]); + } + } + } + } +} + +char *bcf_fmt(const bcf_hdr_t *h, bcf1_t *b) +{ + kstring_t s; + s.l = s.m = 0; s.s = 0; + bcf_fmt_core(h, b, &s); + return s.s; +} + +int bcf_append_info(bcf1_t *b, const char *info, int l) +{ + int shift = b->fmt - b->str; + int l_fmt = b->l_str - shift; + char *ori = b->str; + if (b->l_str + l > b->m_str) { // enlarge if necessary + b->m_str = b->l_str + l; + kroundup32(b->m_str); + b->str = realloc(b->str, b->m_str); + } + memmove(b->str + shift + l, b->str + shift, l_fmt); // move the FORMAT field + memcpy(b->str + shift - 1, info, l); // append to the INFO field + b->str[shift + l - 1] = '\0'; + b->fmt = b->str + shift + l; + b->l_str += l; + if (ori != b->str) bcf_sync(b); // synchronize when realloc changes the pointer + return 0; +} + +int bcf_cpy(bcf1_t *r, const bcf1_t *b) +{ + char *t1 = r->str; + bcf_ginfo_t *t2 = r->gi; + int i, t3 = r->m_str, t4 = r->m_gi; + *r = *b; + r->str = t1; r->gi = t2; r->m_str = t3; r->m_gi = t4; + if (r->m_str < b->m_str) { + r->m_str = b->m_str; + r->str = realloc(r->str, r->m_str); + } + memcpy(r->str, b->str, r->m_str); + bcf_sync(r); // calling bcf_sync() is simple but inefficient + for (i = 0; i < r->n_gi; ++i) + memcpy(r->gi[i].data, b->gi[i].data, r->n_smpl * r->gi[i].len); + return 0; +} diff --git a/bcftools/bcf.h b/bcftools/bcf.h new file mode 100644 index 0000000..3657895 --- /dev/null +++ b/bcftools/bcf.h @@ -0,0 +1,161 @@ +/* The MIT License + + Copyright (c) 2010 Broad Institute + + Permission is hereby granted, free of charge, to any person obtaining + a copy of this software and associated documentation files (the + "Software"), to deal in the Software without restriction, including + without limitation the rights to use, copy, modify, merge, publish, + distribute, sublicense, and/or sell copies of the Software, and to + permit persons to whom the Software is furnished to do so, subject to + the following conditions: + + The above copyright notice and this permission notice shall be + included in all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. +*/ + +/* Contact: Heng Li */ + +#ifndef BCF_H +#define BCF_H + +#include +#include +#include "bgzf.h" + +/* + A member in the structs below is said to "primary" if its content + cannot be inferred from other members in any of structs below; a + member is said to be "derived" if its content can be derived from + other members. For example, bcf1_t::str is primary as this comes from + the input data, while bcf1_t::info is derived as it can always be + correctly set if we know bcf1_t::str. Derived members are for quick + access to the content and must be synchronized with the primary data. + */ + +typedef struct { + uint32_t fmt; // format of the block, set by bcf_str2int(). + int len; // length of data for each individual + void *data; // concatenated data + // derived info: fmt, len (<-bcf1_t::fmt) +} bcf_ginfo_t; + +typedef struct { + int32_t tid, pos; // refID and 0-based position + int32_t l_str, m_str; // length and the allocated size of ->str + float qual; // SNP quality + char *str; // concatenated string of variable length strings in VCF (from col.2 to col.7) + char *ref, *alt, *flt, *info, *fmt; // they all point to ->str; no memory allocation + int n_gi, m_gi; // number and the allocated size of geno fields + bcf_ginfo_t *gi; // array of geno fields + int n_alleles, n_smpl; // number of alleles and samples + // derived info: ref, alt, flt, info, fmt (<-str), n_gi (<-fmt), n_alleles (<-alt), n_smpl (<-bcf_hdr_t::n_smpl) +} bcf1_t; + +typedef struct { + int32_t n_ref, n_smpl; // number of reference sequences and samples + int32_t l_nm; // length of concatenated sequence names; 0 padded + int32_t l_smpl; // length of concatenated sample names; 0 padded + int32_t l_txt; // length of header text (lines started with ##) + char *name, *sname, *txt; // concatenated sequence names, sample names and header text + char **ns, **sns; // array of sequence and sample names; point to name and sname, respectively + // derived info: n_ref (<-name), n_smpl (<-sname), ns (<-name), sns (<-sname) +} bcf_hdr_t; + +typedef struct { + int is_vcf; // if the file in operation is a VCF + void *v; // auxillary data structure for VCF + BGZF *fp; // file handler for BCF +} bcf_t; + +struct __bcf_idx_t; +typedef struct __bcf_idx_t bcf_idx_t; + +#ifdef __cplusplus +extern "C" { +#endif + + // open a BCF file; for BCF file only + bcf_t *bcf_open(const char *fn, const char *mode); + // close file + int bcf_close(bcf_t *b); + // read one record from BCF; return -1 on end-of-file, and <-1 for errors + int bcf_read(bcf_t *bp, const bcf_hdr_t *h, bcf1_t *b); + // call this function if b->str is changed + int bcf_sync(bcf1_t *b); + // write a BCF record + int bcf_write(bcf_t *bp, const bcf_hdr_t *h, const bcf1_t *b); + // read the BCF header; BCF only + bcf_hdr_t *bcf_hdr_read(bcf_t *b); + // write the BCF header + int bcf_hdr_write(bcf_t *b, const bcf_hdr_t *h); + // set bcf_hdr_t::ns and bcf_hdr_t::sns + int bcf_hdr_sync(bcf_hdr_t *b); + // destroy the header + void bcf_hdr_destroy(bcf_hdr_t *h); + // destroy a record + int bcf_destroy(bcf1_t *b); + // BCF->VCF conversion + char *bcf_fmt(const bcf_hdr_t *h, bcf1_t *b); + // append more info + int bcf_append_info(bcf1_t *b, const char *info, int l); + // copy + int bcf_cpy(bcf1_t *r, const bcf1_t *b); + + // open a VCF or BCF file if "b" is set in "mode" + bcf_t *vcf_open(const char *fn, const char *mode); + // close a VCF/BCF file + int vcf_close(bcf_t *bp); + // read the VCF/BCF header + bcf_hdr_t *vcf_hdr_read(bcf_t *bp); + // read a VCF/BCF record; return -1 on end-of-file and <-1 for errors + int vcf_read(bcf_t *bp, bcf_hdr_t *h, bcf1_t *b); + // write the VCF header + int vcf_hdr_write(bcf_t *bp, const bcf_hdr_t *h); + // write a VCF record + int vcf_write(bcf_t *bp, bcf_hdr_t *h, bcf1_t *b); + + // keep the first n alleles and discard the rest + int bcf_shrink_alt(bcf1_t *b, int n); + // convert GL to PL + int bcf_gl2pl(bcf1_t *b); + + // string hash table + void *bcf_build_refhash(bcf_hdr_t *h); + void bcf_str2id_destroy(void *_hash); + int bcf_str2id_add(void *_hash, const char *str); + int bcf_str2id(void *_hash, const char *str); + void *bcf_str2id_init(); + + // indexing related functions + int bcf_idx_build(const char *fn); + uint64_t bcf_idx_query(const bcf_idx_t *idx, int tid, int beg); + int bcf_parse_region(void *str2id, const char *str, int *tid, int *begin, int *end); + bcf_idx_t *bcf_idx_load(const char *fn); + void bcf_idx_destroy(bcf_idx_t *idx); + +#ifdef __cplusplus +} +#endif + +static inline uint32_t bcf_str2int(const char *str, int l) +{ + int i; + uint32_t x = 0; + for (i = 0; i < l && i < 4; ++i) { + if (str[i] == 0) return x; + x = x<<8 | str[i]; + } + return x; +} + +#endif diff --git a/bcftools/bcf.tex b/bcftools/bcf.tex new file mode 100644 index 0000000..5ca1e28 --- /dev/null +++ b/bcftools/bcf.tex @@ -0,0 +1,63 @@ +\documentclass[10pt,pdftex]{article} +\usepackage{color} +\definecolor{gray}{rgb}{0.7,0.7,0.7} + +\setlength{\topmargin}{0.0cm} +\setlength{\textheight}{21.5cm} +\setlength{\oddsidemargin}{0cm} +\setlength{\textwidth}{16.5cm} +\setlength{\columnsep}{0.6cm} + +\begin{document} + +\begin{center} +\begin{tabular}{|l|l|l|l|l|} +\hline +\multicolumn{2}{|c|}{\bf Field} & \multicolumn{1}{c|}{\bf Descrption} & \multicolumn{1}{c|}{\bf Type} & \multicolumn{1}{c|}{\bf Value} \\\hline\hline +\multicolumn{2}{|l|}{\tt magic} & Magic string & {\tt char[4]} & {\tt BCF\char92 4} \\\hline +\multicolumn{2}{|l|}{\tt l\_nm} & Length of concatenated sequence names & {\tt int32\_t} & \\\hline +\multicolumn{2}{|l|}{\tt name} & Concatenated names, {\tt NULL} padded & {\tt char[l\_nm]} & \\\hline +\multicolumn{2}{|l|}{\tt l\_smpl} & Length of concatenated sample names & {\tt int32\_t} & \\\hline +\multicolumn{2}{|l|}{\tt sname} & Concatenated sample names & {\tt char[l\_smpl]} & \\\hline +\multicolumn{2}{|l|}{\tt l\_txt} & Length of the meta text (double-hash lines)& {\tt int32\_t} & \\\hline +\multicolumn{2}{|l|}{\tt text} & Meta text, {\tt NULL} terminated & {\tt char[l\_txt]} & \\\hline +\multicolumn{5}{|c|}{\it \color{gray}{List of records until the end of the file}}\\\cline{2-5} +& {\tt seq\_id} & Reference sequence ID & {\tt int32\_t} & \\\cline{2-5} +& {\tt pos} & Position & {\tt int32\_t} & \\\cline{2-5} +& {\tt qual} & Variant quality & {\tt float} & \\\cline{2-5} +& {\tt l\_str} & Length of str & {\tt int32\_t} & \\\cline{2-5} +& {\tt str} & {\tt ID+REF+ALT+FILTER+INFO+FORMAT}, {\tt NULL} padded & {\tt char[slen]} &\\\cline{2-5} +& \multicolumn{4}{c|}{Blocks of data; \#blocks and formats defined by {\tt FORMAT} (table below)}\\ +\hline +\end{tabular} +\end{center} + +\begin{center} +\begin{tabular}{cll} +\hline +\multicolumn{1}{l}{\bf Field} & \multicolumn{1}{l}{\bf Type} & \multicolumn{1}{l}{\bf Description} \\\hline +{\tt DP} & {\tt uint16\_t[n]} & Read depth \\ +{\tt GL} & {\tt float[n*x]} & Log10 likelihood of data; $x=\frac{m(m+1)}{2}$, $m=\#\{alleles\}$\\ +{\tt GT} & {\tt uint8\_t[n]} & {\tt phase\char60\char60 6 | allele1\char60\char60 3 | allele2} \\ +{\tt GQ} & {\tt uint8\_t[n]} & {Genotype quality}\\ +{\tt HQ} & {\tt uint8\_t[n*2]} & {Haplotype quality}\\ +{\tt PL} & {\tt uint8\_t[n*x]} & {Phred-scaled likelihood of data}\\ +\emph{misc} & {\tt int32\_t+char*} & {\tt NULL} padded concatenated strings (integer equal to the length) \\ +\hline +\end{tabular} +\end{center} + +\begin{itemize} +\item The file is {\tt BGZF} compressed. +\item All integers are little-endian. +\item In a string, a missing value `.' is an empty C string ``{\tt + \char92 0}'' (not ``{\tt .\char92 0}'') +\item For {\tt GL} and {\tt PL}, likelihoods of genotypes appear in the + order of alleles in {\tt REF} and then {\tt ALT}. For example, if {\tt + REF=C}, {\tt ALT=T,A}, likelihoods appear in the order of {\tt + CC,CT,CA,TT,TA,AA}. +\item {\tt GL} is an extension to and is backward compatible with the + {\tt GL} genotype field in {\tt VCFv4.0}. +\end{itemize} + +\end{document} \ No newline at end of file diff --git a/bcftools/bcf2qcall.c b/bcftools/bcf2qcall.c new file mode 100644 index 0000000..8634c9e --- /dev/null +++ b/bcftools/bcf2qcall.c @@ -0,0 +1,91 @@ +#include +#include +#include +#include +#include "bcf.h" + +static int8_t nt4_table[256] = { + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4 /*'-'*/, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 0, 4, 1, 4, 4, 4, 2, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 3, 4, 4, 4, -1, 4, 4, 4, 4, 4, 4, 4, + 4, 0, 4, 1, 4, 4, 4, 2, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 3, 4, 4, 4, -1, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4 +}; + +static int read_I16(bcf1_t *b, int anno[16]) +{ + char *p; + int i; + if ((p = strstr(b->info, "I16=")) == 0) return -1; + p += 4; + for (i = 0; i < 16; ++i) { + anno[i] = strtol(p, &p, 10); + if (anno[i] == 0 && (errno == EINVAL || errno == ERANGE)) return -2; + ++p; + } + return 0; +} + +int bcf_2qcall(bcf_hdr_t *h, bcf1_t *b) +{ + int a[4], k, g[10], l, map[4], k1, j, i, i0, anno[16], dp, mq, d_rest; + char *s; + if (b->ref[1] != 0 || b->n_alleles > 4) return -1; // ref is not a single base + for (i = 0; i < b->n_gi; ++i) + if (b->gi[i].fmt == bcf_str2int("PL", 2)) break; + if (i == b->n_gi) return -1; // no PL + if (read_I16(b, anno) != 0) return -1; // no I16; FIXME: can be improved + d_rest = dp = anno[0] + anno[1] + anno[2] + anno[3]; + if (dp == 0) return -1; // depth is zero + mq = (int)(sqrt((double)(anno[9] + anno[11]) / dp) + .499); + i0 = i; + a[0] = nt4_table[(int)b->ref[0]]; + if (a[0] > 3) return -1; // ref is not A/C/G/T + a[1] = a[2] = a[3] = -2; // -1 has a special meaning + if (b->alt[0] == 0) return -1; // no alternate allele + map[0] = map[1] = map[2] = map[3] = -2; + map[a[0]] = 0; + for (k = 0, s = b->alt, k1 = -1; k < 3 && *s; ++k, s += 2) { + if (s[1] != ',' && s[1] != 0) return -1; // ALT is not single base + a[k+1] = nt4_table[(int)*s]; + if (a[k+1] >= 0) map[a[k+1]] = k+1; + else k1 = k+1; + if (s[1] == 0) break; + } + for (k = 0; k < 4; ++k) + if (map[k] < 0) map[k] = k1; + for (i = 0; i < h->n_smpl; ++i) { + int d; + uint8_t *p = b->gi[i0].data + i * b->gi[i0].len; + for (j = 0; j < b->gi[i0].len; ++j) + if (p[j]) break; + d = (int)((double)d_rest / (h->n_smpl - i) + .499); + if (d == 0) d = 1; + if (j == b->gi[i0].len) d = 0; + d_rest -= d; + for (k = j = 0; k < 4; ++k) { + for (l = k; l < 4; ++l) { + int t, x = map[k], y = map[l]; + if (x > y) t = x, x = y, y = t; + g[j++] = p[x * b->n_alleles - x * (x-1) / 2 + (y - x)]; + } + } + printf("%s\t%d\t%c", h->ns[b->tid], b->pos+1, *b->ref); + printf("\t%d\t%d\t0", d, mq); + for (j = 0; j < 10; ++j) + printf("\t%d", g[j]); + printf("\t%s\n", h->sns[i]); + } + return 0; +} diff --git a/bcftools/bcftools.1 b/bcftools/bcftools.1 new file mode 100644 index 0000000..ebff301 --- /dev/null +++ b/bcftools/bcftools.1 @@ -0,0 +1,115 @@ +.TH bcftools 1 "2 October 2010" "bcftools" "Bioinformatics tools" +.SH NAME +.PP +bcftools - Utilities for the Binary Call Format (BCF) and VCF. +.SH SYNOPSIS +.PP +bcftools index in.bcf +.PP +bcftools view in.bcf chr2:100-200 > out.vcf +.PP +bcftools view -vc in.bcf > out.vcf 2> out.afs + +.SH DESCRIPTION +.PP +Bcftools is a toolkit for processing VCF/BCF files, calling variants and +estimating site allele frequencies and allele frequency spectrums. + +.SH COMMANDS AND OPTIONS + +.TP 10 +.B view +.B bcftools view +.RB [ \-cbuSAGgHvNQ ] +.RB [ \-1 +.IR nGroup1 ] +.RB [ \-l +.IR listFile ] +.RB [ \-t +.IR mutRate ] +.RB [ \-p +.IR varThres ] +.RB [ \-P +.IR prior ] +.I in.bcf +.RI [ region ] + +Convert between BCF and VCF, call variant candidates and estimate allele +frequencies. + +.B OPTIONS: +.RS +.TP 10 +.B -b +Output in the BCF format. The default is VCF. +.TP +.B -c +Call variants. +.TP +.B -v +Output variant sites only (force -c) +.TP +.B -g +Call per-sample genotypes at variant sites (force -c) +.TP +.B -u +Uncompressed BCF output (force -b). +.TP +.B -S +The input is VCF instead of BCF. +.TP +.B -A +Retain all possible alternate alleles at variant sites. By default, this +command discards unlikely alleles. +.TP +.B -G +Suppress all individual genotype information. +.TP +.B -H +Perform Hardy-Weiberg Equilibrium test. This will add computation time, sometimes considerably. +.TP +.B -N +Skip sites where the REF field is not A/C/G/T +.TP +.B -Q +Output the QCALL likelihood format +.TP +.BI "-1 " INT +Number of group-1 samples. This option is used for dividing input into +two groups for comparing. A zero value disables this functionality. [0] +.TP +.BI "-l " FILE +List of sites at which information are outputted [all sites] +.TP +.BI "-t " FLOAT +Scaled muttion rate for variant calling [0.001] +.TP +.BI "-p " FLOAT +A site is considered to be a variant if P(ref|D)n_ref; ++i) { + khint_t k; + k = kh_put(str2id, hash, h->ns[i], &ret); // FIXME: check ret + kh_val(hash, k) = i; + } + return hash; +} + +void *bcf_str2id_init() +{ + return kh_init(str2id); +} + +void bcf_str2id_destroy(void *_hash) +{ + khash_t(str2id) *hash = (khash_t(str2id)*)_hash; + if (hash) kh_destroy(str2id, hash); // Note that strings are not freed. +} + +int bcf_str2id(void *_hash, const char *str) +{ + khash_t(str2id) *hash = (khash_t(str2id)*)_hash; + khint_t k; + if (!hash) return -1; + k = kh_get(str2id, hash, str); + return k == kh_end(hash)? -1 : kh_val(hash, k); +} + +int bcf_str2id_add(void *_hash, const char *str) +{ + khint_t k; + int ret; + khash_t(str2id) *hash = (khash_t(str2id)*)_hash; + if (!hash) return -1; + k = kh_put(str2id, hash, str, &ret); + if (ret == 0) return kh_val(hash, k); + kh_val(hash, k) = kh_size(hash) - 1; + return kh_val(hash, k); +} + +int bcf_shrink_alt(bcf1_t *b, int n) +{ + char *p; + int i, j, k, *z, n_smpl = b->n_smpl; + if (b->n_alleles <= n) return -1; + if (n > 1) { + for (p = b->alt, k = 1; *p; ++p) + if (*p == ',' && ++k == n) break; + *p = '\0'; + } else p = b->alt, *p = '\0'; + ++p; + memmove(p, b->flt, b->str + b->l_str - b->flt); + b->l_str -= b->flt - p; + z = alloca(sizeof(int) / 2 * n * (n+1)); + for (i = k = 0; i < n; ++i) + for (j = 0; j < n - i; ++j) + z[k++] = i * b->n_alleles + j; + for (i = 0; i < b->n_gi; ++i) { + bcf_ginfo_t *g = b->gi + i; + if (g->fmt == bcf_str2int("PL", 2)) { + int l, x = b->n_alleles * (b->n_alleles + 1) / 2; + uint8_t *d = (uint8_t*)g->data; + g->len = n * (n + 1) / 2; + for (l = k = 0; l < n_smpl; ++l) { + uint8_t *dl = d + l * x; + for (j = 0; j < g->len; ++j) d[k++] = dl[z[j]]; + } + } // FIXME: to add GL + } + b->n_alleles = n; + bcf_sync(b); + return 0; +} + +int bcf_gl2pl(bcf1_t *b) +{ + char *p; + int i, n_smpl = b->n_smpl; + bcf_ginfo_t *g; + float *d0; + uint8_t *d1; + if (strstr(b->fmt, "PL")) return -1; + if ((p = strstr(b->fmt, "GL")) == 0) return -1; + *p = 'P'; + for (i = 0; i < b->n_gi; ++i) + if (b->gi[i].fmt == bcf_str2int("GL", 2)) + break; + g = b->gi + i; + g->fmt = bcf_str2int("PL", 2); + g->len /= 4; // 4 == sizeof(float) + d0 = (float*)g->data; d1 = (uint8_t*)g->data; + for (i = 0; i < n_smpl * g->len; ++i) { + int x = (int)(-10. * d0[i] + .499); + if (x > 255) x = 255; + if (x < 0) x = 0; + d1[i] = x; + } + return 0; +} diff --git a/bcftools/call1.c b/bcftools/call1.c new file mode 100644 index 0000000..2b28452 --- /dev/null +++ b/bcftools/call1.c @@ -0,0 +1,372 @@ +#include +#include +#include +#include +#include +#include "bcf.h" +#include "prob1.h" +#include "kstring.h" + +#include "khash.h" +KHASH_SET_INIT_INT64(set64) + +#include "kseq.h" +KSTREAM_INIT(gzFile, gzread, 16384) + +#define VC_NO_GENO 2 +#define VC_BCFOUT 4 +#define VC_CALL 8 +#define VC_VARONLY 16 +#define VC_VCFIN 32 +#define VC_UNCOMP 64 +#define VC_HWE 128 +#define VC_KEEPALT 256 +#define VC_ACGT_ONLY 512 +#define VC_QCALL 1024 +#define VC_CALL_GT 2048 +#define VC_ADJLD 4096 + +typedef struct { + int flag, prior_type, n1; + char *fn_list, *prior_file; + double theta, pref; +} viewconf_t; + +khash_t(set64) *bcf_load_pos(const char *fn, bcf_hdr_t *_h) +{ + void *str2id; + gzFile fp; + kstream_t *ks; + int ret, dret, lineno = 1; + kstring_t *str; + khash_t(set64) *hash = 0; + + hash = kh_init(set64); + str2id = bcf_build_refhash(_h); + str = calloc(1, sizeof(kstring_t)); + fp = strcmp(fn, "-")? gzopen(fn, "r") : gzdopen(fileno(stdin), "r"); + ks = ks_init(fp); + while (ks_getuntil(ks, 0, str, &dret) >= 0) { + int tid = bcf_str2id(str2id, str->s); + if (tid >= 0 && dret != '\n') { + if (ks_getuntil(ks, 0, str, &dret) >= 0) { + uint64_t x = (uint64_t)tid<<32 | (atoi(str->s) - 1); + kh_put(set64, hash, x, &ret); + } else break; + } else fprintf(stderr, "[%s] %s is not a reference name (line %d).\n", __func__, str->s, lineno); + if (dret != '\n') while ((dret = ks_getc(ks)) > 0 && dret != '\n'); + if (dret < 0) break; + ++lineno; + } + bcf_str2id_destroy(str2id); + ks_destroy(ks); + gzclose(fp); + free(str->s); free(str); + return hash; +} + +static double test_hwe(const double g[3]) +{ + extern double kf_gammaq(double p, double x); + double fexp, chi2, f[3], n; + int i; + n = g[0] + g[1] + g[2]; + fexp = (2. * g[2] + g[1]) / (2. * n); + if (fexp > 1. - 1e-10) fexp = 1. - 1e-10; + if (fexp < 1e-10) fexp = 1e-10; + f[0] = n * (1. - fexp) * (1. - fexp); + f[1] = n * 2. * fexp * (1. - fexp); + f[2] = n * fexp * fexp; + for (i = 0, chi2 = 0.; i < 3; ++i) + chi2 += (g[i] - f[i]) * (g[i] - f[i]) / f[i]; + return kf_gammaq(.5, chi2 / 2.); +} + +typedef struct { + double p[4]; + int mq, depth, is_tested, d[4]; +} anno16_t; + +static double ttest(int n1, int n2, int a[4]) +{ + extern double kf_betai(double a, double b, double x); + double t, v, u1, u2; + if (n1 == 0 || n2 == 0 || n1 + n2 < 3) return 1.0; + u1 = (double)a[0] / n1; u2 = (double)a[2] / n2; + if (u1 <= u2) return 1.; + t = (u1 - u2) / sqrt(((a[1] - n1 * u1 * u1) + (a[3] - n2 * u2 * u2)) / (n1 + n2 - 2) * (1./n1 + 1./n2)); + v = n1 + n2 - 2; +// printf("%d,%d,%d,%d,%lf,%lf,%lf\n", a[0], a[1], a[2], a[3], t, u1, u2); + return t < 0.? 1. : .5 * kf_betai(.5*v, .5, v/(v+t*t)); +} + +static int test16_core(int anno[16], anno16_t *a) +{ + extern double kt_fisher_exact(int n11, int n12, int n21, int n22, double *_left, double *_right, double *two); + double left, right; + int i; + a->p[0] = a->p[1] = a->p[2] = a->p[3] = 1.; + memcpy(a->d, anno, 4 * sizeof(int)); + a->depth = anno[0] + anno[1] + anno[2] + anno[3]; + a->is_tested = (anno[0] + anno[1] > 0 && anno[2] + anno[3] > 0); + if (a->depth == 0) return -1; + a->mq = (int)(sqrt((anno[9] + anno[11]) / a->depth) + .499); + kt_fisher_exact(anno[0], anno[1], anno[2], anno[3], &left, &right, &a->p[0]); + for (i = 1; i < 4; ++i) + a->p[i] = ttest(anno[0] + anno[1], anno[2] + anno[3], anno+4*i); + return 0; +} + +static int test16(bcf1_t *b, anno16_t *a) +{ + char *p; + int i, anno[16]; + a->p[0] = a->p[1] = a->p[2] = a->p[3] = 1.; + a->d[0] = a->d[1] = a->d[2] = a->d[3] = 0.; + a->mq = a->depth = a->is_tested = 0; + if ((p = strstr(b->info, "I16=")) == 0) return -1; + p += 4; + for (i = 0; i < 16; ++i) { + anno[i] = strtol(p, &p, 10); + if (anno[i] == 0 && (errno == EINVAL || errno == ERANGE)) return -2; + ++p; + } + return test16_core(anno, a); +} + +static void rm_info(bcf1_t *b, const char *key) +{ + char *p, *q; + if ((p = strstr(b->info, key)) == 0) return; + for (q = p; *q && *q != ';'; ++q); + if (p > b->info && *(p-1) == ';') --p; + memmove(p, q, b->l_str - (q - b->str)); + b->l_str -= q - p; + bcf_sync(b); +} + +static int update_bcf1(int n_smpl, bcf1_t *b, const bcf_p1aux_t *pa, const bcf_p1rst_t *pr, double pref, int flag) +{ + kstring_t s; + int is_var = (pr->p_ref < pref); + double p_hwe, r = is_var? pr->p_ref : 1. - pr->p_ref; + anno16_t a; + + p_hwe = pr->g[0] >= 0.? test_hwe(pr->g) : 1.0; // only do HWE g[] is calculated + test16(b, &a); + rm_info(b, "I16="); + + memset(&s, 0, sizeof(kstring_t)); + kputc('\0', &s); kputs(b->ref, &s); kputc('\0', &s); + kputs(b->alt, &s); kputc('\0', &s); kputc('\0', &s); + kputs(b->info, &s); + if (b->info[0]) kputc(';', &s); + ksprintf(&s, "AF1=%.3lf;AFE=%.3lf", 1.-pr->f_em, 1.-pr->f_exp); + ksprintf(&s, ";DP4=%d,%d,%d,%d;MQ=%d", a.d[0], a.d[1], a.d[2], a.d[3], a.mq); + if (a.is_tested) { + if (pr->pc[0] >= 0.) ksprintf(&s, ";PC4=%lg,%lg,%lg,%lg", pr->pc[0], pr->pc[1], pr->pc[2], pr->pc[3]); + ksprintf(&s, ";PV4=%.2lg,%.2lg,%.2lg,%.2lg", a.p[0], a.p[1], a.p[2], a.p[3]); + } + if (pr->g[0] >= 0. && p_hwe <= .2) + ksprintf(&s, ";GC=%.2lf,%.2lf,%.2lf;HWE=%.3lf", pr->g[2], pr->g[1], pr->g[0], p_hwe); + kputc('\0', &s); + kputs(b->fmt, &s); kputc('\0', &s); + free(b->str); + b->m_str = s.m; b->l_str = s.l; b->str = s.s; + b->qual = r < 1e-100? 99 : -4.343 * log(r); + if (b->qual > 99) b->qual = 99; + bcf_sync(b); + if (!is_var) bcf_shrink_alt(b, 1); + else if (!(flag&VC_KEEPALT)) + bcf_shrink_alt(b, pr->rank0 < 2? 2 : pr->rank0+1); + if (is_var && (flag&VC_CALL_GT)) { // call individual genotype + int i, x, old_n_gi = b->n_gi; + s.m = b->m_str; s.l = b->l_str - 1; s.s = b->str; + kputs(":GT:GQ", &s); kputc('\0', &s); + b->m_str = s.m; b->l_str = s.l; b->str = s.s; + bcf_sync(b); + for (i = 0; i < b->n_smpl; ++i) { + x = bcf_p1_call_gt(pa, pr->f_em, i); + ((uint8_t*)b->gi[old_n_gi].data)[i] = (x&3) == 0? 1<<3|1 : (x&3) == 1? 1 : 0; + ((uint8_t*)b->gi[old_n_gi+1].data)[i] = x>>2; + } + } + return is_var; +} + +double bcf_ld_freq(const bcf1_t *b0, const bcf1_t *b1, double f[4]); + +int bcfview(int argc, char *argv[]) +{ + extern int bcf_2qcall(bcf_hdr_t *h, bcf1_t *b); + bcf_t *bp, *bout = 0; + bcf1_t *b, *blast; + int c; + uint64_t n_processed = 0; + viewconf_t vc; + bcf_p1aux_t *p1 = 0; + bcf_hdr_t *h; + int tid, begin, end; + char moder[4], modew[4]; + khash_t(set64) *hash = 0; + + tid = begin = end = -1; + memset(&vc, 0, sizeof(viewconf_t)); + vc.prior_type = vc.n1 = -1; vc.theta = 1e-3; vc.pref = 0.5; + while ((c = getopt(argc, argv, "N1:l:cHAGvbSuP:t:p:QgL")) >= 0) { + switch (c) { + case '1': vc.n1 = atoi(optarg); break; + case 'l': vc.fn_list = strdup(optarg); break; + case 'N': vc.flag |= VC_ACGT_ONLY; break; + case 'G': vc.flag |= VC_NO_GENO; break; + case 'A': vc.flag |= VC_KEEPALT; break; + case 'b': vc.flag |= VC_BCFOUT; break; + case 'S': vc.flag |= VC_VCFIN; break; + case 'c': vc.flag |= VC_CALL; break; + case 'v': vc.flag |= VC_VARONLY | VC_CALL; break; + case 'u': vc.flag |= VC_UNCOMP | VC_BCFOUT; break; + case 'H': vc.flag |= VC_HWE; break; + case 'g': vc.flag |= VC_CALL_GT | VC_CALL; break; + case 't': vc.theta = atof(optarg); break; + case 'p': vc.pref = atof(optarg); break; + case 'Q': vc.flag |= VC_QCALL; break; + case 'L': vc.flag |= VC_ADJLD; break; + case 'P': + if (strcmp(optarg, "full") == 0) vc.prior_type = MC_PTYPE_FULL; + else if (strcmp(optarg, "cond2") == 0) vc.prior_type = MC_PTYPE_COND2; + else if (strcmp(optarg, "flat") == 0) vc.prior_type = MC_PTYPE_FLAT; + else vc.prior_file = strdup(optarg); + break; + } + } + if (argc == optind) { + fprintf(stderr, "\n"); + fprintf(stderr, "Usage: bcftools view [options] [reg]\n\n"); + fprintf(stderr, "Options: -c SNP calling\n"); + fprintf(stderr, " -v output potential variant sites only (force -c)\n"); + fprintf(stderr, " -g call genotypes at variant sites (force -c)\n"); + fprintf(stderr, " -b output BCF instead of VCF\n"); + fprintf(stderr, " -u uncompressed BCF output (force -b)\n"); + fprintf(stderr, " -S input is VCF\n"); + fprintf(stderr, " -A keep all possible alternate alleles at variant sites\n"); + fprintf(stderr, " -G suppress all individual genotype information\n"); + fprintf(stderr, " -H perform Hardy-Weinberg test (slower)\n"); + fprintf(stderr, " -N skip sites where REF is not A/C/G/T\n"); + fprintf(stderr, " -Q output the QCALL likelihood format\n"); + fprintf(stderr, " -L calculate LD for adjacent sites\n"); + fprintf(stderr, " -1 INT number of group-1 samples [0]\n"); + fprintf(stderr, " -l FILE list of sites to output [all sites]\n"); + fprintf(stderr, " -t FLOAT scaled mutation rate [%.4lg]\n", vc.theta); + fprintf(stderr, " -p FLOAT variant if P(ref|D)n_smpl); + if (vc.prior_file) { + if (bcf_p1_read_prior(p1, vc.prior_file) < 0) { + fprintf(stderr, "[%s] fail to read the prior AFS.\n", __func__); + return 1; + } + } else bcf_p1_init_prior(p1, vc.prior_type, vc.theta); + if (vc.n1 > 0) { + bcf_p1_set_n1(p1, vc.n1); + bcf_p1_init_subprior(p1, vc.prior_type, vc.theta); + } + } + if (vc.fn_list) hash = bcf_load_pos(vc.fn_list, h); + if (optind + 1 < argc && !(vc.flag&VC_VCFIN)) { + void *str2id = bcf_build_refhash(h); + if (bcf_parse_region(str2id, argv[optind+1], &tid, &begin, &end) >= 0) { + bcf_idx_t *idx; + idx = bcf_idx_load(argv[optind]); + if (idx) { + uint64_t off; + off = bcf_idx_query(idx, tid, begin); + if (off == 0) { + fprintf(stderr, "[%s] no records in the query region.\n", __func__); + return 1; // FIXME: a lot of memory leaks... + } + bgzf_seek(bp->fp, off, SEEK_SET); + bcf_idx_destroy(idx); + } + } + } + while (vcf_read(bp, h, b) > 0) { + if (vc.flag & VC_ACGT_ONLY) { + int x; + if (b->ref[0] == 0 || b->ref[1] != 0) continue; + x = toupper(b->ref[0]); + if (x != 'A' && x != 'C' && x != 'G' && x != 'T') continue; + } + if (hash) { + uint64_t x = (uint64_t)b->tid<<32 | b->pos; + khint_t k = kh_get(set64, hash, x); + if (kh_size(hash) == 0) break; + if (k == kh_end(hash)) continue; + kh_del(set64, hash, k); + } + if (tid >= 0) { + int l = strlen(b->ref); + l = b->pos + (l > 0? l : 1); + if (b->tid != tid || b->pos >= end) break; + if (!(l > begin && end > b->pos)) continue; + } + ++n_processed; + if (vc.flag & VC_QCALL) { // output QCALL format; STOP here + bcf_2qcall(h, b); + continue; + } + if (vc.flag & (VC_CALL|VC_ADJLD)) bcf_gl2pl(b); + if (vc.flag & VC_CALL) { // call variants + bcf_p1rst_t pr; + bcf_p1_cal(b, p1, &pr); // pr.g[3] is not calculated here + if (vc.flag&VC_HWE) bcf_p1_cal_g3(p1, pr.g); + if (n_processed % 100000 == 0) { + fprintf(stderr, "[%s] %ld sites processed.\n", __func__, (long)n_processed); + bcf_p1_dump_afs(p1); + } + if (pr.p_ref >= vc.pref && (vc.flag & VC_VARONLY)) continue; + update_bcf1(h->n_smpl, b, p1, &pr, vc.pref, vc.flag); + } + if (vc.flag & VC_ADJLD) { // compute LD + double f[4], r2; + if ((r2 = bcf_ld_freq(blast, b, f)) >= 0) { + kstring_t s; + s.m = s.l = 0; s.s = 0; + if (*b->info) kputc(';', &s); + ksprintf(&s, "NEIR=%.3lf;NEIF=%.3lf,%.3lf", r2, f[0]+f[2], f[0]+f[1]); + bcf_append_info(b, s.s, s.l); + free(s.s); + } + bcf_cpy(blast, b); + } + if (vc.flag & VC_NO_GENO) { // do not output GENO fields + b->n_gi = 0; + b->fmt[0] = '\0'; + } + vcf_write(bout, h, b); + } + if (vc.prior_file) free(vc.prior_file); + if (vc.flag & VC_CALL) bcf_p1_dump_afs(p1); + bcf_hdr_destroy(h); + bcf_destroy(b); bcf_destroy(blast); + vcf_close(bp); vcf_close(bout); + if (hash) kh_destroy(set64, hash); + if (vc.fn_list) free(vc.fn_list); + if (p1) bcf_p1_destroy(p1); + return 0; +} diff --git a/bcftools/fet.c b/bcftools/fet.c new file mode 100644 index 0000000..845f8c2 --- /dev/null +++ b/bcftools/fet.c @@ -0,0 +1,110 @@ +#include +#include + +/* This program is implemented with ideas from this web page: + * + * http://www.langsrud.com/fisher.htm + */ + +// log\binom{n}{k} +static double lbinom(int n, int k) +{ + if (k == 0 || n == k) return 0; + return lgamma(n+1) - lgamma(k+1) - lgamma(n-k+1); +} + +// n11 n12 | n1_ +// n21 n22 | n2_ +//-----------+---- +// n_1 n_2 | n + +// hypergeometric distribution +static double hypergeo(int n11, int n1_, int n_1, int n) +{ + return exp(lbinom(n1_, n11) + lbinom(n-n1_, n_1-n11) - lbinom(n, n_1)); +} + +typedef struct { + int n11, n1_, n_1, n; + double p; +} hgacc_t; + +// incremental version of hypergenometric distribution +static double hypergeo_acc(int n11, int n1_, int n_1, int n, hgacc_t *aux) +{ + if (n1_ || n_1 || n) { + aux->n11 = n11; aux->n1_ = n1_; aux->n_1 = n_1; aux->n = n; + } else { // then only n11 changed; the rest fixed + if (n11%11 && n11 + aux->n - aux->n1_ - aux->n_1) { + if (n11 == aux->n11 + 1) { // incremental + aux->p *= (double)(aux->n1_ - aux->n11) / n11 + * (aux->n_1 - aux->n11) / (n11 + aux->n - aux->n1_ - aux->n_1); + aux->n11 = n11; + return aux->p; + } + if (n11 == aux->n11 - 1) { // incremental + aux->p *= (double)aux->n11 / (aux->n1_ - n11) + * (aux->n11 + aux->n - aux->n1_ - aux->n_1) / (aux->n_1 - n11); + aux->n11 = n11; + return aux->p; + } + } + aux->n11 = n11; + } + aux->p = hypergeo(aux->n11, aux->n1_, aux->n_1, aux->n); + return aux->p; +} + +double kt_fisher_exact(int n11, int n12, int n21, int n22, double *_left, double *_right, double *two) +{ + int i, j, max, min; + double p, q, left, right; + hgacc_t aux; + int n1_, n_1, n; + + n1_ = n11 + n12; n_1 = n11 + n21; n = n11 + n12 + n21 + n22; // calculate n1_, n_1 and n + max = (n_1 < n1_) ? n_1 : n1_; // max n11, for right tail + min = (n1_ + n_1 - n < 0) ? 0 : (n1_ + n_1 - n < 0); // min n11, for left tail + *two = *_left = *_right = 1.; + if (min == max) return 1.; // no need to do test + q = hypergeo_acc(n11, n1_, n_1, n, &aux); // the probability of the current table + // left tail + p = hypergeo_acc(min, 0, 0, 0, &aux); + for (left = 0., i = min + 1; p < 0.99999999 * q; ++i) // loop until underflow + left += p, p = hypergeo_acc(i, 0, 0, 0, &aux); + --i; + if (p < 1.00000001 * q) left += p; + else --i; + // right tail + p = hypergeo_acc(max, 0, 0, 0, &aux); + for (right = 0., j = max - 1; p < 0.99999999 * q; --j) // loop until underflow + right += p, p = hypergeo_acc(j, 0, 0, 0, &aux); + if (p < 1.00000001 * q) right += p; + else ++j; + // two-tail + *two = left + right; + if (*two > 1.) *two = 1.; + // adjust left and right + if (abs(i - n11) < abs(j - n11)) right = 1. - left + q; + else left = 1.0 - right + q; + *_left = left; *_right = right; + return q; +} + +#ifdef FET_MAIN +#include + +int main(int argc, char *argv[]) +{ + char id[1024]; + int n11, n12, n21, n22; + double left, right, twotail, prob; + + while (scanf("%s%d%d%d%d", id, &n11, &n12, &n21, &n22) == 5) { + prob = kt_fisher_exact(n11, n12, n21, n22, &left, &right, &twotail); + printf("%s\t%d\t%d\t%d\t%d\t%.6g\t%.6g\t%.6g\t%.6g\n", id, n11, n12, n21, n22, + prob, left, right, twotail); + } + return 0; +} +#endif diff --git a/bcftools/index.c b/bcftools/index.c new file mode 100644 index 0000000..014856d --- /dev/null +++ b/bcftools/index.c @@ -0,0 +1,335 @@ +#include +#include +#include +#include "bam_endian.h" +#include "kstring.h" +#include "bcf.h" +#ifdef _USE_KNETFILE +#include "knetfile.h" +#endif + +#define TAD_LIDX_SHIFT 13 + +typedef struct { + int32_t n, m; + uint64_t *offset; +} bcf_lidx_t; + +struct __bcf_idx_t { + int32_t n; + bcf_lidx_t *index2; +}; + +/************ + * indexing * + ************/ + +static inline void insert_offset2(bcf_lidx_t *index2, int _beg, int _end, uint64_t offset) +{ + int i, beg, end; + beg = _beg >> TAD_LIDX_SHIFT; + end = (_end - 1) >> TAD_LIDX_SHIFT; + if (index2->m < end + 1) { + int old_m = index2->m; + index2->m = end + 1; + kroundup32(index2->m); + index2->offset = (uint64_t*)realloc(index2->offset, index2->m * 8); + memset(index2->offset + old_m, 0, 8 * (index2->m - old_m)); + } + if (beg == end) { + if (index2->offset[beg] == 0) index2->offset[beg] = offset; + } else { + for (i = beg; i <= end; ++i) + if (index2->offset[i] == 0) index2->offset[i] = offset; + } + if (index2->n < end + 1) index2->n = end + 1; +} + +bcf_idx_t *bcf_idx_core(bcf_t *bp, bcf_hdr_t *h) +{ + bcf_idx_t *idx; + int32_t last_coor, last_tid; + uint64_t last_off; + kstring_t *str; + BGZF *fp = bp->fp; + bcf1_t *b; + int ret; + + b = calloc(1, sizeof(bcf1_t)); + str = calloc(1, sizeof(kstring_t)); + idx = (bcf_idx_t*)calloc(1, sizeof(bcf_idx_t)); + idx->n = h->n_ref; + idx->index2 = calloc(h->n_ref, sizeof(bcf_lidx_t)); + + last_tid = 0xffffffffu; + last_off = bgzf_tell(fp); last_coor = 0xffffffffu; + while ((ret = bcf_read(bp, h, b)) > 0) { + int end, tmp; + if (last_tid != b->tid) { // change of chromosomes + last_tid = b->tid; + } else if (last_coor > b->pos) { + fprintf(stderr, "[bcf_idx_core] the input is out of order\n"); + free(str->s); free(str); free(idx); bcf_destroy(b); + return 0; + } + tmp = strlen(b->ref); + end = b->pos + (tmp > 0? tmp : 1); + insert_offset2(&idx->index2[b->tid], b->pos, end, last_off); + last_off = bgzf_tell(fp); + last_coor = b->pos; + } + free(str->s); free(str); bcf_destroy(b); + return idx; +} + +void bcf_idx_destroy(bcf_idx_t *idx) +{ + int i; + if (idx == 0) return; + for (i = 0; i < idx->n; ++i) free(idx->index2[i].offset); + free(idx->index2); + free(idx); +} + +/****************** + * index file I/O * + ******************/ + +void bcf_idx_save(const bcf_idx_t *idx, BGZF *fp) +{ + int32_t i, ti_is_be; + ti_is_be = bam_is_big_endian(); + bgzf_write(fp, "BCI\4", 4); + if (ti_is_be) { + uint32_t x = idx->n; + bgzf_write(fp, bam_swap_endian_4p(&x), 4); + } else bgzf_write(fp, &idx->n, 4); + for (i = 0; i < idx->n; ++i) { + bcf_lidx_t *index2 = idx->index2 + i; + // write linear index (index2) + if (ti_is_be) { + int x = index2->n; + bgzf_write(fp, bam_swap_endian_4p(&x), 4); + } else bgzf_write(fp, &index2->n, 4); + if (ti_is_be) { // big endian + int x; + for (x = 0; (int)x < index2->n; ++x) + bam_swap_endian_8p(&index2->offset[x]); + bgzf_write(fp, index2->offset, 8 * index2->n); + for (x = 0; (int)x < index2->n; ++x) + bam_swap_endian_8p(&index2->offset[x]); + } else bgzf_write(fp, index2->offset, 8 * index2->n); + } +} + +static bcf_idx_t *bcf_idx_load_core(BGZF *fp) +{ + int i, ti_is_be; + char magic[4]; + bcf_idx_t *idx; + ti_is_be = bam_is_big_endian(); + if (fp == 0) { + fprintf(stderr, "[%s] fail to load index.\n", __func__); + return 0; + } + bgzf_read(fp, magic, 4); + if (strncmp(magic, "BCI\4", 4)) { + fprintf(stderr, "[%s] wrong magic number.\n", __func__); + return 0; + } + idx = (bcf_idx_t*)calloc(1, sizeof(bcf_idx_t)); + bgzf_read(fp, &idx->n, 4); + if (ti_is_be) bam_swap_endian_4p(&idx->n); + idx->index2 = (bcf_lidx_t*)calloc(idx->n, sizeof(bcf_lidx_t)); + for (i = 0; i < idx->n; ++i) { + bcf_lidx_t *index2 = idx->index2 + i; + int j; + bgzf_read(fp, &index2->n, 4); + if (ti_is_be) bam_swap_endian_4p(&index2->n); + index2->m = index2->n; + index2->offset = (uint64_t*)calloc(index2->m, 8); + bgzf_read(fp, index2->offset, index2->n * 8); + if (ti_is_be) + for (j = 0; j < index2->n; ++j) bam_swap_endian_8p(&index2->offset[j]); + } + return idx; +} + +bcf_idx_t *bcf_idx_load_local(const char *fnidx) +{ + BGZF *fp; + fp = bgzf_open(fnidx, "r"); + if (fp) { + bcf_idx_t *idx = bcf_idx_load_core(fp); + bgzf_close(fp); + return idx; + } else return 0; +} + +#ifdef _USE_KNETFILE +static void download_from_remote(const char *url) +{ + const int buf_size = 1 * 1024 * 1024; + char *fn; + FILE *fp; + uint8_t *buf; + knetFile *fp_remote; + int l; + if (strstr(url, "ftp://") != url && strstr(url, "http://") != url) return; + l = strlen(url); + for (fn = (char*)url + l - 1; fn >= url; --fn) + if (*fn == '/') break; + ++fn; // fn now points to the file name + fp_remote = knet_open(url, "r"); + if (fp_remote == 0) { + fprintf(stderr, "[download_from_remote] fail to open remote file.\n"); + return; + } + if ((fp = fopen(fn, "w")) == 0) { + fprintf(stderr, "[download_from_remote] fail to create file in the working directory.\n"); + knet_close(fp_remote); + return; + } + buf = (uint8_t*)calloc(buf_size, 1); + while ((l = knet_read(fp_remote, buf, buf_size)) != 0) + fwrite(buf, 1, l, fp); + free(buf); + fclose(fp); + knet_close(fp_remote); +} +#else +static void download_from_remote(const char *url) +{ + return; +} +#endif + +static char *get_local_version(const char *fn) +{ + struct stat sbuf; + char *fnidx = (char*)calloc(strlen(fn) + 5, 1); + strcat(strcpy(fnidx, fn), ".bci"); + if ((strstr(fnidx, "ftp://") == fnidx || strstr(fnidx, "http://") == fnidx)) { + char *p, *url; + int l = strlen(fnidx); + for (p = fnidx + l - 1; p >= fnidx; --p) + if (*p == '/') break; + url = fnidx; fnidx = strdup(p + 1); + if (stat(fnidx, &sbuf) == 0) { + free(url); + return fnidx; + } + fprintf(stderr, "[%s] downloading the index file...\n", __func__); + download_from_remote(url); + free(url); + } + if (stat(fnidx, &sbuf) == 0) return fnidx; + free(fnidx); return 0; +} + +bcf_idx_t *bcf_idx_load(const char *fn) +{ + bcf_idx_t *idx; + char *fname = get_local_version(fn); + if (fname == 0) return 0; + idx = bcf_idx_load_local(fname); + free(fname); + return idx; +} + +int bcf_idx_build2(const char *fn, const char *_fnidx) +{ + char *fnidx; + BGZF *fpidx; + bcf_t *bp; + bcf_idx_t *idx; + bcf_hdr_t *h; + if ((bp = bcf_open(fn, "r")) == 0) { + fprintf(stderr, "[bcf_idx_build2] fail to open the BAM file.\n"); + return -1; + } + h = bcf_hdr_read(bp); + idx = bcf_idx_core(bp, h); + bcf_close(bp); + if (_fnidx == 0) { + fnidx = (char*)calloc(strlen(fn) + 5, 1); + strcpy(fnidx, fn); strcat(fnidx, ".bci"); + } else fnidx = strdup(_fnidx); + fpidx = bgzf_open(fnidx, "w"); + if (fpidx == 0) { + fprintf(stderr, "[bcf_idx_build2] fail to create the index file.\n"); + free(fnidx); + return -1; + } + bcf_idx_save(idx, fpidx); + bcf_idx_destroy(idx); + bgzf_close(fpidx); + free(fnidx); + return 0; +} + +int bcf_idx_build(const char *fn) +{ + return bcf_idx_build2(fn, 0); +} + +/******************************************** + * parse a region in the format chr:beg-end * + ********************************************/ + +int bcf_parse_region(void *str2id, const char *str, int *tid, int *begin, int *end) +{ + char *s, *p; + int i, l, k; + l = strlen(str); + p = s = (char*)malloc(l+1); + /* squeeze out "," */ + for (i = k = 0; i != l; ++i) + if (str[i] != ',' && !isspace(str[i])) s[k++] = str[i]; + s[k] = 0; + for (i = 0; i != k; ++i) if (s[i] == ':') break; + s[i] = 0; + if ((*tid = bcf_str2id(str2id, s)) < 0) { + free(s); + return -1; + } + if (i == k) { /* dump the whole sequence */ + *begin = 0; *end = 1<<29; free(s); + return 0; + } + for (p = s + i + 1; i != k; ++i) if (s[i] == '-') break; + *begin = atoi(p); + if (i < k) { + p = s + i + 1; + *end = atoi(p); + } else *end = 1<<29; + if (*begin > 0) --*begin; + free(s); + if (*begin > *end) return -1; + return 0; +} + +/******************************* + * retrieve a specified region * + *******************************/ + +uint64_t bcf_idx_query(const bcf_idx_t *idx, int tid, int beg) +{ + uint64_t min_off, *offset; + int i; + if (beg < 0) beg = 0; + offset = idx->index2[tid].offset; + for (i = beg>>TAD_LIDX_SHIFT; i < idx->index2[tid].n && offset[i] == 0; ++i); + min_off = (i == idx->index2[tid].n)? offset[idx->index2[tid].n-1] : offset[i]; + return min_off; +} + +int bcf_main_index(int argc, char *argv[]) +{ + if (argc == 1) { + fprintf(stderr, "Usage: bcftools index \n"); + return 1; + } + bcf_idx_build(argv[1]); + return 0; +} diff --git a/bcftools/kfunc.c b/bcftools/kfunc.c new file mode 100644 index 0000000..a637b6c --- /dev/null +++ b/bcftools/kfunc.c @@ -0,0 +1,162 @@ +#include + + +/* Log gamma function + * \log{\Gamma(z)} + * AS245, 2nd algorithm, http://lib.stat.cmu.edu/apstat/245 + */ +double kf_lgamma(double z) +{ + double x = 0; + x += 0.1659470187408462e-06 / (z+7); + x += 0.9934937113930748e-05 / (z+6); + x -= 0.1385710331296526 / (z+5); + x += 12.50734324009056 / (z+4); + x -= 176.6150291498386 / (z+3); + x += 771.3234287757674 / (z+2); + x -= 1259.139216722289 / (z+1); + x += 676.5203681218835 / z; + x += 0.9999999999995183; + return log(x) - 5.58106146679532777 - z + (z-0.5) * log(z+6.5); +} + +/* complementary error function + * \frac{2}{\sqrt{\pi}} \int_x^{\infty} e^{-t^2} dt + * AS66, 2nd algorithm, http://lib.stat.cmu.edu/apstat/66 + */ +double kf_erfc(double x) +{ + const double p0 = 220.2068679123761; + const double p1 = 221.2135961699311; + const double p2 = 112.0792914978709; + const double p3 = 33.912866078383; + const double p4 = 6.37396220353165; + const double p5 = .7003830644436881; + const double p6 = .03526249659989109; + const double q0 = 440.4137358247522; + const double q1 = 793.8265125199484; + const double q2 = 637.3336333788311; + const double q3 = 296.5642487796737; + const double q4 = 86.78073220294608; + const double q5 = 16.06417757920695; + const double q6 = 1.755667163182642; + const double q7 = .08838834764831844; + double expntl, z, p; + z = fabs(x) * M_SQRT2; + if (z > 37.) return x > 0.? 0. : 2.; + expntl = exp(z * z * - .5); + if (z < 10. / M_SQRT2) // for small z + p = expntl * ((((((p6 * z + p5) * z + p4) * z + p3) * z + p2) * z + p1) * z + p0) + / (((((((q7 * z + q6) * z + q5) * z + q4) * z + q3) * z + q2) * z + q1) * z + q0); + else p = expntl / 2.506628274631001 / (z + 1. / (z + 2. / (z + 3. / (z + 4. / (z + .65))))); + return x > 0.? 2. * p : 2. * (1. - p); +} + +/* The following computes regularized incomplete gamma functions. + * Formulas are taken from Wiki, with additional input from Numerical + * Recipes in C (for modified Lentz's algorithm) and AS245 + * (http://lib.stat.cmu.edu/apstat/245). + * + * A good online calculator is available at: + * + * http://www.danielsoper.com/statcalc/calc23.aspx + * + * It calculates upper incomplete gamma function, which equals + * kf_gammaq(s,z)*tgamma(s). + */ + +#define KF_GAMMA_EPS 1e-14 +#define KF_TINY 1e-290 + +// regularized lower incomplete gamma function, by series expansion +static double _kf_gammap(double s, double z) +{ + double sum, x; + int k; + for (k = 1, sum = x = 1.; k < 100; ++k) { + sum += (x *= z / (s + k)); + if (x / sum < KF_GAMMA_EPS) break; + } + return exp(s * log(z) - z - kf_lgamma(s + 1.) + log(sum)); +} +// regularized upper incomplete gamma function, by continued fraction +static double _kf_gammaq(double s, double z) +{ + int j; + double C, D, f; + f = 1. + z - s; C = f; D = 0.; + // Modified Lentz's algorithm for computing continued fraction + // See Numerical Recipes in C, 2nd edition, section 5.2 + for (j = 1; j < 100; ++j) { + double a = j * (s - j), b = (j<<1) + 1 + z - s, d; + D = b + a * D; + if (D < KF_TINY) D = KF_TINY; + C = b + a / C; + if (C < KF_TINY) C = KF_TINY; + D = 1. / D; + d = C * D; + f *= d; + if (fabs(d - 1.) < KF_GAMMA_EPS) break; + } + return exp(s * log(z) - z - kf_lgamma(s) - log(f)); +} + +double kf_gammap(double s, double z) +{ + return z <= 1. || z < s? _kf_gammap(s, z) : 1. - _kf_gammaq(s, z); +} + +double kf_gammaq(double s, double z) +{ + return z <= 1. || z < s? 1. - _kf_gammap(s, z) : _kf_gammaq(s, z); +} + +/* Regularized incomplete beta function. The method is taken from + * Numerical Recipe in C, 2nd edition, section 6.4. The following web + * page calculates the incomplete beta function, which equals + * kf_betai(a,b,x) * gamma(a) * gamma(b) / gamma(a+b): + * + * http://www.danielsoper.com/statcalc/calc36.aspx + */ +static double kf_betai_aux(double a, double b, double x) +{ + double C, D, f; + int j; + if (x == 0.) return 0.; + if (x == 1.) return 1.; + f = 1.; C = f; D = 0.; + // Modified Lentz's algorithm for computing continued fraction + for (j = 1; j < 200; ++j) { + double aa, d; + int m = j>>1; + aa = (j&1)? -(a + m) * (a + b + m) * x / ((a + 2*m) * (a + 2*m + 1)) + : m * (b - m) * x / ((a + 2*m - 1) * (a + 2*m)); + D = 1. + aa * D; + if (D < KF_TINY) D = KF_TINY; + C = 1. + aa / C; + if (C < KF_TINY) C = KF_TINY; + D = 1. / D; + d = C * D; + f *= d; + if (fabs(d - 1.) < KF_GAMMA_EPS) break; + } + return exp(kf_lgamma(a+b) - kf_lgamma(a) - kf_lgamma(b) + a * log(x) + b * log(1.-x)) / a / f; +} +double kf_betai(double a, double b, double x) +{ + return x < (a + 1.) / (a + b + 2.)? kf_betai_aux(a, b, x) : 1. - kf_betai_aux(b, a, 1. - x); +} + +#ifdef KF_MAIN +#include +int main(int argc, char *argv[]) +{ + double x = 5.5, y = 3; + double a, b; + printf("erfc(%lg): %lg, %lg\n", x, erfc(x), kf_erfc(x)); + printf("upper-gamma(%lg,%lg): %lg\n", x, y, kf_gammaq(y, x)*tgamma(y)); + a = 2; b = 2; x = 0.5; + printf("incomplete-beta(%lg,%lg,%lg): %lg\n", a, b, x, kf_betai(a, b, x) / exp(kf_lgamma(a+b) - kf_lgamma(a) - kf_lgamma(b))); + return 0; +} +#endif diff --git a/bcftools/ld.c b/bcftools/ld.c new file mode 100644 index 0000000..aa7ec07 --- /dev/null +++ b/bcftools/ld.c @@ -0,0 +1,100 @@ +#include +#include +#include +#include "bcf.h" + +static double g_q2p[256]; + +#define LD_ITER_MAX 50 +#define LD_ITER_EPS 1e-4 + +#define _G1(h, k) ((h>>1&1) + (k>>1&1)) +#define _G2(h, k) ((h&1) + (k&1)) + +// 0: the previous site; 1: the current site +static int freq_iter(int n, double *pdg[2], double f[4]) +{ + double ff[4]; + int i, k, h; + memset(ff, 0, 4 * sizeof(double)); + for (i = 0; i < n; ++i) { + double *p[2], sum, tmp; + p[0] = pdg[0] + i * 3; p[1] = pdg[1] + i * 3; + for (k = 0, sum = 0.; k < 4; ++k) + for (h = 0; h < 4; ++h) + sum += f[k] * f[h] * p[0][_G1(k,h)] * p[1][_G2(k,h)]; + for (k = 0; k < 4; ++k) { + tmp = f[0] * (p[0][_G1(0,k)] * p[1][_G2(0,k)] + p[0][_G1(k,0)] * p[1][_G2(k,0)]) + + f[1] * (p[0][_G1(1,k)] * p[1][_G2(1,k)] + p[0][_G1(k,1)] * p[1][_G2(k,1)]) + + f[2] * (p[0][_G1(2,k)] * p[1][_G2(2,k)] + p[0][_G1(k,2)] * p[1][_G2(k,2)]) + + f[3] * (p[0][_G1(3,k)] * p[1][_G2(3,k)] + p[0][_G1(k,3)] * p[1][_G2(k,3)]); + ff[k] += f[k] * tmp / sum; + } + } + for (k = 0; k < 4; ++k) f[k] = ff[k] / (2 * n); + return 0; +} + +double bcf_ld_freq(const bcf1_t *b0, const bcf1_t *b1, double f[4]) +{ + const bcf1_t *b[2]; + uint8_t *PL[2]; + int i, j, PL_len[2], n_smpl; + double *pdg[2], flast[4], r; + // initialize g_q2p if necessary + if (g_q2p[0] == 0.) + for (i = 0; i < 256; ++i) + g_q2p[i] = pow(10., -i / 10.); + // initialize others + if (b0->n_smpl != b1->n_smpl) return -1; // different number of samples + n_smpl = b0->n_smpl; + b[0] = b0; b[1] = b1; + f[0] = f[1] = f[2] = f[3] = -1.; + if (b[0]->n_alleles < 2 || b[1]->n_alleles < 2) return -1; // one allele only + // set PL and PL_len + for (j = 0; j < 2; ++j) { + const bcf1_t *bj = b[j]; + for (i = 0; i < bj->n_gi; ++i) { + if (bj->gi[i].fmt == bcf_str2int("PL", 2)) { + PL[j] = (uint8_t*)bj->gi[i].data; + PL_len[j] = bj->gi[i].len; + break; + } + } + if (i == bj->n_gi) return -1; // no PL + } + // fill pdg[2] + pdg[0] = malloc(3 * n_smpl * sizeof(double)); + pdg[1] = malloc(3 * n_smpl * sizeof(double)); + for (j = 0; j < 2; ++j) { + for (i = 0; i < n_smpl; ++i) { + const uint8_t *pi = PL[j] + i * PL_len[j]; + double *p = pdg[j] + i * 3; + p[0] = g_q2p[pi[b[j]->n_alleles]]; p[1] = g_q2p[pi[1]]; p[2] = g_q2p[pi[0]]; + } + } + // iteration + f[0] = f[1] = f[2] = f[3] = 0.25; // this is a really bad guess... + for (j = 0; j < LD_ITER_MAX; ++j) { + double eps = 0; + memcpy(flast, f, 4 * sizeof(double)); + freq_iter(n_smpl, pdg, f); + for (i = 0; i < 4; ++i) { + double x = fabs(f[0] - flast[0]); + if (x > eps) eps = x; + } + if (eps < LD_ITER_EPS) break; + } + // free + free(pdg[0]); free(pdg[1]); + { // calculate r^2 + double p[2], q[2], D; + p[0] = f[0] + f[1]; q[0] = 1 - p[0]; + p[1] = f[0] + f[2]; q[1] = 1 - p[1]; + D = f[0] * f[3] - f[1] * f[2]; + r = sqrt(D * D / (p[0] * p[1] * q[0] * q[1])); + // fprintf(stderr, "R(%lf,%lf,%lf,%lf)=%lf\n", f[0], f[1], f[2], f[3], r2); + if (isnan(r)) r = -1.; + } + return r; +} diff --git a/bcftools/main.c b/bcftools/main.c new file mode 100644 index 0000000..7ffc2a0 --- /dev/null +++ b/bcftools/main.c @@ -0,0 +1,64 @@ +#include +#include +#include +#include "bcf.h" + +int bcfview(int argc, char *argv[]); +int bcf_main_index(int argc, char *argv[]); + +#define BUF_SIZE 0x10000 + +int bcf_cat(int n, char * const *fn) +{ + int i; + bcf_t *out; + uint8_t *buf; + buf = malloc(BUF_SIZE); + out = bcf_open("-", "w"); + for (i = 0; i < n; ++i) { + bcf_t *in; + bcf_hdr_t *h; + off_t end; + struct stat s; + in = bcf_open(fn[i], "r"); + h = bcf_hdr_read(in); + if (i == 0) bcf_hdr_write(out, h); + bcf_hdr_destroy(h); +#ifdef _USE_KNETFILE + fstat(knet_fileno(in->fp->x.fpr), &s); + end = s.st_size - 28; + while (knet_tell(in->fp->x.fpr) < end) { + int size = knet_tell(in->fp->x.fpr) + BUF_SIZE < end? BUF_SIZE : end - knet_tell(in->fp->x.fpr); + knet_read(in->fp->x.fpr, buf, size); + fwrite(buf, 1, size, out->fp->x.fpw); + } +#else + abort(); // FIXME: not implemented +#endif + bcf_close(in); + } + bcf_close(out); + free(buf); + return 0; +} + +int main(int argc, char *argv[]) +{ + if (argc == 1) { + fprintf(stderr, "\n"); + fprintf(stderr, "Usage: bcftools \n\n"); + fprintf(stderr, "Command: view print, extract, convert and call SNPs from BCF\n"); + fprintf(stderr, " index index BCF\n"); + fprintf(stderr, " cat concatenate BCFs\n"); + fprintf(stderr, "\n"); + return 1; + } + if (strcmp(argv[1], "view") == 0) return bcfview(argc-1, argv+1); + else if (strcmp(argv[1], "index") == 0) return bcf_main_index(argc-1, argv+1); + else if (strcmp(argv[1], "cat") == 0) return bcf_cat(argc-2, argv+2); + else { + fprintf(stderr, "[main] Unrecognized command.\n"); + return 1; + } + return 0; +} diff --git a/bcftools/prob1.c b/bcftools/prob1.c new file mode 100644 index 0000000..e3b0f73 --- /dev/null +++ b/bcftools/prob1.c @@ -0,0 +1,394 @@ +#include +#include +#include +#include +#include +#include "prob1.h" + +#include "kseq.h" +KSTREAM_INIT(gzFile, gzread, 16384) + +#define MC_AVG_ERR 0.007 +#define MC_MAX_EM_ITER 16 +#define MC_EM_EPS 1e-4 + +unsigned char seq_nt4_table[256] = { + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4 /*'-'*/, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 0, 4, 1, 4, 4, 4, 2, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 0, 4, 1, 4, 4, 4, 2, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, + 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4 +}; + +struct __bcf_p1aux_t { + int n, M, n1; + double *q2p, *pdg; // pdg -> P(D|g) + double *phi; + double *z, *zswap; // aux for afs + double *z1, *z2, *phi1, *phi2; // only calculated when n1 is set + double t, t1, t2; + double *afs, *afs1; // afs: accumulative AFS; afs1: site posterior distribution + const uint8_t *PL; // point to PL + int PL_len; +}; + +static void init_prior(int type, double theta, int M, double *phi) +{ + int i; + if (type == MC_PTYPE_COND2) { + for (i = 0; i <= M; ++i) + phi[i] = 2. * (i + 1) / (M + 1) / (M + 2); + } else if (type == MC_PTYPE_FLAT) { + for (i = 0; i <= M; ++i) + phi[i] = 1. / (M + 1); + } else { + double sum; + for (i = 0, sum = 0.; i < M; ++i) + sum += (phi[i] = theta / (M - i)); + phi[M] = 1. - sum; + } +} + +void bcf_p1_init_prior(bcf_p1aux_t *ma, int type, double theta) +{ + init_prior(type, theta, ma->M, ma->phi); +} + +void bcf_p1_init_subprior(bcf_p1aux_t *ma, int type, double theta) +{ + if (ma->n1 <= 0 || ma->n1 >= ma->M) return; + init_prior(type, theta, 2*ma->n1, ma->phi1); + init_prior(type, theta, 2*(ma->n - ma->n1), ma->phi2); +} + +int bcf_p1_read_prior(bcf_p1aux_t *ma, const char *fn) +{ + gzFile fp; + kstring_t s; + kstream_t *ks; + long double sum; + int dret, k; + memset(&s, 0, sizeof(kstring_t)); + fp = strcmp(fn, "-")? gzopen(fn, "r") : gzdopen(fileno(stdin), "r"); + ks = ks_init(fp); + memset(ma->phi, 0, sizeof(double) * (ma->M + 1)); + while (ks_getuntil(ks, '\n', &s, &dret) >= 0) { + if (strstr(s.s, "[afs] ") == s.s) { + char *p = s.s + 6; + for (k = 0; k <= ma->M; ++k) { + int x; + double y; + x = strtol(p, &p, 10); + if (x != k && (errno == EINVAL || errno == ERANGE)) return -1; + ++p; + y = strtod(p, &p); + if (y == 0. && (errno == EINVAL || errno == ERANGE)) return -1; + ma->phi[ma->M - k] += y; + } + } + } + ks_destroy(ks); + gzclose(fp); + free(s.s); + for (sum = 0., k = 0; k <= ma->M; ++k) sum += ma->phi[k]; + fprintf(stderr, "[prior]"); + for (k = 0; k <= ma->M; ++k) ma->phi[k] /= sum; + for (k = 0; k <= ma->M; ++k) fprintf(stderr, " %d:%.3lg", k, ma->phi[ma->M - k]); + fputc('\n', stderr); + for (sum = 0., k = 1; k < ma->M; ++k) sum += ma->phi[ma->M - k] * (2.* k * (ma->M - k) / ma->M / (ma->M - 1)); + fprintf(stderr, "[%s] heterozygosity=%lf, ", __func__, (double)sum); + for (sum = 0., k = 1; k <= ma->M; ++k) sum += k * ma->phi[ma->M - k] / ma->M; + fprintf(stderr, "theta=%lf\n", (double)sum); + return 0; +} + +bcf_p1aux_t *bcf_p1_init(int n) +{ + bcf_p1aux_t *ma; + int i; + ma = calloc(1, sizeof(bcf_p1aux_t)); + ma->n1 = -1; + ma->n = n; ma->M = 2 * n; + ma->q2p = calloc(256, sizeof(double)); + ma->pdg = calloc(3 * ma->n, sizeof(double)); + ma->phi = calloc(ma->M + 1, sizeof(double)); + ma->phi1 = calloc(ma->M + 1, sizeof(double)); + ma->phi2 = calloc(ma->M + 1, sizeof(double)); + ma->z = calloc(2 * ma->n + 1, sizeof(double)); + ma->zswap = calloc(2 * ma->n + 1, sizeof(double)); + ma->z1 = calloc(ma->M + 1, sizeof(double)); // actually we do not need this large + ma->z2 = calloc(ma->M + 1, sizeof(double)); + ma->afs = calloc(2 * ma->n + 1, sizeof(double)); + ma->afs1 = calloc(2 * ma->n + 1, sizeof(double)); + for (i = 0; i < 256; ++i) + ma->q2p[i] = pow(10., -i / 10.); + bcf_p1_init_prior(ma, MC_PTYPE_FULL, 1e-3); // the simplest prior + return ma; +} + +int bcf_p1_set_n1(bcf_p1aux_t *b, int n1) +{ + if (n1 == 0 || n1 >= b->n) return -1; + b->n1 = n1; + return 0; +} + +void bcf_p1_destroy(bcf_p1aux_t *ma) +{ + if (ma) { + free(ma->q2p); free(ma->pdg); + free(ma->phi); free(ma->phi1); free(ma->phi2); + free(ma->z); free(ma->zswap); free(ma->z1); free(ma->z2); + free(ma->afs); free(ma->afs1); + free(ma); + } +} + +static int cal_pdg(const bcf1_t *b, bcf_p1aux_t *ma) +{ + int i, j, k; + long *p, tmp; + p = alloca(b->n_alleles * sizeof(long)); + memset(p, 0, sizeof(long) * b->n_alleles); + for (j = 0; j < ma->n; ++j) { + const uint8_t *pi = ma->PL + j * ma->PL_len; + double *pdg = ma->pdg + j * 3; + pdg[0] = ma->q2p[pi[b->n_alleles]]; pdg[1] = ma->q2p[pi[1]]; pdg[2] = ma->q2p[pi[0]]; + for (i = k = 0; i < b->n_alleles; ++i) { + p[i] += (int)pi[k]; + k += b->n_alleles - i; + } + } + for (i = 0; i < b->n_alleles; ++i) p[i] = p[i]<<4 | i; + for (i = 1; i < b->n_alleles; ++i) // insertion sort + for (j = i; j > 0 && p[j] < p[j-1]; --j) + tmp = p[j], p[j] = p[j-1], p[j-1] = tmp; + for (i = b->n_alleles - 1; i >= 0; --i) + if ((p[i]&0xf) == 0) break; + return i; +} +// f0 is the reference allele frequency +static double mc_freq_iter(double f0, const bcf_p1aux_t *ma) +{ + double f, f3[3]; + int i; + f3[0] = (1.-f0)*(1.-f0); f3[1] = 2.*f0*(1.-f0); f3[2] = f0*f0; + for (i = 0, f = 0.; i < ma->n; ++i) { + double *pdg; + pdg = ma->pdg + i * 3; + f += (pdg[1] * f3[1] + 2. * pdg[2] * f3[2]) + / (pdg[0] * f3[0] + pdg[1] * f3[1] + pdg[2] * f3[2]); + } + f /= ma->n * 2.; + return f; +} + +int bcf_p1_call_gt(const bcf_p1aux_t *ma, double f0, int k) +{ + double sum, g[3]; + double max, f3[3], *pdg = ma->pdg + k * 3; + int q, i, max_i; + f3[0] = (1.-f0)*(1.-f0); f3[1] = 2.*f0*(1.-f0); f3[2] = f0*f0; + for (i = 0, sum = 0.; i < 3; ++i) + sum += (g[i] = pdg[i] * f3[i]); + for (i = 0, max = -1., max_i = 0; i < 3; ++i) { + g[i] /= sum; + if (g[i] > max) max = g[i], max_i = i; + } + max = 1. - max; + if (max < 1e-308) max = 1e-308; + q = (int)(-4.343 * log(max) + .499); + if (q > 99) q = 99; + return q<<2|max_i; +} + +#define TINY 1e-20 + +static void mc_cal_y_core(bcf_p1aux_t *ma, int beg) +{ + double *z[2], *tmp, *pdg; + int _j, last_min, last_max; + z[0] = ma->z; + z[1] = ma->zswap; + pdg = ma->pdg; + memset(z[0], 0, sizeof(double) * (ma->M + 1)); + memset(z[1], 0, sizeof(double) * (ma->M + 1)); + z[0][0] = 1.; + last_min = last_max = 0; + ma->t = 0.; + for (_j = beg; _j < ma->n; ++_j) { + int k, j = _j - beg, _min = last_min, _max = last_max; + double p[3], sum; + pdg = ma->pdg + _j * 3; + p[0] = pdg[0]; p[1] = 2. * pdg[1]; p[2] = pdg[2]; + for (; _min < _max && z[0][_min] < TINY; ++_min) z[0][_min] = z[1][_min] = 0.; + for (; _max > _min && z[0][_max] < TINY; --_max) z[0][_max] = z[1][_max] = 0.; + _max += 2; + if (_min == 0) + k = 0, z[1][k] = (2*j+2-k)*(2*j-k+1) * p[0] * z[0][k]; + if (_min <= 1) + k = 1, z[1][k] = (2*j+2-k)*(2*j-k+1) * p[0] * z[0][k] + k*(2*j+2-k) * p[1] * z[0][k-1]; + for (k = _min < 2? 2 : _min; k <= _max; ++k) + z[1][k] = (2*j+2-k)*(2*j-k+1) * p[0] * z[0][k] + + k*(2*j+2-k) * p[1] * z[0][k-1] + + k*(k-1)* p[2] * z[0][k-2]; + for (k = _min, sum = 0.; k <= _max; ++k) sum += z[1][k]; + ma->t += log(sum / ((2. * j + 2) * (2. * j + 1))); + for (k = _min; k <= _max; ++k) z[1][k] /= sum; + if (_min >= 1) z[1][_min-1] = 0.; + if (_min >= 2) z[1][_min-2] = 0.; + if (j < ma->n - 1) z[1][_max+1] = z[1][_max+2] = 0.; + if (_j == ma->n1 - 1) { // set pop1 + ma->t1 = ma->t; + memcpy(ma->z1, z[1], sizeof(double) * (ma->n1 * 2 + 1)); + } + tmp = z[0]; z[0] = z[1]; z[1] = tmp; + last_min = _min; last_max = _max; + } + if (z[0] != ma->z) memcpy(ma->z, z[0], sizeof(double) * (ma->M + 1)); +} + +static void mc_cal_y(bcf_p1aux_t *ma) +{ + if (ma->n1 > 0 && ma->n1 < ma->n) { + int k; + long double x; + memset(ma->z1, 0, sizeof(double) * (2 * ma->n1 + 1)); + memset(ma->z2, 0, sizeof(double) * (2 * (ma->n - ma->n1) + 1)); + ma->t1 = ma->t2 = 0.; + mc_cal_y_core(ma, ma->n1); + ma->t2 = ma->t; + memcpy(ma->z2, ma->z, sizeof(double) * (2 * (ma->n - ma->n1) + 1)); + mc_cal_y_core(ma, 0); + // rescale z + x = expl(ma->t - (ma->t1 + ma->t2)); + for (k = 0; k <= ma->M; ++k) ma->z[k] *= x; + } else mc_cal_y_core(ma, 0); +} + +static void contrast(bcf_p1aux_t *ma, double pc[4]) // mc_cal_y() must be called before hand +{ + int k, n1 = ma->n1, n2 = ma->n - ma->n1; + long double sum1, sum2; + pc[0] = pc[1] = pc[2] = pc[3] = -1.; + if (n1 <= 0 || n2 <= 0) return; + for (k = 0, sum1 = 0.; k <= 2*n1; ++k) sum1 += ma->phi1[k] * ma->z1[k]; + for (k = 0, sum2 = 0.; k <= 2*n2; ++k) sum2 += ma->phi2[k] * ma->z2[k]; + pc[2] = ma->phi1[2*n1] * ma->z1[2*n1] / sum1; + pc[3] = ma->phi2[2*n2] * ma->z2[2*n2] / sum2; + for (k = 2; k < 4; ++k) { + pc[k] = pc[k] > .5? -(-4.343 * log(1. - pc[k] + TINY) + .499) : -4.343 * log(pc[k] + TINY) + .499; + pc[k] = (int)pc[k]; + if (pc[k] > 99) pc[k] = 99; + if (pc[k] < -99) pc[k] = -99; + } + pc[0] = ma->phi2[2*n2] * ma->z2[2*n2] / sum2 * (1. - ma->phi1[2*n1] * ma->z1[2*n1] / sum1); + pc[1] = ma->phi1[2*n1] * ma->z1[2*n1] / sum1 * (1. - ma->phi2[2*n2] * ma->z2[2*n2] / sum2); + pc[0] = pc[0] == 1.? 99 : (int)(-4.343 * log(1. - pc[0]) + .499); + pc[1] = pc[1] == 1.? 99 : (int)(-4.343 * log(1. - pc[1]) + .499); +} + +static double mc_cal_afs(bcf_p1aux_t *ma) +{ + int k; + long double sum = 0.; + memset(ma->afs1, 0, sizeof(double) * (ma->M + 1)); + mc_cal_y(ma); + for (k = 0, sum = 0.; k <= ma->M; ++k) + sum += (long double)ma->phi[k] * ma->z[k]; + for (k = 0; k <= ma->M; ++k) { + ma->afs1[k] = ma->phi[k] * ma->z[k] / sum; + if (isnan(ma->afs1[k]) || isinf(ma->afs1[k])) return -1.; + } + for (k = 0, sum = 0.; k <= ma->M; ++k) { + ma->afs[k] += ma->afs1[k]; + sum += k * ma->afs1[k]; + } + return sum / ma->M; +} + +long double bcf_p1_cal_g3(bcf_p1aux_t *p1a, double g[3]) +{ + long double pd = 0., g2[3]; + int i, k; + memset(g2, 0, sizeof(long double) * 3); + for (k = 0; k < p1a->M; ++k) { + double f = (double)k / p1a->M, f3[3], g1[3]; + long double z = 1.; + g1[0] = g1[1] = g1[2] = 0.; + f3[0] = (1. - f) * (1. - f); f3[1] = 2. * f * (1. - f); f3[2] = f * f; + for (i = 0; i < p1a->n; ++i) { + double *pdg = p1a->pdg + i * 3; + double x = pdg[0] * f3[0] + pdg[1] * f3[1] + pdg[2] * f3[2]; + z *= x; + g1[0] += pdg[0] * f3[0] / x; + g1[1] += pdg[1] * f3[1] / x; + g1[2] += pdg[2] * f3[2] / x; + } + pd += p1a->phi[k] * z; + for (i = 0; i < 3; ++i) + g2[i] += p1a->phi[k] * z * g1[i]; + } + for (i = 0; i < 3; ++i) g[i] = g2[i] / pd; + return pd; +} + +int bcf_p1_cal(bcf1_t *b, bcf_p1aux_t *ma, bcf_p1rst_t *rst) +{ + int i, k; + long double sum = 0.; + // set PL and PL_len + for (i = 0; i < b->n_gi; ++i) { + if (b->gi[i].fmt == bcf_str2int("PL", 2)) { + ma->PL = (uint8_t*)b->gi[i].data; + ma->PL_len = b->gi[i].len; + break; + } + } + if (b->n_alleles < 2) return -1; // FIXME: find a better solution + // + rst->rank0 = cal_pdg(b, ma); + rst->f_exp = mc_cal_afs(ma); + rst->p_ref = ma->afs1[ma->M]; + // calculate f_flat and f_em + for (k = 0, sum = 0.; k <= ma->M; ++k) + sum += (long double)ma->z[k]; + rst->f_flat = 0.; + for (k = 0; k <= ma->M; ++k) { + double p = ma->z[k] / sum; + rst->f_flat += k * p; + } + rst->f_flat /= ma->M; + { // calculate f_em + double flast = rst->f_flat; + for (i = 0; i < MC_MAX_EM_ITER; ++i) { + rst->f_em = mc_freq_iter(flast, ma); + if (fabs(rst->f_em - flast) < MC_EM_EPS) break; + flast = rst->f_em; + } + } + rst->g[0] = rst->g[1] = rst->g[2] = -1.; + contrast(ma, rst->pc); + return 0; +} + +void bcf_p1_dump_afs(bcf_p1aux_t *ma) +{ + int k; + fprintf(stderr, "[afs]"); + for (k = 0; k <= ma->M; ++k) + fprintf(stderr, " %d:%.3lf", k, ma->afs[ma->M - k]); + fprintf(stderr, "\n"); + memset(ma->afs, 0, sizeof(double) * (ma->M + 1)); +} diff --git a/bcftools/prob1.h b/bcftools/prob1.h new file mode 100644 index 0000000..7158fe2 --- /dev/null +++ b/bcftools/prob1.h @@ -0,0 +1,39 @@ +#ifndef BCF_PROB1_H +#define BCF_PROB1_H + +#include "bcf.h" + +struct __bcf_p1aux_t; +typedef struct __bcf_p1aux_t bcf_p1aux_t; + +typedef struct { + int rank0; + double f_em, f_exp, f_flat, p_ref; + double pc[4]; + double g[3]; +} bcf_p1rst_t; + +#define MC_PTYPE_FULL 1 +#define MC_PTYPE_COND2 2 +#define MC_PTYPE_FLAT 3 + +#ifdef __cplusplus +extern "C" { +#endif + + bcf_p1aux_t *bcf_p1_init(int n); + void bcf_p1_init_prior(bcf_p1aux_t *ma, int type, double theta); + void bcf_p1_init_subprior(bcf_p1aux_t *ma, int type, double theta); + void bcf_p1_destroy(bcf_p1aux_t *ma); + int bcf_p1_cal(bcf1_t *b, bcf_p1aux_t *ma, bcf_p1rst_t *rst); + int bcf_p1_call_gt(const bcf_p1aux_t *ma, double f0, int k); + void bcf_p1_dump_afs(bcf_p1aux_t *ma); + int bcf_p1_read_prior(bcf_p1aux_t *ma, const char *fn); + long double bcf_p1_cal_g3(bcf_p1aux_t *p1a, double g[3]); + int bcf_p1_set_n1(bcf_p1aux_t *b, int n1); + +#ifdef __cplusplus +} +#endif + +#endif diff --git a/bcftools/vcf.c b/bcftools/vcf.c new file mode 100644 index 0000000..ebca869 --- /dev/null +++ b/bcftools/vcf.c @@ -0,0 +1,212 @@ +#include +#include +#include +#include +#include "bcf.h" +#include "kstring.h" +#include "kseq.h" +KSTREAM_INIT(gzFile, gzread, 4096) + +typedef struct { + gzFile fp; + FILE *fpout; + kstream_t *ks; + void *refhash; + kstring_t line; + int max_ref; +} vcf_t; + +bcf_hdr_t *vcf_hdr_read(bcf_t *bp) +{ + kstring_t meta, smpl; + int dret; + vcf_t *v; + bcf_hdr_t *h; + if (!bp->is_vcf) return bcf_hdr_read(bp); + h = calloc(1, sizeof(bcf_hdr_t)); + v = (vcf_t*)bp->v; + v->line.l = 0; + memset(&meta, 0, sizeof(kstring_t)); + memset(&smpl, 0, sizeof(kstring_t)); + while (ks_getuntil(v->ks, '\n', &v->line, &dret) >= 0) { + if (v->line.l < 2) continue; + if (v->line.s[0] != '#') return 0; // no sample line + if (v->line.s[0] == '#' && v->line.s[1] == '#') { + kputsn(v->line.s, v->line.l, &meta); kputc('\n', &meta); + } else if (v->line.s[0] == '#') { + int k; + ks_tokaux_t aux; + char *p; + for (p = kstrtok(v->line.s, "\t\n", &aux), k = 0; p; p = kstrtok(0, 0, &aux), ++k) { + if (k >= 9) { + kputsn(p, aux.p - p, &smpl); + kputc('\0', &smpl); + } + } + break; + } + } + kputc('\0', &meta); + h->name = 0; + h->sname = smpl.s; h->l_smpl = smpl.l; + h->txt = meta.s; h->l_txt = meta.l; + bcf_hdr_sync(h); + return h; +} + +bcf_t *vcf_open(const char *fn, const char *mode) +{ + bcf_t *bp; + vcf_t *v; + if (strchr(mode, 'b')) return bcf_open(fn, mode); + bp = calloc(1, sizeof(bcf_t)); + v = calloc(1, sizeof(vcf_t)); + bp->is_vcf = 1; + bp->v = v; + v->refhash = bcf_str2id_init(); + if (strchr(mode, 'r')) { + v->fp = strcmp(fn, "-")? gzopen(fn, "r") : gzdopen(fileno(stdin), "r"); + v->ks = ks_init(v->fp); + } else if (strchr(mode, 'w')) + v->fpout = strcmp(fn, "-")? fopen(fn, "w") : stdout; + return bp; +} + +int vcf_close(bcf_t *bp) +{ + vcf_t *v; + if (bp == 0) return -1; + if (!bp->is_vcf) return bcf_close(bp); + v = (vcf_t*)bp->v; + if (v->fp) { + ks_destroy(v->ks); + gzclose(v->fp); + } + if (v->fpout) fclose(v->fpout); + free(v->line.s); + bcf_str2id_destroy(v->refhash); + free(v); + free(bp); + return 0; +} + +int vcf_hdr_write(bcf_t *bp, const bcf_hdr_t *h) +{ + vcf_t *v = (vcf_t*)bp->v; + int i, has_ref = 0, has_ver = 0; + if (!bp->is_vcf) return bcf_hdr_write(bp, h); + if (h->l_txt > 0) { + if (strstr(h->txt, "##fileformat=")) has_ver = 1; + if (has_ver == 0) fprintf(v->fpout, "##fileformat=VCFv4.0\n"); + fwrite(h->txt, 1, h->l_txt - 1, v->fpout); + if (strstr(h->txt, "##SQ=")) has_ref = 1; + } + if (has_ver == 0) fprintf(v->fpout, "##fileformat=VCFv4.0\n"); + fprintf(v->fpout, "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT"); + for (i = 0; i < h->n_smpl; ++i) + fprintf(v->fpout, "\t%s", h->sns[i]); + fputc('\n', v->fpout); + return 0; +} + +int vcf_write(bcf_t *bp, bcf_hdr_t *h, bcf1_t *b) +{ + vcf_t *v = (vcf_t*)bp->v; + extern void bcf_fmt_core(const bcf_hdr_t *h, bcf1_t *b, kstring_t *s); + if (!bp->is_vcf) return bcf_write(bp, h, b); + bcf_fmt_core(h, b, &v->line); + fwrite(v->line.s, 1, v->line.l, v->fpout); + fputc('\n', v->fpout); + return v->line.l + 1; +} + +int vcf_read(bcf_t *bp, bcf_hdr_t *h, bcf1_t *b) +{ + int dret, k, i, sync = 0; + vcf_t *v = (vcf_t*)bp->v; + char *p, *q; + kstring_t str, rn; + ks_tokaux_t aux, a2; + if (!bp->is_vcf) return bcf_read(bp, h, b); + v->line.l = 0; + str.l = 0; str.m = b->m_str; str.s = b->str; + rn.l = rn.m = h->l_nm; rn.s = h->name; + if (ks_getuntil(v->ks, '\n', &v->line, &dret) < 0) return -1; + b->n_smpl = h->n_smpl; + for (p = kstrtok(v->line.s, "\t", &aux), k = 0; p; p = kstrtok(0, 0, &aux), ++k) { + *(char*)aux.p = 0; + if (k == 0) { // ref + int tid = bcf_str2id(v->refhash, p); + if (tid < 0) { + tid = bcf_str2id_add(v->refhash, p); + kputs(p, &rn); kputc('\0', &rn); + sync = 1; + } + b->tid = tid; + } else if (k == 1) { // pos + b->pos = atoi(p) - 1; + } else if (k == 5) { // qual + b->qual = (p[0] >= '0' && p[0] <= '9')? atof(p) : 0; + } else if (k <= 8) { // variable length strings + kputs(p, &str); kputc('\0', &str); + b->l_str = str.l; b->m_str = str.m; b->str = str.s; + if (k == 8) bcf_sync(b); + } else { // k > 9 + if (strncmp(p, "./.", 3) == 0) { + for (i = 0; i < b->n_gi; ++i) { + if (b->gi[i].fmt == bcf_str2int("GT", 2)) { + ((uint8_t*)b->gi[i].data)[k-9] = 1<<7; + } else if (b->gi[i].fmt == bcf_str2int("GQ", 2)) { + ((uint8_t*)b->gi[i].data)[k-9] = 0; + } else if (b->gi[i].fmt == bcf_str2int("DP", 2)) { + ((uint16_t*)b->gi[i].data)[k-9] = 0; + } else if (b->gi[i].fmt == bcf_str2int("PL", 2)) { + int y = b->n_alleles * (b->n_alleles + 1) / 2; + memset((uint8_t*)b->gi[i].data + (k - 9) * y, 0, y); + } else if (b->gi[i].fmt == bcf_str2int("GL", 2)) { + int y = b->n_alleles * (b->n_alleles + 1) / 2; + memset((float*)b->gi[i].data + (k - 9) * y, 0, y * 4); + } + } + goto endblock; + } + for (q = kstrtok(p, ":", &a2), i = 0; q && i < b->n_gi; q = kstrtok(0, 0, &a2), ++i) { + if (b->gi[i].fmt == bcf_str2int("GT", 2)) { + ((uint8_t*)b->gi[i].data)[k-9] = (q[0] - '0')<<3 | (q[2] - '0') | (q[1] == '/'? 0 : 1) << 6; + } else if (b->gi[i].fmt == bcf_str2int("GQ", 2)) { + double _x = strtod(q, &q); + int x = (int)(_x + .499); + if (x > 255) x = 255; + ((uint8_t*)b->gi[i].data)[k-9] = x; + } else if (b->gi[i].fmt == bcf_str2int("DP", 2)) { + int x = strtol(q, &q, 10); + if (x > 0xffff) x = 0xffff; + ((uint16_t*)b->gi[i].data)[k-9] = x; + } else if (b->gi[i].fmt == bcf_str2int("PL", 2)) { + int x, y, j; + uint8_t *data = (uint8_t*)b->gi[i].data; + y = b->n_alleles * (b->n_alleles + 1) / 2; + for (j = 0; j < y; ++j) { + x = strtol(q, &q, 10); + if (x > 255) x = 255; + data[(k-9) * y + j] = x; + ++q; + } + } else if (b->gi[i].fmt == bcf_str2int("GL", 2)) { + int j, y; + float x, *data = (float*)b->gi[i].data; + y = b->n_alleles * (b->n_alleles + 1) / 2; + for (j = 0; j < y; ++j) { + x = strtod(q, &q); + data[(k-9) * y + j] = x; + ++q; + } + } + } + endblock: i = i; + } + } + h->l_nm = rn.l; h->name = rn.s; + if (sync) bcf_hdr_sync(h); + return v->line.l + 1; +} diff --git a/bcftools/vcfutils.pl b/bcftools/vcfutils.pl new file mode 100755 index 0000000..d0b7971 --- /dev/null +++ b/bcftools/vcfutils.pl @@ -0,0 +1,477 @@ +#!/usr/bin/perl -w + +# Author: lh3 + +use strict; +use warnings; +use Getopt::Std; + +&main; +exit; + +sub main { + my $version = '0.1.0'; + &usage if (@ARGV < 1); + my $command = shift(@ARGV); + my %func = (subsam=>\&subsam, listsam=>\&listsam, fillac=>\&fillac, qstats=>\&qstats, varFilter=>\&varFilter, + hapmap2vcf=>\&hapmap2vcf, ucscsnp2vcf=>\&ucscsnp2vcf, filter4vcf=>\&filter4vcf, ldstats=>\&ldstats); + die("Unknown command \"$command\".\n") if (!defined($func{$command})); + &{$func{$command}}; +} + +sub subsam { + die(qq/Usage: vcfutils.pl subsam [samples]\n/) if (@ARGV == 0); + my ($fh, %h); + my $fn = shift(@ARGV); + my @col; + open($fh, ($fn =~ /\.gz$/)? "gzip -dc $fn |" : $fn) || die; + $h{$_} = 1 for (@ARGV); + while (<$fh>) { + if (/^##/) { + print; + } elsif (/^#/) { + my @t = split; + my @s = @t[0..8]; # all fixed fields + FORMAT + for (9 .. $#t) { + if ($h{$t[$_]}) { + push(@s, $t[$_]); + push(@col, $_); + } + } + pop(@s) if (@s == 9); # no sample selected; remove the FORMAT field + print join("\t", @s), "\n"; + } else { + my @t = split; + if (@col == 0) { + print join("\t", @t[0..7]), "\n"; + } else { + print join("\t", @t[0..8], map {$t[$_]} @col), "\n"; + } + } + } + close($fh); +} + +sub listsam { + die(qq/Usage: vcfutils.pl listsam \n/) if (@ARGV == 0 && -t STDIN); + while (<>) { + if (/^#/ && !/^##/) { + my @t = split; + print join("\n", @t[9..$#t]), "\n"; + exit; + } + } +} + +sub fillac { + die(qq/Usage: vcfutils.pl fillac \n\nNote: The GT field MUST BE present and always appear as the first field.\n/) if (@ARGV == 0 && -t STDIN); + while (<>) { + if (/^#/) { + print; + } else { + my @t = split; + my @c = (0); + my $n = 0; + my $s = -1; + @_ = split(":", $t[8]); + for (0 .. $#_) { + if ($_[$_] eq 'GT') { $s = $_; last; } + } + if ($s < 0) { + print join("\t", @t), "\n"; + next; + } + for (9 .. $#t) { + if ($t[$_] =~ /^0,0,0/) { + } elsif ($t[$_] =~ /^([^\s:]+:){$s}(\d+).(\d+)/) { + ++$c[$2]; ++$c[$3]; + $n += 2; + } + } + my $AC = "AC=" . join("\t", @c[1..$#c]) . ";AN=$n"; + my $info = $t[7]; + $info =~ s/(;?)AC=(\d+)//; + $info =~ s/(;?)AN=(\d+)//; + if ($info eq '.') { + $info = $AC; + } else { + $info .= ";$AC"; + } + $t[7] = $info; + print join("\t", @t), "\n"; + } + } +} + +sub ldstats { + my %opts = (t=>0.9); + getopts('t:', \%opts); + die("Usage: vcfutils.pl ldstats [-t $opts{t}] \n") if (@ARGV == 0 && -t STDIN); + my $cutoff = $opts{t}; + my ($last, $lastchr) = (0x7fffffff, ''); + my ($x, $y, $n) = (0, 0, 0); + while (<>) { + if (/^([^#\s]+)\s(\d+)/) { + my ($chr, $pos) = ($1, $2); + if (/NEIR=([\d\.]+)/) { + ++$n; + ++$y, $x += $pos - $last if ($lastchr eq $chr && $pos > $last && $1 > $cutoff); + } + $last = $pos; $lastchr = $chr; + } + } + print "Number of SNP intervals in strong LD (r > $opts{t}): $y\n"; + print "Fraction: ", $y/$n, "\n"; + print "Length: $x\n"; +} + +sub qstats { + my %opts = (r=>'', s=>0.02, v=>undef); + getopts('r:s:v', \%opts); + die("Usage: vcfutils.pl qstats [-r ref.vcf] \n +Note: This command discards indels. Output: QUAL #non-indel #SNPs #transitions #joint ts/tv #joint/#ref #joint/#non-indel \n") if (@ARGV == 0 && -t STDIN); + my %ts = (AG=>1, GA=>1, CT=>1, TC=>1); + my %h = (); + my $is_vcf = defined($opts{v})? 1 : 0; + if ($opts{r}) { # read the reference positions + my $fh; + open($fh, $opts{r}) || die; + while (<$fh>) { + next if (/^#/); + if ($is_vcf) { + my @t = split; + $h{$t[0],$t[1]} = $t[4]; + } else { + $h{$1,$2} = 1 if (/^(\S+)\s+(\d+)/); + } + } + close($fh); + } + my $hsize = scalar(keys %h); + my @a; + while (<>) { + next if (/^#/); + my @t = split; + next if (length($t[3]) != 1 || uc($t[3]) eq 'N'); + $t[3] = uc($t[3]); $t[4] = uc($t[4]); + my @s = split(',', $t[4]); + $t[5] = 3 if ($t[5] < 0); + next if (length($s[0]) != 1); + my $hit; + if ($is_vcf) { + $hit = 0; + my $aa = $h{$t[0],$t[1]}; + if (defined($aa)) { + my @aaa = split(",", $aa); + for (@aaa) { + $hit = 1 if ($_ eq $s[0]); + } + } + } else { + $hit = defined($h{$t[0],$t[1]})? 1 : 0; + } + push(@a, [$t[5], ($t[4] eq '.' || $t[4] eq $t[3])? 0 : 1, $ts{$t[3].$s[0]}? 1 : 0, $hit]); + } + push(@a, [-1, 0, 0, 0]); # end marker + die("[qstats] No SNP data!\n") if (@a == 0); + @a = sort {$b->[0]<=>$a->[0]} @a; + my $next = $opts{s}; + my $last = $a[0]; + my @c = (0, 0, 0, 0); + my @lc; + $lc[1] = $lc[2] = 0; + for my $p (@a) { + if ($p->[0] == -1 || ($p->[0] != $last && $c[0]/@a > $next)) { + my @x; + $x[0] = sprintf("%.4f", $c[1]-$c[2]? $c[2] / ($c[1] - $c[2]) : 100); + $x[1] = sprintf("%.4f", $hsize? $c[3] / $hsize : 0); + $x[2] = sprintf("%.4f", $c[3] / $c[1]); + my $a = $c[1] - $lc[1]; + my $b = $c[2] - $lc[2]; + $x[3] = sprintf("%.4f", $a-$b? $b / ($a-$b) : 100); + print join("\t", $last, @c, @x), "\n"; + $next = $c[0]/@a + $opts{s}; + $lc[1] = $c[1]; $lc[2] = $c[2]; + } + ++$c[0]; $c[1] += $p->[1]; $c[2] += $p->[2]; $c[3] += $p->[3]; + $last = $p->[0]; + } +} + +sub varFilter { + my %opts = (d=>1, D=>10000, l=>30, Q=>25, q=>10, G=>25, s=>100, w=>10, W=>10, N=>2, p=>undef, F=>.001); + getopts('pq:d:D:l:Q:w:W:N:G:F:', \%opts); + die(qq/ +Usage: vcfutils.pl varFilter [options] + +Options: -Q INT minimum RMS mapping quality for SNPs [$opts{Q}] + -q INT minimum RMS mapping quality for gaps [$opts{q}] + -d INT minimum read depth [$opts{d}] + -D INT maximum read depth [$opts{D}] + + -G INT min indel score for nearby SNP filtering [$opts{G}] + -w INT SNP within INT bp around a gap to be filtered [$opts{w}] + + -W INT window size for filtering dense SNPs [$opts{W}] + -N INT max number of SNPs in a window [$opts{N}] + + -l INT window size for filtering adjacent gaps [$opts{l}] + + -p print filtered variants +\n/) if (@ARGV == 0 && -t STDIN); + + # calculate the window size + my ($ol, $ow, $oW) = ($opts{l}, $opts{w}, $opts{W}); + my $max_dist = $ol > $ow? $ol : $ow; + $max_dist = $oW if ($max_dist < $oW); + # the core loop + my @staging; # (indel_filtering_score, flt_tag) + while (<>) { + my @t = split; + next if (/^#/); + next if ($t[4] eq '.'); # skip non-var sites + my $is_snp = 1; + if (length($t[3]) > 1) { + $is_snp = 0; + } else { + my @s = split(',', $t[4]); + for (@s) { + $is_snp = 0 if (length > 1); + } + } + # clear the out-of-range elements + while (@staging) { + # Still on the same chromosome and the first element's window still affects this position? + last if ($staging[0][3] eq $t[0] && $staging[0][4] + $staging[0][2] + $max_dist >= $t[1]); + varFilter_aux(shift(@staging), $opts{p}); # calling a function is a bit slower, not much + } + my ($flt, $score) = (0, -1); + + # collect key annotations + my ($dp, $mq, $af) = (-1, -1, 1); + if ($t[7] =~ /DP=(\d+)/i) { + $dp = $1; + } elsif ($t[7] =~ /DP4=(\d+),(\d+),(\d+),(\d+)/i) { + $dp = $1 + $2 + $3 + $4; + } + if ($t[7] =~ /MQ=(\d+)/i) { + $mq = $1; + } + if ($t[7] =~ /AF=([^\s;=]+)/i) { + $af = $1; + } elsif ($t[7] =~ /AF1=([^\s;=]+)/i) { + $af = $1; + } + # the depth filter + if ($dp >= 0) { + if ($dp < $opts{d}) { + $flt = 2; + } elsif ($dp > $opts{D}) { + $flt = 3; + } + } + + # site dependent filters + my $dlen = 0; + if ($flt == 0) { + if (!$is_snp) { # an indel + # If deletion, remember the length of the deletion + $dlen = length($t[3]) - 1; + $flt = 1 if ($mq < $opts{q}); + # filtering SNPs + if ($t[5] >= $opts{G}) { + for my $x (@staging) { + # Is it a SNP and is it outside the SNP filter window? + next if ($x->[0] >= 0 || $x->[4] + $x->[2] + $ow < $t[1]); + $x->[1] = 5 if ($x->[1] == 0); + } + } + # the indel filtering score + $score = $t[5]; + # check the staging list for indel filtering + for my $x (@staging) { + # Is it a SNP and is it outside the gap filter window + next if ($x->[0] < 0 || $x->[4] + $x->[2] + $ol < $t[1]); + if ($x->[0] < $score) { + $x->[1] = 6; + } else { + $flt = 6; last; + } + } + } else { # a SNP + $flt = 1 if ($mq < $opts{Q}); + # check adjacent SNPs + my $k = 1; + for my $x (@staging) { + ++$k if ($x->[0] < 0 && -($x->[0] + 1) > $opts{F} && $x->[4] + $x->[2] + $oW >= $t[1] && ($x->[1] == 0 || $x->[1] == 4 || $x->[1] == 5)); + } + # filtering is necessary + if ($k > $opts{N}) { + $flt = 4; + for my $x (@staging) { + $x->[1] = 4 if ($x->[0] < 0 && $x->[4] + $x->[2] + $oW >= $t[1] && $x->[1] == 0); + } + } else { # then check gap filter + for my $x (@staging) { + next if ($x->[0] < 0 || $x->[4] + $x->[2] + $ow < $t[1]); + if ($x->[0] >= $opts{G}) { + $flt = 5; last; + } + } + } + } + } + push(@staging, [$score < 0? -$af-1 : $score, $flt, $dlen, @t]); + } + # output the last few elements in the staging list + while (@staging) { + varFilter_aux(shift @staging, $opts{p}); + } +} + +sub varFilter_aux { + my ($first, $is_print) = @_; + if ($first->[1] == 0) { + print join("\t", @$first[3 .. @$first-1]), "\n"; + } elsif ($is_print) { + print STDERR join("\t", substr("UQdDWGgsiX", $first->[1], 1), @$first[3 .. @$first-1]), "\n"; + } +} + +sub filter4vcf { + my %opts = (d=>3, D=>2000, 1=>1e-4, 2=>1e-100, 3=>0, 4=>1e-4, Q=>10, q=>3); + getopts('d:D:1:2:3:4:Q:q:', \%opts); + die(qq/ +Usage: vcfutils.pl filter4vcf [options] + +Options: -d INT min total depth (given DP or DP4) [$opts{d}] + -D INT max total depth [$opts{D}] + -q INT min SNP quality [$opts{q}] + -Q INT min RMS mapQ (given MQ) [$opts{Q}] + -1 FLOAT min P-value for strand bias (given PV4) [$opts{1}] + -2 FLOAT min P-value for baseQ bias [$opts{2}] + -3 FLOAT min P-value for mapQ bias [$opts{3}] + -4 FLOAT min P-value for end distance bias [$opts{4}]\n +/) if (@ARGV == 0 && -t STDIN); + + my %ts = (AG=>1, GA=>1, CT=>1, TC=>1); + + my @n = (0, 0); + while (<>) { + if (/^#/) { + print; + next; + } + next if (/PV4=([^,]+),([^,]+),([^,]+),([^,;\t]+)/ && ($1<$opts{1} || $2<$opts{2} || $3<$opts{3} || $4<$opts{4})); + my $depth = -1; + $depth = $1 if (/DP=(\d+)/); + $depth = $1+$2+$3+$4 if (/DP4=(\d+),(\d+),(\d+),(\d+)/); + next if ($depth > 0 && ($depth < $opts{d} || $depth > $opts{D})); + next if (/MQ=(\d+)/ && $1 < $opts{Q}); + my @t = split; + next if ($t[5] >= 0 && $t[5] < $opts{q}); + ++$n[0]; + my @s = split(',', $t[4]); + ++$n[1] if ($ts{$t[3].$s[0]}); + print; + } +} + +sub ucscsnp2vcf { + die("Usage: vcfutils.pl \n") if (@ARGV == 0 && -t STDIN); + print "##fileformat=VCFv4.0\n"; + print join("\t", "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO"), "\n"; + while (<>) { + my @t = split("\t"); + my $indel = ($t[9] =~ /^[ACGT](\/[ACGT])+$/)? 0 : 1; + my $pos = $t[2] + 1; + my @alt; + push(@alt, $t[7]); + if ($t[6] eq '-') { + $t[9] = reverse($t[9]); + $t[9] =~ tr/ACGTRYMKWSNacgtrymkwsn/TGCAYRKMWSNtgcayrkmwsn/; + } + my @a = split("/", $t[9]); + for (@a) { + push(@alt, $_) if ($_ ne $alt[0]); + } + if ($indel) { + --$pos; + for (0 .. $#alt) { + $alt[$_] =~ tr/-//d; + $alt[$_] = "N$alt[$_]"; + } + } + my $ref = shift(@alt); + my $af = $t[13] > 0? ";AF=$t[13]" : ''; + my $valid = ($t[12] eq 'unknown')? '' : ";valid=$t[12]"; + my $info = "molType=$t[10];class=$t[11]$valid$af"; + print join("\t", $t[1], $pos, $t[4], $ref, join(",", @alt), 0, '.', $info), "\n"; + } +} + +sub hapmap2vcf { + die("Usage: vcfutils.pl \n") if (@ARGV == 0); + my $fn = shift(@ARGV); + # parse UCSC SNP + warn("Parsing UCSC SNPs...\n"); + my ($fh, %map); + open($fh, ($fn =~ /\.gz$/)? "gzip -dc $fn |" : $fn) || die; + while (<$fh>) { + my @t = split; + next if ($t[3] - $t[2] != 1); # not SNP + @{$map{$t[4]}} = @t[1,3,7]; + } + close($fh); + # write VCF + warn("Writing VCF...\n"); + print "##fileformat=VCFv4.0\n"; + while (<>) { + my @t = split; + if ($t[0] eq 'rs#') { # the first line + print join("\t", "#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT", @t[11..$#t]), "\n"; + } else { + next unless ($map{$t[0]}); + next if (length($t[1]) != 3); # skip non-SNPs + my $a = \@{$map{$t[0]}}; + my $ref = $a->[2]; + my @u = split('/', $t[1]); + if ($u[1] eq $ref) { + $u[1] = $u[0]; $u[0] = $ref; + } elsif ($u[0] ne $ref) { next; } + my $alt = $u[1]; + my %w; + $w{$u[0]} = 0; $w{$u[1]} = 1; + my @s = (@$a[0,1], $t[0], $ref, $alt, 0, '.', '.', 'GT'); + my $is_tri = 0; + for (@t[11..$#t]) { + if ($_ eq 'NN') { + push(@s, './.'); + } else { + my @a = ($w{substr($_,0,1)}, $w{substr($_,1,1)}); + if (!defined($a[0]) || !defined($a[1])) { + $is_tri = 1; + last; + } + push(@s, "$a[0]/$a[1]"); + } + } + next if ($is_tri); + print join("\t", @s), "\n"; + } + } +} + +sub usage { + die(qq/ +Usage: vcfutils.pl []\n +Command: subsam get a subset of samples + listsam list the samples + fillac fill the allele count field + qstats SNP stats stratified by QUAL + varFilter filtering short variants + filter4vcf filtering VCFs produced by samtools+bcftools + hapmap2vcf convert the hapmap format to VCF + ucscsnp2vcf convert UCSC SNP SQL dump to VCF +\n/); +} diff --git a/bgzf.c b/bgzf.c index a6923da..66d6b02 100644 --- a/bgzf.c +++ b/bgzf.c @@ -177,7 +177,7 @@ BGZF* bgzf_open(const char* __restrict path, const char* __restrict mode) { BGZF* fp = NULL; - if (mode[0] == 'r' || mode[0] == 'R') { /* The reading mode is preferred. */ + if (strchr(mode, 'r') || strchr(mode, 'R')) { /* The reading mode is preferred. */ #ifdef _USE_KNETFILE knetFile *file = knet_open(path, mode); if (file == 0) return 0; @@ -194,14 +194,14 @@ bgzf_open(const char* __restrict path, const char* __restrict mode) if (fd == -1) return 0; fp = open_read(fd); #endif - } else if (mode[0] == 'w' || mode[0] == 'W') { + } else if (strchr(mode, 'w') || strchr(mode, 'W')) { int fd, oflag = O_WRONLY | O_CREAT | O_TRUNC; #ifdef _WIN32 oflag |= O_BINARY; #endif fd = open(path, oflag, 0666); if (fd == -1) return 0; - fp = open_write(fd, strstr(mode, "u")? 1 : 0); + fp = open_write(fd, strchr(mode, 'u')? 1 : 0); } if (fp != NULL) fp->owned_file = 1; return fp; diff --git a/bgzip.c b/bgzip.c index ac2a98e..ebcafa2 100644 --- a/bgzip.c +++ b/bgzip.c @@ -27,22 +27,24 @@ #include #include #include +#include +#include #include "bgzf.h" static const int WINDOW_SIZE = 64 * 1024; static int bgzip_main_usage() { - printf("\n"); - printf("Usage: bgzip [options] [file] ...\n\n"); - printf("Options: -c write on standard output, keep original files unchanged\n"); - printf(" -d decompress\n"); - // printf(" -l list compressed file contents\n"); - printf(" -b INT decompress at virtual file pointer INT\n"); - printf(" -s INT decompress INT bytes in the uncompressed file\n"); - printf(" -h give this help\n"); - printf("\n"); - return 0; + fprintf(stderr, "\n"); + fprintf(stderr, "Usage: bgzip [options] [file] ...\n\n"); + fprintf(stderr, "Options: -c write on standard output, keep original files unchanged\n"); + fprintf(stderr, " -d decompress\n"); + fprintf(stderr, " -f overwrite files without asking\n"); + fprintf(stderr, " -b INT decompress at virtual file pointer INT\n"); + fprintf(stderr, " -s INT decompress INT bytes in the uncompressed file\n"); + fprintf(stderr, " -h give this help\n"); + fprintf(stderr, "\n"); + return 1; } static int write_open(const char *fn, int is_forced) @@ -51,129 +53,154 @@ static int write_open(const char *fn, int is_forced) char c; if (!is_forced) { if ((fd = open(fn, O_WRONLY | O_CREAT | O_TRUNC | O_EXCL, 0666)) < 0 && errno == EEXIST) { - printf("bgzip: %s already exists; do you wish to overwrite (y or n)? ", fn); + fprintf(stderr, "[bgzip] %s already exists; do you wish to overwrite (y or n)? ", fn); scanf("%c", &c); if (c != 'Y' && c != 'y') { - printf("bgzip: not overwritten\n"); + fprintf(stderr, "[bgzip] not overwritten\n"); exit(1); } } } if (fd < 0) { if ((fd = open(fn, O_WRONLY | O_CREAT | O_TRUNC, 0666)) < 0) { - fprintf(stderr, "bgzip: %s: Fail to write\n", fn); + fprintf(stderr, "[bgzip] %s: Fail to write\n", fn); exit(1); } } return fd; } -static -void -fail(BGZF* fp) +static void fail(BGZF* fp) { - printf("Error: %s\n", fp->error); + fprintf(stderr, "Error: %s\n", fp->error); exit(1); } int main(int argc, char **argv) { int c, compress, pstdout, is_forced; - BGZF *rz; + BGZF *fp; void *buffer; long start, end, size; compress = 1; pstdout = 0; start = 0; size = -1; end = -1; is_forced = 0; - while((c = getopt(argc, argv, "cdlhfb:s:")) >= 0){ + while((c = getopt(argc, argv, "cdhfb:s:")) >= 0){ switch(c){ case 'h': return bgzip_main_usage(); case 'd': compress = 0; break; case 'c': pstdout = 1; break; - // case 'l': compress = 2; break; case 'b': start = atol(optarg); break; case 's': size = atol(optarg); break; case 'f': is_forced = 1; break; } } if (size >= 0) end = start + size; - if(end >= 0 && end < start){ - fprintf(stderr, " -- Illegal region: [%ld, %ld] --\n", start, end); + if (end >= 0 && end < start) { + fprintf(stderr, "[bgzip] Illegal region: [%ld, %ld]\n", start, end); return 1; } - if(compress == 1){ - int f_src, f_dst = -1; - if(argc > optind){ - if((f_src = open(argv[optind], O_RDONLY)) < 0){ - fprintf(stderr, " -- Cannot open file: %s --\n", argv[optind]); + if (compress == 1) { + struct stat sbuf; + int f_src = fileno(stdin); + int f_dst = fileno(stdout); + + if ( argc>optind ) + { + if ( stat(argv[optind],&sbuf)<0 ) + { + fprintf(stderr, "[bgzip] %s: %s\n", strerror(errno), argv[optind]); + return 1; + } + + if ((f_src = open(argv[optind], O_RDONLY)) < 0) { + fprintf(stderr, "[bgzip] %s: %s\n", strerror(errno), argv[optind]); return 1; } - if(pstdout){ + + if (pstdout) f_dst = fileno(stdout); - } else { - char *name = malloc(sizeof(strlen(argv[optind]) + 5)); + else + { + char *name = malloc(strlen(argv[optind]) + 5); strcpy(name, argv[optind]); strcat(name, ".gz"); f_dst = write_open(name, is_forced); if (f_dst < 0) return 1; free(name); } - } else if(pstdout){ - f_src = fileno(stdin); - f_dst = fileno(stdout); - } else return bgzip_main_usage(); - rz = bgzf_fdopen(f_dst, "w"); + } + else if (!pstdout && isatty(fileno((FILE *)stdout)) ) + return bgzip_main_usage(); + + fp = bgzf_fdopen(f_dst, "w"); buffer = malloc(WINDOW_SIZE); - while((c = read(f_src, buffer, WINDOW_SIZE)) > 0) { - if (bgzf_write(rz, buffer, c) < 0) { - fail(rz); - } - } - // f_dst will be closed here - if (bgzf_close(rz) < 0) { - fail(rz); - } - if (argc > optind) unlink(argv[optind]); + while ((c = read(f_src, buffer, WINDOW_SIZE)) > 0) + if (bgzf_write(fp, buffer, c) < 0) fail(fp); + // f_dst will be closed here + if (bgzf_close(fp) < 0) fail(fp); + if (argc > optind && !pstdout) unlink(argv[optind]); free(buffer); close(f_src); return 0; } else { - if(argc <= optind) return bgzip_main_usage(); - int f_dst; - if (argc > optind && !pstdout) { - char *name; - if (strstr(argv[optind], ".gz") - argv[optind] != strlen(argv[optind]) - 3) { - printf("bgzip: %s: unknown suffix -- ignored\n", argv[optind]); - return 1; - } - name = strdup(argv[optind]); - name[strlen(name) - 3] = '\0'; - f_dst = write_open(name, is_forced); - free(name); - } else f_dst = fileno(stdout); - rz = bgzf_open(argv[optind], "r"); - if (rz == NULL) { - printf("Could not open file: %s\n", argv[optind]); - return 1; - } - buffer = malloc(WINDOW_SIZE); - if (bgzf_seek(rz, start, SEEK_SET) < 0) { - fail(rz); - } - while(1){ - if(end < 0) c = bgzf_read(rz, buffer, WINDOW_SIZE); - else c = bgzf_read(rz, buffer, (end - start > WINDOW_SIZE)? WINDOW_SIZE:(end - start)); - if(c == 0) break; - if (c < 0) fail(rz); - start += c; - write(f_dst, buffer, c); - if(end >= 0 && start >= end) break; - } - free(buffer); - if (bgzf_close(rz) < 0) { - fail(rz); - } - if (!pstdout) unlink(argv[optind]); + struct stat sbuf; + int f_dst; + + if ( argc>optind ) + { + if ( stat(argv[optind],&sbuf)<0 ) + { + fprintf(stderr, "[bgzip] %s: %s\n", strerror(errno), argv[optind]); + return 1; + } + char *name; + int len = strlen(argv[optind]); + if ( strcmp(argv[optind]+len-3,".gz") ) + { + fprintf(stderr, "[bgzip] %s: unknown suffix -- ignored\n", argv[optind]); + return 1; + } + fp = bgzf_open(argv[optind], "r"); + if (fp == NULL) { + fprintf(stderr, "[bgzip] Could not open file: %s\n", argv[optind]); + return 1; + } + + if (pstdout) { + f_dst = fileno(stdout); + } + else { + name = strdup(argv[optind]); + name[strlen(name) - 3] = '\0'; + f_dst = write_open(name, is_forced); + free(name); + } + } + else if (!pstdout && isatty(fileno((FILE *)stdin)) ) + return bgzip_main_usage(); + else + { + f_dst = fileno(stdout); + fp = bgzf_fdopen(fileno(stdin), "r"); + if (fp == NULL) { + fprintf(stderr, "[bgzip] Could not read from stdin: %s\n", strerror(errno)); + return 1; + } + } + buffer = malloc(WINDOW_SIZE); + if (bgzf_seek(fp, start, SEEK_SET) < 0) fail(fp); + while (1) { + if (end < 0) c = bgzf_read(fp, buffer, WINDOW_SIZE); + else c = bgzf_read(fp, buffer, (end - start > WINDOW_SIZE)? WINDOW_SIZE:(end - start)); + if (c == 0) break; + if (c < 0) fail(fp); + start += c; + write(f_dst, buffer, c); + if (end >= 0 && start >= end) break; + } + free(buffer); + if (bgzf_close(fp) < 0) fail(fp); + if (!pstdout) unlink(argv[optind]); return 0; } } - diff --git a/errmod.c b/errmod.c new file mode 100644 index 0000000..fba9a8d --- /dev/null +++ b/errmod.c @@ -0,0 +1,130 @@ +#include +#include "errmod.h" +#include "ksort.h" +KSORT_INIT_GENERIC(uint16_t) + +typedef struct __errmod_coef_t { + double *fk, *beta, *lhet; +} errmod_coef_t; + +typedef struct { + double fsum[16], bsum[16]; + uint32_t c[16]; +} call_aux_t; + +static errmod_coef_t *cal_coef(double depcorr, double eta) +{ + int k, n, q; + long double sum, sum1; + double *lC; + errmod_coef_t *ec; + + ec = calloc(1, sizeof(errmod_coef_t)); + // initialize ->fk + ec->fk = (double*)calloc(256, sizeof(double)); + ec->fk[0] = 1.0; + for (n = 1; n != 256; ++n) + ec->fk[n] = pow(1. - depcorr, n) * (1.0 - eta) + eta; + // initialize ->coef + ec->beta = (double*)calloc(256 * 256 * 64, sizeof(double)); + lC = (double*)calloc(256 * 256, sizeof(double)); + for (n = 1; n != 256; ++n) { + double lgn = lgamma(n+1); + for (k = 1; k <= n; ++k) + lC[n<<8|k] = lgn - lgamma(k+1) - lgamma(n-k+1); + } + for (q = 1; q != 64; ++q) { + double e = pow(10.0, -q/10.0); + double le = log(e); + double le1 = log(1.0 - e); + for (n = 1; n <= 255; ++n) { + double *beta = ec->beta + (q<<16|n<<8); + sum1 = sum = 0.0; + for (k = n; k >= 0; --k, sum1 = sum) { + sum = sum1 + expl(lC[n<<8|k] + k*le + (n-k)*le1); + beta[k] = -10. / M_LN10 * logl(sum1 / sum); + } + } + } + // initialize ->lhet + ec->lhet = (double*)calloc(256 * 256, sizeof(double)); + for (n = 0; n < 256; ++n) + for (k = 0; k < 256; ++k) + ec->lhet[n<<8|k] = lC[n<<8|k] - M_LN2 * n; + free(lC); + return ec; +} + +errmod_t *errmod_init(float depcorr) +{ + errmod_t *em; + em = (errmod_t*)calloc(1, sizeof(errmod_t)); + em->depcorr = depcorr; + em->coef = cal_coef(depcorr, 0.03); + return em; +} + +void errmod_destroy(errmod_t *em) +{ + if (em == 0) return; + free(em->coef->lhet); free(em->coef->fk); free(em->coef->beta); + free(em->coef); free(em); +} +// qual:6, strand:1, base:4 +int errmod_cal(const errmod_t *em, int n, int m, uint16_t *bases, float *q) +{ + call_aux_t aux; + int i, j, k, w[32]; + + if (m > m) return -1; + memset(q, 0, m * m * sizeof(float)); + if (n == 0) return 0; + // calculate aux.esum and aux.fsum + if (n > 255) { // then sample 255 bases + ks_shuffle(uint16_t, n, bases); + n = 255; + } + ks_introsort(uint16_t, n, bases); + memset(w, 0, 32 * sizeof(int)); + memset(&aux, 0, sizeof(call_aux_t)); + for (j = n - 1; j >= 0; --j) { // calculate esum and fsum + uint16_t b = bases[j]; + int q = b>>5 < 4? 4 : b>>5; + if (q > 63) q = 63; + k = b&0x1f; + aux.fsum[k&0xf] += em->coef->fk[w[k]]; + aux.bsum[k&0xf] += em->coef->fk[w[k]] * em->coef->beta[q<<16|n<<8|aux.c[k&0xf]]; + ++aux.c[k&0xf]; + ++w[k]; + } + // generate likelihood + for (j = 0; j != m; ++j) { + float tmp1, tmp3; + int tmp2, bar_e; + // homozygous + for (k = 0, tmp1 = tmp3 = 0.0, tmp2 = 0; k != m; ++k) { + if (k == j) continue; + tmp1 += aux.bsum[k]; tmp2 += aux.c[k]; tmp3 += aux.fsum[k]; + } + if (tmp2) { + bar_e = (int)(tmp1 / tmp3 + 0.499); + if (bar_e > 63) bar_e = 63; + q[j*m+j] = tmp1; + } + // heterozygous + for (k = j + 1; k < m; ++k) { + int cjk = aux.c[j] + aux.c[k]; + for (i = 0, tmp2 = 0, tmp1 = tmp3 = 0.0; i < m; ++i) { + if (i == j || i == k) continue; + tmp1 += aux.bsum[i]; tmp2 += aux.c[i]; tmp3 += aux.fsum[i]; + } + if (tmp2) { + bar_e = (int)(tmp1 / tmp3 + 0.499); + if (bar_e > 63) bar_e = 63; + q[j*m+k] = q[k*m+j] = -4.343 * em->coef->lhet[cjk<<8|aux.c[k]] + tmp1; + } else q[j*m+k] = q[k*m+j] = -4.343 * em->coef->lhet[cjk<<8|aux.c[k]]; // all the bases are either j or k + } + for (k = 0; k != m; ++k) if (q[j*m+k] < 0.0) q[j*m+k] = 0.0; + } + return 0; +} diff --git a/errmod.h b/errmod.h new file mode 100644 index 0000000..e3e9a90 --- /dev/null +++ b/errmod.h @@ -0,0 +1,17 @@ +#ifndef ERRMOD_H +#define ERRMOD_H + +#include + +struct __errmod_coef_t; + +typedef struct { + double depcorr; + struct __errmod_coef_t *coef; +} errmod_t; + +errmod_t *errmod_init(float depcorr); +void errmod_destroy(errmod_t *em); +int errmod_cal(const errmod_t *em, int n, int m, uint16_t *bases, float *q); + +#endif diff --git a/examples/Makefile b/examples/Makefile index 8f0386f..ec976ae 100644 --- a/examples/Makefile +++ b/examples/Makefile @@ -1,4 +1,5 @@ -all:../libbam.a ../samtools ex1.glf ex1.pileup.gz ex1.bam.bai ex1f-rmduppe.bam ex1f-rmdupse.bam ex1.glfview.gz calDepth +all:../libbam.a ../samtools ../bcftools/bcftools \ + ex1.glf ex1.pileup.gz ex1.bam.bai ex1f-rmduppe.bam ex1f-rmdupse.bam ex1.glfview.gz ex1.bcf calDepth @echo; echo \# You can now launch the viewer with: \'samtools tview ex1.bam ex1.fa\'; echo; ex1.fa.fai:ex1.fa @@ -18,7 +19,7 @@ ex1a.bam:ex1.bam ex1b.bam:ex1.bam ../samtools view -h ex1.bam | awk 'BEGIN{FS=OFS="\t"}{if(/^@/)print;else{$$1=$$1"b";print}}' | ../samtools view -bS - > $@ ex1f.rg: - (echo "@RG ID:ex1 LB:ex1"; echo "@RG ID:ex1a LB:ex1"; echo "@RG ID:ex1b LB:ex1b") > $@ + (echo "@RG ID:ex1 LB:ex1 SM:ex1"; echo "@RG ID:ex1a LB:ex1 SM:ex1"; echo "@RG ID:ex1b LB:ex1b SM:ex1b") > $@ ex1f.bam:ex1.bam ex1a.bam ex1b.bam ex1f.rg ../samtools merge -rh ex1f.rg $@ ex1.bam ex1a.bam ex1b.bam ex1f-rmduppe.bam:ex1f.bam @@ -26,6 +27,12 @@ ex1f-rmduppe.bam:ex1f.bam ex1f-rmdupse.bam:ex1f.bam ../samtools rmdup -S ex1f.bam $@ +ex1.bcf:ex1.bam ex1.fa.fai + ../samtools mpileup -gf ex1.fa ex1.bam > $@ + +../bcftools/bcftools: + (cd ../bcftools; make bcftools) + ../samtools: (cd ..; make samtools) @@ -36,4 +43,8 @@ calDepth:../libbam.a calDepth.c gcc -g -Wall -O2 -I.. calDepth.c -o $@ -lm -lz -L.. -lbam clean: - rm -fr *.bam *.bai *.glf* *.fai *.pileup* *~ calDepth *.dSYM ex1*.rg \ No newline at end of file + rm -fr *.bam *.bai *.glf* *.fai *.pileup* *~ calDepth *.dSYM ex1*.rg ex1.bcf + +# ../samtools pileup ex1.bam|perl -ape '$_=$F[4];s/(\d+)(??{".{$1}"})|\^.//g;@_=(tr/A-Z//,tr/a-z//);$_=join("\t",@F[0,1],@_)."\n"' + +# ../samtools pileup -cf ex1.fa ex1.bam|perl -ape '$_=$F[8];s/\^.//g;s/(\d+)(??{".{$1}"})|\^.//g;@_=(tr/A-Za-z//,tr/,.//);$_=join("\t",@F[0,1],@_)."\n"' \ No newline at end of file diff --git a/examples/toy.fa b/examples/toy.fa index 38312c1..afe990a 100644 --- a/examples/toy.fa +++ b/examples/toy.fa @@ -1,2 +1,4 @@ >ref AGCATGTTAGATAAGATAGCTGTGCTAGTAGGCAGTCAGCGCCAT +>ref2 +aggttttataaaacaattaagtctacagagcaactacgcg diff --git a/examples/toy.sam b/examples/toy.sam index baf7388..1aff220 100644 --- a/examples/toy.sam +++ b/examples/toy.sam @@ -1,7 +1,14 @@ @SQ SN:ref LN:45 -r001 163 ref 7 30 8M2I4M1D3M = 37 39 TTAGATAAAGGATACTG * -r002 0 ref 9 30 1S2I6M1P1I4M2I * 0 0 AAAAGATAAGGATAAA * +@SQ SN:ref2 LN:40 +r001 163 ref 7 30 8M4I4M1D3M = 37 39 TTAGATAAAGAGGATACTG * +r002 0 ref 9 30 1S2I6M1P1I1P1I4M2I * 0 0 AAAAGATAAGGGATAAA * r003 0 ref 9 30 5H6M * 0 0 AGCTAA * r004 0 ref 16 30 6M14N1I5M * 0 0 ATAGCTCTCAGC * r003 16 ref 29 30 6H5M * 0 0 TAGGC * -r001 83 ref 37 30 9M = 7 -39 CAGCGCCAT * \ No newline at end of file +r001 83 ref 37 30 9M = 7 -39 CAGCGCCAT * +x1 0 ref2 1 30 20M * 0 0 aggttttataaaacaaataa ???????????????????? +x2 0 ref2 2 30 21M * 0 0 ggttttataaaacaaataatt ????????????????????? +x3 0 ref2 6 30 9M4I13M * 0 0 ttataaaacAAATaattaagtctaca ?????????????????????????? +x4 0 ref2 10 30 25M * 0 0 CaaaTaattaagtctacagagcaac ????????????????????????? +x5 0 ref2 12 30 24M * 0 0 aaTaattaagtctacagagcaact ???????????????????????? +x6 0 ref2 14 30 23M * 0 0 Taattaagtctacagagcaacta ??????????????????????? diff --git a/kaln.c b/kaln.c index 9fa40d0..6807791 100644 --- a/kaln.c +++ b/kaln.c @@ -27,6 +27,7 @@ #include #include #include +#include #include "kaln.h" #define FROM_M 0 @@ -72,8 +73,8 @@ int aln_sm_blast[] = { -2, -2, -2, -2, -2 }; -ka_param_t ka_param_blast = { 5, 2, 2, aln_sm_blast, 5, 50 }; -ka_param_t ka_param_aa2aa = { 10, 2, 2, aln_sm_blosum62, 22, 50 }; +ka_param_t ka_param_blast = { 5, 2, 5, 2, aln_sm_blast, 5, 50 }; +ka_param_t ka_param_aa2aa = { 10, 2, 10, 2, aln_sm_blosum62, 22, 50 }; static uint32_t *ka_path2cigar32(const path_t *path, int path_len, int *n_cigar) { @@ -141,13 +142,13 @@ static uint32_t *ka_path2cigar32(const path_t *path, int path_len, int *n_cigar) } #define set_end_I(II, cur, p) \ { \ - if (gap_end >= 0) { \ - if ((p)->M - gap_open > (p)->I) { \ + if (gap_end_ext >= 0) { \ + if ((p)->M - gap_end_open > (p)->I) { \ (cur)->It = FROM_M; \ - (II) = (p)->M - gap_open - gap_end; \ + (II) = (p)->M - gap_end_open - gap_end_ext; \ } else { \ (cur)->It = FROM_I; \ - (II) = (p)->I - gap_end; \ + (II) = (p)->I - gap_end_ext; \ } \ } else set_I(II, cur, p); \ } @@ -163,13 +164,13 @@ static uint32_t *ka_path2cigar32(const path_t *path, int path_len, int *n_cigar) } #define set_end_D(DD, cur, p) \ { \ - if (gap_end >= 0) { \ - if ((p)->M - gap_open > (p)->D) { \ + if (gap_end_ext >= 0) { \ + if ((p)->M - gap_end_open > (p)->D) { \ (cur)->Dt = FROM_M; \ - (DD) = (p)->M - gap_open - gap_end; \ + (DD) = (p)->M - gap_end_open - gap_end_ext; \ } else { \ (cur)->Dt = FROM_D; \ - (DD) = (p)->D - gap_end; \ + (DD) = (p)->D - gap_end_ext; \ } \ } else set_D(DD, cur, p); \ } @@ -195,13 +196,14 @@ uint32_t *ka_global_core(uint8_t *seq1, int len1, uint8_t *seq2, int len2, const uint8_t type, ctype; uint32_t *cigar = 0; - int gap_open, gap_ext, gap_end, b; + int gap_open, gap_ext, gap_end_open, gap_end_ext, b; int *score_matrix, N_MATRIX_ROW; /* initialize some align-related parameters. just for compatibility */ gap_open = ap->gap_open; gap_ext = ap->gap_ext; - gap_end = ap->gap_end; + gap_end_open = ap->gap_end_open; + gap_end_ext = ap->gap_end_ext; b = ap->band_width; score_matrix = ap->matrix; N_MATRIX_ROW = ap->row; @@ -368,3 +370,212 @@ uint32_t *ka_global_core(uint8_t *seq1, int len1, uint8_t *seq2, int len2, const return cigar; } + +/***************************************** + * Probabilistic banded glocal alignment * + *****************************************/ + +static float g_qual2prob[256]; + +#define EI .25 +#define EM .3333333333333 +#define set_u(u, b, i, k) { int x=(i)-(b); x=x>0?x:0; (u)=((k)-x+1)*3; } + +ka_probpar_t ka_probpar_def = { 0.001, 0.1, 10 }; + +/* + The topology of the profile HMM: + + /\ /\ /\ /\ + I[1] I[k-1] I[k] I[L] + ^ \ \ ^ \ ^ \ \ ^ + | \ \ | \ | \ \ | + M[0] M[1] -> ... -> M[k-1] -> M[k] -> ... -> M[L] M[L+1] + \ \/ \/ \/ / + \ /\ /\ /\ / + -> D[k-1] -> D[k] -> + + M[0] points to every {M,I}[k] and every {M,I}[k] points M[L+1]. + + On input, _ref is the reference sequence and _query is the query + sequence. Both are sequences of 0/1/2/3/4 where 4 stands for an + ambiguous residue. iqual is the base quality. c sets the gap open + probability, gap extension probability and band width. + + On output, state and q are arrays of length l_query. The higher 30 + bits give the reference position the query base is matched to and the + lower two bits can be 0 (an alignment match) or 1 (an + insertion). q[i] gives the phred scaled posterior probability of + state[i] being wrong. + */ +int ka_prob_glocal(const uint8_t *_ref, int l_ref, const uint8_t *_query, int l_query, const uint8_t *iqual, + const ka_probpar_t *c, int *state, uint8_t *q) +{ + double **f, **b, *s, m[9], sI, sM, bI, bM, pb; + float *qual, *_qual; + const uint8_t *ref, *query; + int bw, bw2, i, k, is_diff = 0; + + /*** initialization ***/ + ref = _ref - 1; query = _query - 1; // change to 1-based coordinate + bw = l_ref > l_query? l_ref : l_query; + if (bw > c->bw) bw = c->bw; + if (bw < abs(l_ref - l_query)) bw = abs(l_ref - l_query); + bw2 = bw * 2 + 1; + // allocate the forward and backward matrices f[][] and b[][] and the scaling array s[] + f = calloc(l_query+1, sizeof(void*)); + b = calloc(l_query+1, sizeof(void*)); + for (i = 0; i <= l_query; ++i) { + f[i] = calloc(bw2 * 3 + 6, sizeof(double)); // FIXME: this is over-allocated for very short seqs + b[i] = calloc(bw2 * 3 + 6, sizeof(double)); + } + s = calloc(l_query+2, sizeof(double)); // s[] is the scaling factor to avoid underflow + // initialize qual + _qual = calloc(l_query, sizeof(float)); + if (g_qual2prob[0] == 0) + for (i = 0; i < 256; ++i) + g_qual2prob[i] = pow(10, -i/10.); + for (i = 0; i < l_query; ++i) _qual[i] = g_qual2prob[iqual? iqual[i] : 30]; + qual = _qual - 1; + // initialize transition probability + sM = sI = 1. / (2 * l_query + 2); // the value here seems not to affect results; FIXME: need proof + m[0*3+0] = (1 - c->d - c->d) * (1 - sM); m[0*3+1] = m[0*3+2] = c->d * (1 - sM); + m[1*3+0] = (1 - c->e) * (1 - sI); m[1*3+1] = c->e * (1 - sI); m[1*3+2] = 0.; + m[2*3+0] = 1 - c->e; m[2*3+1] = 0.; m[2*3+2] = c->e; + bM = (1 - c->d) / l_query; bI = c->d / l_query; // (bM+bI)*l_query==1 + /*** forward ***/ + // f[0] + set_u(k, bw, 0, 0); + f[0][k] = s[0] = 1.; + { // f[1] + double *fi = f[1], sum; + int beg = 1, end = l_ref < bw + 1? l_ref : bw + 1, _beg, _end; + for (k = beg, sum = 0.; k <= end; ++k) { + int u; + double e = (ref[k] > 3 || query[1] > 3)? 1. : ref[k] == query[1]? 1. - qual[1] : qual[1] * EM; + set_u(u, bw, 1, k); + fi[u+0] = e * bM; fi[u+1] = EI * bI; + sum += fi[u] + fi[u+1]; + } + // rescale + s[1] = sum; + set_u(_beg, bw, 1, beg); set_u(_end, bw, 1, end); _end += 2; + for (k = _beg; k <= _end; ++k) fi[k] /= sum; + } + // f[2..l_query] + for (i = 2; i <= l_query; ++i) { + double *fi = f[i], *fi1 = f[i-1], sum, qli = qual[i]; + int beg = 1, end = l_ref, x, _beg, _end; + uint8_t qyi = query[i]; + x = i - bw; beg = beg > x? beg : x; // band start + x = i + bw; end = end < x? end : x; // band end + for (k = beg, sum = 0.; k <= end; ++k) { + int u, v11, v01, v10; + double e; + e = (ref[k] > 3 || qyi > 3)? 1. : ref[k] == qyi? 1. - qli : qli * EM; + set_u(u, bw, i, k); set_u(v11, bw, i-1, k-1); set_u(v10, bw, i-1, k); set_u(v01, bw, i, k-1); + fi[u+0] = e * (m[0] * fi1[v11+0] + m[3] * fi1[v11+1] + m[6] * fi1[v11+2]); + fi[u+1] = EI * (m[1] * fi1[v10+0] + m[4] * fi1[v10+1]); + fi[u+2] = m[2] * fi[v01+0] + m[8] * fi[v01+2]; + sum += fi[u] + fi[u+1] + fi[u+2]; +// fprintf(stderr, "F (%d,%d;%d): %lg,%lg,%lg\n", i, k, u, fi[u], fi[u+1], fi[u+2]); // DEBUG + } + // rescale + s[i] = sum; + set_u(_beg, bw, i, beg); set_u(_end, bw, i, end); _end += 2; + for (k = _beg, sum = 1./sum; k <= _end; ++k) fi[k] *= sum; + } + { // f[l_query+1] + double sum; + for (k = 1, sum = 0.; k <= l_ref; ++k) { + int u; + set_u(u, bw, l_query, k); + if (u < 3 || u >= bw2*3+3) continue; + sum += f[l_query][u+0] * sM + f[l_query][u+1] * sI; + } + s[l_query+1] = sum; // the last scaling factor + } + /*** backward ***/ + // b[l_query] (b[l_query+1][0]=1 and thus \tilde{b}[][]=1/s[l_query+1]; this is where s[l_query+1] comes from) + for (k = 1; k <= l_ref; ++k) { + int u; + double *bi = b[l_query]; + set_u(u, bw, l_query, k); + if (u < 3 || u >= bw2*3+3) continue; + bi[u+0] = sM / s[l_query] / s[l_query+1]; bi[u+1] = sI / s[l_query] / s[l_query+1]; + } + // b[l_query-1..1] + for (i = l_query - 1; i >= 1; --i) { + int beg = 1, end = l_ref, x, _beg, _end; + double *bi = b[i], *bi1 = b[i+1], y = (i > 1), qli1 = qual[i+1]; + uint8_t qyi1 = query[i+1]; + x = i - bw; beg = beg > x? beg : x; + x = i + bw; end = end < x? end : x; + for (k = end; k >= beg; --k) { + int u, v11, v01, v10; + double e; + set_u(u, bw, i, k); set_u(v11, bw, i+1, k+1); set_u(v10, bw, i+1, k); set_u(v01, bw, i, k+1); + e = (k >= l_ref? 0 : (ref[k+1] > 3 || qyi1 > 3)? 1. : ref[k+1] == qyi1? 1. - qli1 : qli1 * EM) * bi1[v11]; + bi[u+0] = e * m[0] + EI * m[1] * bi1[v10+1] + m[2] * bi[v01+2]; // bi1[v11] has been foled into e. + bi[u+1] = e * m[3] + EI * m[4] * bi1[v10+1]; + bi[u+2] = (e * m[6] + m[8] * bi[v01+2]) * y; +// fprintf(stderr, "B (%d,%d;%d): %lg,%lg,%lg\n", i, k, u, bi[u], bi[u+1], bi[u+2]); // DEBUG + } + // rescale + set_u(_beg, bw, i, beg); set_u(_end, bw, i, end); _end += 2; + for (k = _beg, y = 1./s[i]; k <= _end; ++k) bi[k] *= y; + } + { // b[0] + int beg = 1, end = l_ref < bw + 1? l_ref : bw + 1; + double sum = 0.; + for (k = end; k >= beg; --k) { + int u; + double e = (ref[k] > 3 || query[1] > 3)? 1. : ref[k] == query[1]? 1. - qual[1] : qual[1] * EM; + set_u(u, bw, 1, k); + if (u < 3 || u >= bw2*3+3) continue; + sum += e * b[1][u+0] * bM + EI * b[1][u+1] * bI; + } + set_u(k, bw, 0, 0); + pb = b[0][k] = sum / s[0]; // if everything works as is expected, pb == 1.0 + } + is_diff = fabs(pb - 1.) > 1e-7? 1 : 0; + /*** MAP ***/ + for (i = 1; i <= l_query; ++i) { + double sum = 0., *fi = f[i], *bi = b[i], max = 0.; + int beg = 1, end = l_ref, x, max_k = -1; + x = i - bw; beg = beg > x? beg : x; + x = i + bw; end = end < x? end : x; + for (k = beg; k <= end; ++k) { + int u; + double z; + set_u(u, bw, i, k); + z = fi[u+0] * bi[u+0]; if (z > max) max = z, max_k = (k-1)<<2 | 0; sum += z; + z = fi[u+1] * bi[u+1]; if (z > max) max = z, max_k = (k-1)<<2 | 1; sum += z; + } + max /= sum; sum *= s[i]; // if everything works as is expected, sum == 1.0 + if (state) state[i-1] = max_k; + if (q) k = (int)(-4.343 * log(1. - max) + .499), q[i-1] = k > 100? 99 : k; +#ifdef _MAIN + fprintf(stderr, "(%.10lg,%.10lg) (%d,%d:%d)~%lg\n", pb, sum, i-1, max_k>>2, max_k&3, max); // DEBUG +#endif + } + /*** free ***/ + for (i = 0; i <= l_query; ++i) { + free(f[i]); free(b[i]); + } + free(f); free(b); free(s); free(_qual); + return 0; +} + +#ifdef _MAIN +int main() +{ + int l_ref = 5, l_query = 4; + uint8_t *ref = (uint8_t*)"\0\1\3\3\1"; + uint8_t *query = (uint8_t*)"\0\3\3\1"; +// uint8_t *query = (uint8_t*)"\1\3\3\1"; + static uint8_t qual[4] = {20, 20, 20, 20}; + ka_prob_glocal(ref, l_ref, query, l_query, qual, &ka_probpar_def, 0, 0); + return 0; +} +#endif diff --git a/kaln.h b/kaln.h index b04d8cc..b24597c 100644 --- a/kaln.h +++ b/kaln.h @@ -33,23 +33,33 @@ typedef struct { int gap_open; int gap_ext; - int gap_end; + int gap_end_open; + int gap_end_ext; int *matrix; int row; int band_width; } ka_param_t; +typedef struct { + float d, e; + int bw; +} ka_probpar_t; + #ifdef __cplusplus extern "C" { #endif - uint32_t *ka_global_core(uint8_t *seq1, int len1, uint8_t *seq2, int len2, const ka_param_t *ap, int *_score, int *n_cigar); + uint32_t *ka_global_core(uint8_t *seq1, int len1, uint8_t *seq2, int len2, const ka_param_t *ap, + int *_score, int *n_cigar); + int ka_prob_glocal(const uint8_t *_ref, int l_ref, const uint8_t *_query, int l_query, const uint8_t *iqual, + const ka_probpar_t *c, int *state, uint8_t *q); #ifdef __cplusplus } #endif -extern ka_param_t ka_param_blast; /* = { 5, 2, 2, aln_sm_blast, 5, 50 }; */ +extern ka_param_t ka_param_blast; /* = { 5, 2, 5, 2, aln_sm_blast, 5, 50 }; */ +extern ka_probpar_t ka_probpar_def; /* { 0.0001, 0.1, 10 } */ #endif diff --git a/ksort.h b/ksort.h index 16a03fd..fa850ab 100644 --- a/ksort.h +++ b/ksort.h @@ -250,6 +250,15 @@ typedef struct { if (hh <= k) low = ll; \ if (hh >= k) high = hh - 1; \ } \ + } \ + void ks_shuffle_##name(size_t n, type_t a[]) \ + { \ + int i, j; \ + for (i = n; i > 1; --i) { \ + type_t tmp; \ + j = (int)(drand48() * i); \ + tmp = a[j]; a[j] = a[i-1]; a[i-1] = tmp; \ + } \ } #define ks_mergesort(name, n, a, t) ks_mergesort_##name(n, a, t) @@ -259,6 +268,7 @@ typedef struct { #define ks_heapmake(name, n, a) ks_heapmake_##name(n, a) #define ks_heapadjust(name, i, n, a) ks_heapadjust_##name(i, n, a) #define ks_ksmall(name, n, a, k) ks_ksmall_##name(n, a, k) +#define ks_shuffle(name, n, a) ks_shuffle_##name(n, a) #define ks_lt_generic(a, b) ((a) < (b)) #define ks_lt_str(a, b) (strcmp((a), (b)) < 0) diff --git a/kstring.c b/kstring.c index e0203fa..43d524c 100644 --- a/kstring.c +++ b/kstring.c @@ -24,6 +24,24 @@ int ksprintf(kstring_t *s, const char *fmt, ...) return l; } +char *kstrtok(const char *str, const char *sep, ks_tokaux_t *aux) +{ + const char *p, *start; + if (sep) { // set up the table + if (str == 0 && (aux->tab[0]&1)) return 0; // no need to set up if we have finished + aux->tab[0] = aux->tab[1] = aux->tab[2] = aux->tab[3] = 0; + for (p = sep; *p; ++p) + aux->tab[*p/64] |= 1ull<<(*p%64); + } + if (str) aux->p = str - 1, aux->tab[0] &= ~1ull; + else if (aux->tab[0]&1) return 0; + for (p = start = aux->p + 1; *p; ++p) + if (aux->tab[*p/64]>>(*p%64)&1) break; + aux->p = p; // end of token + if (*p == 0) aux->tab[0] |= 1; // no more tokens + return (char*)start; +} + // s MUST BE a null terminated string; l = strlen(s) int ksplit_core(char *s, int delimiter, int *_max, int **_offsets) { @@ -66,11 +84,13 @@ int ksplit_core(char *s, int delimiter, int *_max, int **_offsets) * Boyer-Moore search * **********************/ +typedef unsigned char ubyte_t; + // reference: http://www-igm.univ-mlv.fr/~lecroq/string/node14.html -int *ksBM_prep(const uint8_t *pat, int m) +static int *ksBM_prep(const ubyte_t *pat, int m) { int i, *suff, *prep, *bmGs, *bmBc; - prep = calloc(m + 256, 1); + prep = calloc(m + 256, sizeof(int)); bmGs = prep; bmBc = prep + m; { // preBmBc() for (i = 0; i < 256; ++i) bmBc[i] = m; @@ -107,39 +127,49 @@ int *ksBM_prep(const uint8_t *pat, int m) return prep; } -int *ksBM_search(const uint8_t *str, int n, const uint8_t *pat, int m, int *_prep, int *n_matches) +void *kmemmem(const void *_str, int n, const void *_pat, int m, int **_prep) { - int i, j, *prep, *bmGs, *bmBc; - int *matches = 0, mm = 0, nm = 0; - prep = _prep? _prep : ksBM_prep(pat, m); + int i, j, *prep = 0, *bmGs, *bmBc; + const ubyte_t *str, *pat; + str = (const ubyte_t*)_str; pat = (const ubyte_t*)_pat; + prep = (_prep == 0 || *_prep == 0)? ksBM_prep(pat, m) : *_prep; + if (_prep && *_prep == 0) *_prep = prep; bmGs = prep; bmBc = prep + m; j = 0; while (j <= n - m) { for (i = m - 1; i >= 0 && pat[i] == str[i+j]; --i); - if (i < 0) { - if (nm == mm) { - mm = mm? mm<<1 : 1; - matches = realloc(matches, mm * sizeof(int)); - } - matches[nm++] = j; - j += bmGs[0]; - } else { + if (i >= 0) { int max = bmBc[str[i+j]] - m + 1 + i; if (max < bmGs[i]) max = bmGs[i]; j += max; - } + } else return (void*)(str + j); } - *n_matches = nm; if (_prep == 0) free(prep); - return matches; + return 0; } +char *kstrstr(const char *str, const char *pat, int **_prep) +{ + return (char*)kmemmem(str, strlen(str), pat, strlen(pat), _prep); +} + +char *kstrnstr(const char *str, const char *pat, int n, int **_prep) +{ + return (char*)kmemmem(str, n, pat, strlen(pat), _prep); +} + +/*********************** + * The main() function * + ***********************/ + #ifdef KSTRING_MAIN #include int main() { kstring_t *s; int *fields, n, i; + ks_tokaux_t aux; + char *p; s = (kstring_t*)calloc(1, sizeof(kstring_t)); // test ksprintf() ksprintf(s, " abcdefg: %d ", 100); @@ -148,17 +178,26 @@ int main() fields = ksplit(s, 0, &n); for (i = 0; i < n; ++i) printf("field[%d] = '%s'\n", i, s->s + fields[i]); - free(s); + // test kstrtok() + s->l = 0; + for (p = kstrtok("ab:cde:fg/hij::k", ":/", &aux); p; p = kstrtok(0, 0, &aux)) { + kputsn(p, aux.p - p, s); + kputc('\n', s); + } + printf("%s", s->s); + // free + free(s->s); free(s); free(fields); { - static char *str = "abcdefgcdg"; + static char *str = "abcdefgcdgcagtcakcdcd"; static char *pat = "cd"; - int n, *matches; - matches = ksBM_search(str, strlen(str), pat, strlen(pat), 0, &n); - printf("%d: \n", n); - for (i = 0; i < n; ++i) - printf("- %d\n", matches[i]); - free(matches); + char *ret, *s = str; + int *prep = 0; + while ((ret = kstrstr(s, pat, &prep)) != 0) { + printf("match: %s\n", ret); + s = ret + prep[0]; + } + free(prep); } return 0; } diff --git a/kstring.h b/kstring.h index 925117a..c46a62b 100644 --- a/kstring.h +++ b/kstring.h @@ -17,17 +17,31 @@ typedef struct __kstring_t { } kstring_t; #endif -int ksprintf(kstring_t *s, const char *fmt, ...); -int ksplit_core(char *s, int delimiter, int *_max, int **_offsets); +typedef struct { + uint64_t tab[4]; + const char *p; // end of the current token +} ks_tokaux_t; -// calculate the auxiliary array, allocated by calloc() -int *ksBM_prep(const uint8_t *pat, int m); +#ifdef __cplusplus +extern "C" { +#endif + + int ksprintf(kstring_t *s, const char *fmt, ...); + int ksplit_core(char *s, int delimiter, int *_max, int **_offsets); + char *kstrstr(const char *str, const char *pat, int **_prep); + char *kstrnstr(const char *str, const char *pat, int n, int **_prep); + void *kmemmem(const void *_str, int n, const void *_pat, int m, int **_prep); -/* Search pat in str and returned the list of matches. The size of the - * list is returned as n_matches. _prep is the array returned by - * ksBM_prep(). If it is a NULL pointer, ksBM_prep() will be called. */ -int *ksBM_search(const uint8_t *str, int n, const uint8_t *pat, int m, int *_prep, int *n_matches); + /* kstrtok() is similar to strtok_r() except that str is not + * modified and both str and sep can be NULL. For efficiency, it is + * actually recommended to set both to NULL in the subsequent calls + * if sep is not changed. */ + char *kstrtok(const char *str, const char *sep, ks_tokaux_t *aux); +#ifdef __cplusplus +} +#endif + static inline int kputsn(const char *p, int l, kstring_t *s) { if (s->l + l + 1 >= s->m) { @@ -35,7 +49,7 @@ static inline int kputsn(const char *p, int l, kstring_t *s) kroundup32(s->m); s->s = (char*)realloc(s->s, s->m); } - strncpy(s->s + s->l, p, l); + memcpy(s->s + s->l, p, l); s->l += l; s->s[s->l] = 0; return l; diff --git a/misc/HmmGlocal.java b/misc/HmmGlocal.java new file mode 100644 index 0000000..9e93b13 --- /dev/null +++ b/misc/HmmGlocal.java @@ -0,0 +1,178 @@ +import java.io.*; +import java.lang.*; + +public class HmmGlocal +{ + private double[] qual2prob; + private double cd, ce; // gap open probility [1e-3], gap extension probability [0.1] + private int cb; // band width [7] + + public HmmGlocal(final double d, final double e, final int b) { + cd = d; ce = e; cb = b; + qual2prob = new double[256]; + for (int i = 0; i < 256; ++i) + qual2prob[i] = Math.pow(10, -i/10.); + } + private static int set_u(final int b, final int i, final int k) { + int x = i - b; + x = x > 0? x : 0; + return (k + 1 - x) * 3; + } + public int hmm_glocal(final byte[] _ref, final byte[] _query, final byte[] _iqual, int[] state, byte[] q) { + int i, k; + /*** initialization ***/ + // change coordinates + int l_ref = _ref.length; + byte[] ref = new byte[l_ref+1]; + for (i = 0; i < l_ref; ++i) ref[i+1] = _ref[i]; // FIXME: this is silly... + int l_query = _query.length; + byte[] query = new byte[l_query+1]; + double[] qual = new double[l_query+1]; + for (i = 0; i < l_query; ++i) { + query[i+1] = _query[i]; + qual[i+1] = qual2prob[_iqual[i]]; + } + // set band width + int bw2, bw = l_ref > l_query? l_ref : l_query; + if (bw > cb) bw = cb; + if (bw < Math.abs(l_ref - l_query)) bw = Math.abs(l_ref - l_query); + bw2 = bw * 2 + 1; + // allocate the forward and backward matrices f[][] and b[][] and the scaling array s[] + double[][] f = new double[l_query+1][bw2*3 + 6]; + double[][] b = new double[l_query+1][bw2*3 + 6]; + double[] s = new double[l_query+2]; + // initialize transition probabilities + double sM, sI, bM, bI; + sM = sI = 1. / (2 * l_query + 2); + bM = (1 - cd) / l_query; bI = cd / l_query; // (bM+bI)*l_query==1 + double[] m = new double[9]; + m[0*3+0] = (1 - cd - cd) * (1 - sM); m[0*3+1] = m[0*3+2] = cd * (1 - sM); + m[1*3+0] = (1 - ce) * (1 - sI); m[1*3+1] = ce * (1 - sI); m[1*3+2] = 0.; + m[2*3+0] = 1 - ce; m[2*3+1] = 0.; m[2*3+2] = ce; + /*** forward ***/ + // f[0] + f[0][set_u(bw, 0, 0)] = s[0] = 1.; + { // f[1] + double[] fi = f[1]; + double sum; + int beg = 1, end = l_ref < bw + 1? l_ref : bw + 1, _beg, _end; + for (k = beg, sum = 0.; k <= end; ++k) { + int u; + double e = (ref[k] > 3 || query[1] > 3)? 1. : ref[k] == query[1]? 1. - qual[1] : qual[1] / 3.; + u = set_u(bw, 1, k); + fi[u+0] = e * bM; fi[u+1] = .25 * bI; + sum += fi[u] + fi[u+1]; + } + // rescale + s[1] = sum; + _beg = set_u(bw, 1, beg); _end = set_u(bw, 1, end); _end += 2; + for (k = _beg; k <= _end; ++k) fi[k] /= sum; + } + // f[2..l_query] + for (i = 2; i <= l_query; ++i) { + double[] fi = f[i], fi1 = f[i-1]; + double sum, qli = qual[i]; + int beg = 1, end = l_ref, x, _beg, _end; + byte qyi = query[i]; + x = i - bw; beg = beg > x? beg : x; // band start + x = i + bw; end = end < x? end : x; // band end + for (k = beg, sum = 0.; k <= end; ++k) { + int u, v11, v01, v10; + double e; + e = (ref[k] > 3 || qyi > 3)? 1. : ref[k] == qyi? 1. - qli : qli / 3.; + u = set_u(bw, i, k); v11 = set_u(bw, i-1, k-1); v10 = set_u(bw, i-1, k); v01 = set_u(bw, i, k-1); + fi[u+0] = e * (m[0] * fi1[v11+0] + m[3] * fi1[v11+1] + m[6] * fi1[v11+2]); + fi[u+1] = .25 * (m[1] * fi1[v10+0] + m[4] * fi1[v10+1]); + fi[u+2] = m[2] * fi[v01+0] + m[8] * fi[v01+2]; + sum += fi[u] + fi[u+1] + fi[u+2]; + //System.out.println("("+i+","+k+";"+u+"): "+fi[u]+","+fi[u+1]+","+fi[u+2]); + } + // rescale + s[i] = sum; + _beg = set_u(bw, i, beg); _end = set_u(bw, i, end); _end += 2; + for (k = _beg, sum = 1./sum; k <= _end; ++k) fi[k] *= sum; + } + { // f[l_query+1] + double sum; + for (k = 1, sum = 0.; k <= l_ref; ++k) { + int u = set_u(bw, l_query, k); + if (u < 3 || u >= bw2*3+3) continue; + sum += f[l_query][u+0] * sM + f[l_query][u+1] * sI; + } + s[l_query+1] = sum; // the last scaling factor + } + /*** backward ***/ + // b[l_query] (b[l_query+1][0]=1 and thus \tilde{b}[][]=1/s[l_query+1]; this is where s[l_query+1] comes from) + for (k = 1; k <= l_ref; ++k) { + int u = set_u(bw, l_query, k); + double[] bi = b[l_query]; + if (u < 3 || u >= bw2*3+3) continue; + bi[u+0] = sM / s[l_query] / s[l_query+1]; bi[u+1] = sI / s[l_query] / s[l_query+1]; + } + // b[l_query-1..1] + for (i = l_query - 1; i >= 1; --i) { + int beg = 1, end = l_ref, x, _beg, _end; + double[] bi = b[i], bi1 = b[i+1]; + double y = (i > 1)? 1. : 0., qli1 = qual[i+1]; + byte qyi1 = query[i+1]; + x = i - bw; beg = beg > x? beg : x; + x = i + bw; end = end < x? end : x; + for (k = end; k >= beg; --k) { + int u, v11, v01, v10; + double e; + u = set_u(bw, i, k); v11 = set_u(bw, i+1, k+1); v10 = set_u(bw, i+1, k); v01 = set_u(bw, i, k+1); + e = (k >= l_ref? 0 : (ref[k+1] > 3 || qyi1 > 3)? 1. : ref[k+1] == qyi1? 1. - qli1 : qli1 / 3.) * bi1[v11]; + bi[u+0] = e * m[0] + .25 * m[1] * bi1[v10+1] + m[2] * bi[v01+2]; // bi1[v11] has been foled into e. + bi[u+1] = e * m[3] + .25 * m[4] * bi1[v10+1]; + bi[u+2] = (e * m[6] + m[8] * bi[v01+2]) * y; + } + // rescale + _beg = set_u(bw, i, beg); _end = set_u(bw, i, end); _end += 2; + for (k = _beg, y = 1./s[i]; k <= _end; ++k) bi[k] *= y; + } + double pb; + { // b[0] + int beg = 1, end = l_ref < bw + 1? l_ref : bw + 1; + double sum = 0.; + for (k = end; k >= beg; --k) { + int u = set_u(bw, 1, k); + double e = (ref[k] > 3 || query[1] > 3)? 1. : ref[k] == query[1]? 1. - qual[1] : qual[1] / 3.; + if (u < 3 || u >= bw2*3+3) continue; + sum += e * b[1][u+0] * bM + .25 * b[1][u+1] * bI; + } + pb = b[0][set_u(bw, 0, 0)] = sum / s[0]; // if everything works as is expected, pb == 1.0 + } + int is_diff = Math.abs(pb - 1.) > 1e-7? 1 : 0; + /*** MAP ***/ + for (i = 1; i <= l_query; ++i) { + double sum = 0., max = 0.; + double[] fi = f[i], bi = b[i]; + int beg = 1, end = l_ref, x, max_k = -1; + x = i - bw; beg = beg > x? beg : x; + x = i + bw; end = end < x? end : x; + for (k = beg; k <= end; ++k) { + int u = set_u(bw, i, k); + double z; + sum += (z = fi[u+0] * bi[u+0]); if (z > max) { max = z; max_k = (k-1)<<2 | 0; } + sum += (z = fi[u+1] * bi[u+1]); if (z > max) { max = z; max_k = (k-1)<<2 | 1; } + } + max /= sum; sum *= s[i]; // if everything works as is expected, sum == 1.0 + if (state != null) state[i-1] = max_k; + if (q != null) { + k = (int)(-4.343 * Math.log(1. - max) + .499); + q[i-1] = (byte)(k > 100? 99 : k); + } + //System.out.println("("+pb+","+sum+")"+" ("+(i-1)+","+(max_k>>2)+","+(max_k&3)+","+max+")"); + } + return 0; + } + + public static void main(String[] args) { + byte[] ref = {'\0', '\1', '\3', '\3', '\1'}; + byte[] query = {'\0', '\3', '\3', '\1'}; + byte[] qual = new byte[4]; + qual[0] = qual[1] = qual[2] = qual[3] = (byte)20; + HmmGlocal hg = new HmmGlocal(1e-3, 0.1, 7); + hg.hmm_glocal(ref, query, qual, null, null); + } +} \ No newline at end of file diff --git a/misc/Makefile b/misc/Makefile index 2d7139d..6c25c78 100644 --- a/misc/Makefile +++ b/misc/Makefile @@ -1,6 +1,6 @@ CC= gcc CXX= g++ -CFLAGS= -g -Wall #-O2 #-m64 #-arch ppc +CFLAGS= -g -Wall -O2 #-m64 #-arch ppc CXXFLAGS= $(CFLAGS) DFLAGS= -D_FILE_OFFSET_BITS=64 OBJS= diff --git a/misc/samtools.pl b/misc/samtools.pl index 9f48b8f..ce8449d 100755 --- a/misc/samtools.pl +++ b/misc/samtools.pl @@ -104,7 +104,6 @@ Options: -Q INT minimum RMS mapping quality for SNPs [$opts{Q}] my $len=0; if ($flt == 0) { if ($t[2] eq '*') { # an indel - # If deletion, remember the length of the deletion my ($a,$b) = split(m{/},$t[3]); my $alen = length($a) - 1; diff --git a/razip.c b/razip.c index dff9347..825e732 100644 --- a/razip.c +++ b/razip.c @@ -94,7 +94,7 @@ int main(int argc, char **argv) buffer = malloc(WINDOW_SIZE); while((c = read(f_src, buffer, WINDOW_SIZE)) > 0) razf_write(rz, buffer, c); razf_close(rz); // f_dst will be closed here - if (argc > optind) unlink(argv[optind]); + if (argc > optind && !pstdout) unlink(argv[optind]); free(buffer); close(f_src); return 0; diff --git a/sam_view.c b/sam_view.c index 3b10e2e..d0fdad2 100644 --- a/sam_view.c +++ b/sam_view.c @@ -124,15 +124,18 @@ int main_samview(int argc, char *argv[]) if (fn_list == 0 && fn_ref) fn_list = samfaipath(fn_ref); // open file handlers if ((in = samopen(argv[optind], in_mode, fn_list)) == 0) { - fprintf(stderr, "[main_samview] fail to open file for reading.\n"); + fprintf(stderr, "[main_samview] fail to open \"%s\" for reading.\n", argv[optind]); + ret = 1; goto view_end; } if (in->header == 0) { - fprintf(stderr, "[main_samview] fail to read the header.\n"); + fprintf(stderr, "[main_samview] fail to read the header from \"%s\".\n", argv[optind]); + ret = 1; goto view_end; } if ((out = samopen(fn_out? fn_out : "-", out_mode, in->header)) == 0) { - fprintf(stderr, "[main_samview] fail to open file for writing.\n"); + fprintf(stderr, "[main_samview] fail to open \"%s\" for writing.\n", fn_out? fn_out : "standard output"); + ret = 1; goto view_end; } if (is_header_only) goto view_end; // no need to print alignments @@ -146,7 +149,10 @@ int main_samview(int argc, char *argv[]) samwrite(out, b); // write the alignment to `out' } } - if (r < -1) fprintf(stderr, "[main_samview] truncated file.\n"); + if (r < -1) { + fprintf(stderr, "[main_samview] truncated file.\n"); + ret = 1; + } bam_destroy1(b); } else { // retrieve alignments in specified regions int i; @@ -161,10 +167,15 @@ int main_samview(int argc, char *argv[]) int tid, beg, end; bam_parse_region(in->header, argv[i], &tid, &beg, &end); // parse a region in the format like `chr2:100-200' if (tid < 0) { // reference name is not found - fprintf(stderr, "[main_samview] fail to get the reference name. Continue anyway.\n"); + fprintf(stderr, "[main_samview] region \"%s\" specifies an unknown reference name. Continue anyway.\n", argv[i]); continue; } - bam_fetch(in->x.bam, idx, tid, beg, end, out, view_func); // fetch alignments + // fetch alignments + if (bam_fetch(in->x.bam, idx, tid, beg, end, out, view_func) < 0) { + fprintf(stderr, "[main_samview] retrieval of region \"%s\" failed due to truncated file or corrupt BAM index file\n", argv[i]); + ret = 1; + break; + } } bam_index_destroy(idx); // destroy the BAM index } diff --git a/sample.c b/sample.c new file mode 100644 index 0000000..b3d2642 --- /dev/null +++ b/sample.c @@ -0,0 +1,92 @@ +#include +#include +#include "sample.h" +#include "khash.h" +KHASH_MAP_INIT_STR(sm, int) + +bam_sample_t *bam_smpl_init(void) +{ + bam_sample_t *s; + s = calloc(1, sizeof(bam_sample_t)); + s->rg2smid = kh_init(sm); + s->sm2id = kh_init(sm); + return s; +} + +void bam_smpl_destroy(bam_sample_t *sm) +{ + int i; + khint_t k; + khash_t(sm) *rg2smid = (khash_t(sm)*)sm->rg2smid; + if (sm == 0) return; + for (i = 0; i < sm->n; ++i) free(sm->smpl[i]); + free(sm->smpl); + for (k = kh_begin(rg2smid); k != kh_end(rg2smid); ++k) + if (kh_exist(rg2smid, k)) free((char*)kh_key(rg2smid, k)); + kh_destroy(sm, sm->rg2smid); + kh_destroy(sm, sm->sm2id); + free(sm); +} + +static void add_pair(bam_sample_t *sm, khash_t(sm) *sm2id, const char *key, const char *val) +{ + khint_t k_rg, k_sm; + int ret; + khash_t(sm) *rg2smid = (khash_t(sm)*)sm->rg2smid; + k_rg = kh_get(sm, rg2smid, key); + if (k_rg != kh_end(rg2smid)) return; // duplicated @RG-ID + k_rg = kh_put(sm, rg2smid, strdup(key), &ret); + k_sm = kh_get(sm, sm2id, val); + if (k_sm == kh_end(sm2id)) { // absent + if (sm->n == sm->m) { + sm->m = sm->m? sm->m<<1 : 1; + sm->smpl = realloc(sm->smpl, sizeof(void*) * sm->m); + } + sm->smpl[sm->n] = strdup(val); + k_sm = kh_put(sm, sm2id, sm->smpl[sm->n], &ret); + kh_val(sm2id, k_sm) = sm->n++; + } + kh_val(rg2smid, k_rg) = kh_val(sm2id, k_sm); +} + +int bam_smpl_add(bam_sample_t *sm, const char *fn, const char *txt) +{ + const char *p = txt, *q, *r; + kstring_t buf; + int n = 0; + khash_t(sm) *sm2id = (khash_t(sm)*)sm->sm2id; + memset(&buf, 0, sizeof(kstring_t)); + while ((q = strstr(p, "@RG")) != 0) { + p = q + 3; + r = q = 0; + if ((q = strstr(p, "\tID:")) != 0) q += 4; + if ((r = strstr(p, "\tSM:")) != 0) r += 4; + if (r && q) { + char *u, *v; + int oq, or; + for (u = (char*)q; *u && *u != '\t' && *u != '\n'; ++u); + for (v = (char*)r; *v && *v != '\t' && *v != '\n'; ++v); + oq = *u; or = *v; *u = *v = '\0'; + buf.l = 0; kputs(fn, &buf); kputc('/', &buf); kputs(q, &buf); + add_pair(sm, sm2id, buf.s, r); + *u = oq; *v = or; + } else break; + p = q > r? q : r; + ++n; + } + if (n == 0) add_pair(sm, sm2id, fn, fn); + free(buf.s); + return 0; +} + +int bam_smpl_rg2smid(const bam_sample_t *sm, const char *fn, const char *rg, kstring_t *str) +{ + khint_t k; + khash_t(sm) *rg2smid = (khash_t(sm)*)sm->rg2smid; + if (rg) { + str->l = 0; + kputs(fn, str); kputc('/', str); kputs(rg, str); + k = kh_get(sm, rg2smid, str->s); + } else k = kh_get(sm, rg2smid, fn); + return k == kh_end(rg2smid)? -1 : kh_val(rg2smid, k); +} diff --git a/sample.h b/sample.h new file mode 100644 index 0000000..85fe499 --- /dev/null +++ b/sample.h @@ -0,0 +1,17 @@ +#ifndef BAM_SAMPLE_H +#define BAM_SAMPLE_H + +#include "kstring.h" + +typedef struct { + int n, m; + char **smpl; + void *rg2smid, *sm2id; +} bam_sample_t; + +bam_sample_t *bam_smpl_init(void); +int bam_smpl_add(bam_sample_t *sm, const char *abs, const char *txt); +int bam_smpl_rg2smid(const bam_sample_t *sm, const char *fn, const char *rg, kstring_t *str); +void bam_smpl_destroy(bam_sample_t *sm); + +#endif diff --git a/samtools.1 b/samtools.1 index d79d176..cc421d4 100644 --- a/samtools.1 +++ b/samtools.1 @@ -1,4 +1,4 @@ -.TH samtools 1 "11 July 2010" "samtools-0.1.8" "Bioinformatics tools" +.TH samtools 1 "27 October 2010" "samtools-0.1.9" "Bioinformatics tools" .SH NAME .PP samtools - Utilities for the Sequence Alignment/Map (SAM) format @@ -18,9 +18,9 @@ samtools merge out.bam in1.bam in2.bam in3.bam .PP samtools faidx ref.fasta .PP -samtools pileup -f ref.fasta aln.sorted.bam +samtools pileup -vcf ref.fasta aln.sorted.bam .PP -samtools mpileup -f ref.fasta -r chr3:1,000-2,000 in1.bam in2.bam +samtools mpileup -C50 -agf ref.fasta -r chr3:1,000-2,000 in1.bam in2.bam .PP samtools tview aln.sorted.bam ref.fasta @@ -65,10 +65,12 @@ format: `chr2' (the whole chr2), `chr2:1000000' (region starting from .B -b Output in the BAM format. .TP -.B -u -Output uncompressed BAM. This option saves time spent on -compression/decomprssion and is thus preferred when the output is piped -to another samtools command. +.BI -f " INT" +Only output alignments with all bits in INT present in the FLAG +field. INT can be in hex in the format of /^0x[0-9A-F]+/ [0] +.TP +.BI -F " INT" +Skip alignments with bits present in INT [0] .TP .B -h Include the header in the output. @@ -76,12 +78,29 @@ Include the header in the output. .B -H Output the header only. .TP +.BI -l " STR" +Only output reads in library STR [null] +.TP +.BI -o " FILE" +Output file [stdout] +.TP +.BI -q " INT" +Skip alignments with MAPQ smaller than INT [0] +.TP +.BI -r " STR" +Only output reads in read group STR [null] +.TP +.BI -R " FILE" +Output reads in read groups listed in +.I FILE +[null] +.TP .B -S Input is in SAM. If @SQ header lines are absent, the .B `-t' option is required. .TP -.B -t FILE +.BI -t " FILE" This file is TAB-delimited. Each line must contain the reference name and the length of the reference, one line for each distinct reference; additional fields are ignored. This file also defines the order of the @@ -92,29 +111,10 @@ can be used as this .I file. .TP -.B -o FILE -Output file [stdout] -.TP -.B -f INT -Only output alignments with all bits in INT present in the FLAG -field. INT can be in hex in the format of /^0x[0-9A-F]+/ [0] -.TP -.B -F INT -Skip alignments with bits present in INT [0] -.TP -.B -q INT -Skip alignments with MAPQ smaller than INT [0] -.TP -.B -l STR -Only output reads in library STR [null] -.TP -.B -r STR -Only output reads in read group STR [null] -.TP -.B -R FILE -Output reads in read groups listed in -.I FILE -[null] +.B -u +Output uncompressed BAM. This option saves time spent on +compression/decomprssion and is thus preferred when the output is piped +to another samtools command. .RE .TP @@ -128,8 +128,10 @@ viewing the same reference sequence. .TP .B pileup -samtools pileup [-f in.ref.fasta] [-t in.ref_list] [-l in.site_list] -[-iscgS2] [-T theta] [-N nHap] [-r pairDiffRate] | +samtools pileup [-2sSBicv] [-f in.ref.fasta] [-t in.ref_list] [-l +in.site_list] [-C capMapQ] [-M maxMapQ] [-T theta] [-N nHap] [-r +pairDiffRate] [-m mask] [-d maxIndelDepth] [-G indelPrior] +| Print the alignment in the pileup format. In the pileup format, each line represents a genomic position, consisting of chromosome name, @@ -138,17 +140,17 @@ mapping qualities. Information on match, mismatch, indel, strand, mapping quality and start and end of a read are all encoded at the read base column. At this column, a dot stands for a match to the reference base on the forward strand, a comma for a match on the reverse strand, -`ACGTN' for a mismatch on the forward strand and `acgtn' for a mismatch -on the reverse strand. A pattern `\\+[0-9]+[ACGTNacgtn]+' indicates -there is an insertion between this reference position and the next -reference position. The length of the insertion is given by the integer -in the pattern, followed by the inserted sequence. Similarly, a pattern -`-[0-9]+[ACGTNacgtn]+' represents a deletion from the reference. The -deleted bases will be presented as `*' in the following lines. Also at -the read base column, a symbol `^' marks the start of a read segment -which is a contiguous subsequence on the read separated by `N/S/H' CIGAR -operations. The ASCII of the character following `^' minus 33 gives the -mapping quality. A symbol `$' marks the end of a read segment. +a '>' or '<' for a reference skip, `ACGTN' for a mismatch on the forward +strand and `acgtn' for a mismatch on the reverse strand. A pattern +`\\+[0-9]+[ACGTNacgtn]+' indicates there is an insertion between this +reference position and the next reference position. The length of the +insertion is given by the integer in the pattern, followed by the +inserted sequence. Similarly, a pattern `-[0-9]+[ACGTNacgtn]+' +represents a deletion from the reference. The deleted bases will be +presented as `*' in the following lines. Also at the read base column, a +symbol `^' marks the start of a read. The ASCII of the character +following `^' minus 33 gives the mapping quality. A symbol `$' marks the +end of a read segment. If option .B -c @@ -168,102 +170,139 @@ The position of indels is offset by -1. .B OPTIONS: .RS .TP 10 -.B -s -Print the mapping quality as the last column. This option makes the -output easier to parse, although this format is not space efficient. +.B -B +Disable the BAQ computation. See the +.B mpileup +command for details. .TP -.B -S -The input file is in SAM. +.B -c +Call the consensus sequence using SOAPsnp consensus model. Options +.BR -T ", " -N ", " -I " and " -r +are only effective when +.BR -c " or " -g +is in use. .TP -.B -i -Only output pileup lines containing indels. +.BI -C " INT" +Coefficient for downgrading the mapping quality of poorly mapped +reads. See the +.B mpileup +command for details. [0] .TP -.B -f FILE +.BI -d " INT" +Use the first +.I NUM +reads in the pileup for indel calling for speed up. Zero for unlimited. [1024] +.TP +.BI -f " FILE" The reference sequence in the FASTA format. Index file .I FILE.fai will be created if absent. .TP -.B -M INT -Cap mapping quality at INT [60] +.B -g +Generate genotype likelihood in the binary GLFv3 format. This option +suppresses -c, -i and -s. This option is deprecated by the +.B mpileup +command. .TP -.B -m INT +.B -i +Only output pileup lines containing indels. +.TP +.BI -I " INT" +Phred probability of an indel in sequencing/prep. [40] +.TP +.BI -l " FILE" +List of sites at which pileup is output. This file is space +delimited. The first two columns are required to be chromosome and +1-based coordinate. Additional columns are ignored. It is +recommended to use option +.TP +.BI -m " INT" Filter reads with flag containing bits in -.I -INT +.I INT [1796] .TP -.B -d INT -Use the first -.I NUM -reads in the pileup for indel calling for speed up. Zero for unlimited. [0] +.BI -M " INT" +Cap mapping quality at INT [60] +.TP +.BI -N " INT" +Number of haplotypes in the sample (>=2) [2] .TP -.B -t FILE +.BI -r " FLOAT" +Expected fraction of differences between a pair of haplotypes [0.001] +.TP +.B -s +Print the mapping quality as the last column. This option makes the +output easier to parse, although this format is not space efficient. +.TP +.B -S +The input file is in SAM. +.TP +.BI -t " FILE" List of reference names ane sequence lengths, in the format described for the .B import command. If this option is present, samtools assumes the input .I is in SAM format; otherwise it assumes in BAM format. -.TP -.B -l FILE -List of sites at which pileup is output. This file is space -delimited. The first two columns are required to be chromosome and -1-based coordinate. Additional columns are ignored. It is -recommended to use option .B -s together with .B -l as in the default format we may not know the mapping quality. .TP -.B -c -Call the consensus sequence using SOAPsnp consensus model. Options -.B -T, -.B -N, -.B -I -and -.B -r -are only effective when -.B -c -or -.B -g -is in use. -.TP -.B -g -Generate genotype likelihood in the binary GLFv3 format. This option -suppresses -c, -i and -s. -.TP -.B -T FLOAT +.BI -T " FLOAT" The theta parameter (error dependency coefficient) in the maq consensus calling model [0.85] -.TP -.B -N INT -Number of haplotypes in the sample (>=2) [2] -.TP -.B -r FLOAT -Expected fraction of differences between a pair of haplotypes [0.001] -.TP -.B -I INT -Phred probability of an indel in sequencing/prep. [40] .RE .TP .B mpileup -samtools mpileup [-r reg] [-f in.fa] in.bam [in2.bam [...]] +samtools mpileup [-Bug] [-C capQcoef] [-r reg] [-f in.fa] [-l list] [-M capMapQ] [-Q minBaseQ] [-q minMapQ] in.bam [in2.bam [...]] -Generate pileup for multiple BAM files. Consensus calling is not -implemented. +Generate BCF or pileup for one or multiple BAM files. Alignment records +are grouped by sample identifiers in @RG header lines. If sample +identifiers are absent, each input file is regarded as one sample. .B OPTIONS: .RS .TP 8 -.B -r STR +.B -B +Disable probabilistic realignment for the computation of base alignment +quality (BAQ). BAQ is the Phred-scaled probability of a read base being +misaligned. Applying this option greatly helps to reduce false SNPs +caused by misalignments. +.TP +.BI -C " INT" +Coefficient for downgrading mapping quality for reads containing +excessive mismatches. Given a read with a phred-scaled probability q of +being generated from the mapped position, the new mapping quality is +about sqrt((INT-q)/INT)*INT. A zero value disables this +functionality; if enabled, the recommended value is 50. [0] +.TP +.BI -f " FILE" +The reference file [null] +.TP +.B -g +Compute genotype likelihoods and output them in the binary call format (BCF). +.TP +.B -u +Similar to +.B -g +except that the output is uncompressed BCF, which is preferred for pipeing. +.TP +.BI -l " FILE" +File containing a list of sites where pileup or BCF is outputted [null] +.TP +.BI -q " INT" +Minimum mapping quality for an alignment to be used [0] +.TP +.BI -Q " INT" +Minimum base quality for a base to be considered [13] +.TP +.BI -r " STR" Only generate pileup in region .I STR [all sites] -.TP -.B -f FILE -The reference file [null] .RE .TP @@ -303,7 +342,7 @@ Approximately the maximum required memory. [500000000] .TP .B merge -samtools merge [-h inh.sam] [-nr] [...] +samtools merge [-nur] [-h inh.sam] [-R reg] [...] Merge multiple sorted alignments. The header reference lists of all the input BAM files, and the @SQ headers of @@ -320,7 +359,7 @@ and the headers of other files will be ignored. .B OPTIONS: .RS .TP 8 -.B -h FILE +.BI -h " FILE" Use the lines of .I FILE as `@' headers to be copied to @@ -331,12 +370,19 @@ replacing any header lines that would otherwise be copied from is actually in SAM format, though any alignment records it may contain are ignored.) .TP +.BI -R " STR" +Merge files in the specified region indicated by +.I STR +.TP .B -r Attach an RG tag to each alignment. The tag value is inferred from file names. .TP .B -n The input alignments are sorted by read names rather than by chromosomal coordinates +.TP +.B -u +Uncompressed BAM output .RE .TP @@ -402,7 +448,7 @@ Treat paired-end reads and single-end reads. .TP .B calmd -samtools calmd [-eubS] +samtools calmd [-eubSr] [-C capQcoef] Generate the MD tag. If the MD tag is already present, this command will give a warning if the MD tag generated is different from the existing @@ -423,6 +469,15 @@ Output compressed BAM .TP .B -S The input is SAM with header lines +.TP +.BI -C " INT" +Coefficient to cap mapping quality of poorly mapped reads. See the +.B pileup +command for details. [0] +.TP +.B -r +Perform probabilistic realignment to compute BAQ, which will be used to +cap base quality. .RE .SH SAM FORMAT @@ -472,6 +527,125 @@ _ 0x0400 d the read is either a PCR or an optical duplicate .TE +.SH EXAMPLES +.IP o 2 +Import SAM to BAM when +.B @SQ +lines are present in the header: + + samtools view -bS aln.sam > aln.bam + +If +.B @SQ +lines are absent: + + samtools faidx ref.fa + samtools view -bt ref.fa.fai aln.sam > aln.bam + +where +.I ref.fa.fai +is generated automatically by the +.B faidx +command. + +.IP o 2 +Attach the +.B RG +tag while merging sorted alignments: + + perl -e 'print "@RG\\tID:ga\\tSM:hs\\tLB:ga\\tPL:Illumina\\n@RG\\tID:454\\tSM:hs\\tLB:454\\tPL:454\\n"' > rg.txt + samtools merge -rh rg.txt merged.bam ga.bam 454.bam + +The value in a +.B RG +tag is determined by the file name the read is coming from. In this +example, in the +.IR merged.bam , +reads from +.I ga.bam +will be attached +.IR RG:Z:ga , +while reads from +.I 454.bam +will be attached +.IR RG:Z:454 . + +.IP o 2 +Call SNPs and short indels for one diploid individual: + + samtools pileup -vcf ref.fa aln.bam > var.raw.plp + samtools.pl varFilter -D 100 var.raw.plp > var.flt.plp + awk '($3=="*"&&$6>=50)||($3!="*"&&$6>=20)' var.flt.plp > var.final.plp + +The +.B -D +option of varFilter controls the maximum read depth, which should be +adjusted to about twice the average read depth. One may consider to add +.B -C50 +to +.B pileup +if mapping quality is overestimated for reads containing excessive +mismatches. Applying this option usually helps +.B BWA-short +but may not other mappers. It also potentially increases reference +biases. + +.IP o 2 +Call SNPs (not short indels) for multiple diploid individuals: + + samtools mpileup -augf ref.fa *.bam | bcftools view -bcv - > snp.raw.bcf + bcftools view snp.raw.bcf | vcfutils.pl filter4vcf -D 2000 | bgzip > snp.flt.vcf.gz + +Individuals are identified from the +.B SM +tags in the +.B @RG +header lines. Individuals can be pooled in one alignment file; one +individual can also be separated into multiple files. Similarly, one may +consider to apply +.B -C50 +to +.BR mpileup . +SNP calling in this way also works for single sample and has the +advantage of enabling more powerful filtering. The drawback is the lack +of short indel calling, which may be implemented in future. + +.IP o 2 +Derive the allele frequency spectrum (AFS) on a list of sites from multiple individuals: + + samtools mpileup -gf ref.fa *.bam > all.bcf + bcftools view -bl sites.list all.bcf > sites.bcf + bcftools view -cGP cond2 sites.bcf > /dev/null 2> sites.1.afs + bcftools view -cGP sites.1.afs sites.bcf > /dev/null 2> sites.2.afs + bcftools view -cGP sites.2.afs sites.bcf > /dev/null 2> sites.3.afs + ...... + +where +.I sites.list +contains the list of sites with each line consisting of the reference +sequence name and position. The following +.B bcftools +commands estimate AFS by EM. + +.IP o 2 +Dump BAQ applied alignment for other SNP callers: + + samtools calmd -br aln.bam > aln.baq.bam + +It adds and corrects the +.B NM +and +.B MD +tags at the same time. The +.B calmd +command also comes with the +.B -C +option, the same as the on in +.B pileup +and +.BR mpileup . +Apply if it helps. + .SH LIMITATIONS .PP .IP o 2 diff --git a/samtools.txt b/samtools.txt index 20e6c15..09c1ccd 100644 --- a/samtools.txt +++ b/samtools.txt @@ -20,60 +20,73 @@ SYNOPSIS samtools faidx ref.fasta - samtools pileup -f ref.fasta aln.sorted.bam + samtools pileup -vcf ref.fasta aln.sorted.bam - samtools mpileup -f ref.fasta -r chr3:1,000-2,000 in1.bam in2.bam + samtools mpileup -C50 -agf ref.fasta -r chr3:1,000-2,000 in1.bam + in2.bam samtools tview aln.sorted.bam ref.fasta DESCRIPTION - Samtools is a set of utilities that manipulate alignments in the BAM + Samtools is a set of utilities that manipulate alignments in the BAM format. It imports from and exports to the SAM (Sequence Alignment/Map) - format, does sorting, merging and indexing, and allows to retrieve + format, does sorting, merging and indexing, and allows to retrieve reads in any regions swiftly. - Samtools is designed to work on a stream. It regards an input file `-' - as the standard input (stdin) and an output file `-' as the standard + Samtools is designed to work on a stream. It regards an input file `-' + as the standard input (stdin) and an output file `-' as the standard output (stdout). Several commands can thus be combined with Unix pipes. Samtools always output warning and error messages to the standard error output (stderr). - Samtools is also able to open a BAM (not SAM) file on a remote FTP or - HTTP server if the BAM file name starts with `ftp://' or `http://'. - Samtools checks the current working directory for the index file and - will download the index upon absence. Samtools does not retrieve the + Samtools is also able to open a BAM (not SAM) file on a remote FTP or + HTTP server if the BAM file name starts with `ftp://' or `http://'. + Samtools checks the current working directory for the index file and + will download the index upon absence. Samtools does not retrieve the entire alignment file unless it is asked to do so. COMMANDS AND OPTIONS view samtools view [-bhuHS] [-t in.refList] [-o output] [-f - reqFlag] [-F skipFlag] [-q minMapQ] [-l library] [-r read- + reqFlag] [-F skipFlag] [-q minMapQ] [-l library] [-r read- Group] [-R rgFile] | [region1 [...]] - Extract/print all or sub alignments in SAM or BAM format. If - no region is specified, all the alignments will be printed; - otherwise only alignments overlapping the specified regions - will be output. An alignment may be given multiple times if + Extract/print all or sub alignments in SAM or BAM format. If + no region is specified, all the alignments will be printed; + otherwise only alignments overlapping the specified regions + will be output. An alignment may be given multiple times if it is overlapping several regions. A region can be presented, - for example, in the following format: `chr2' (the whole - chr2), `chr2:1000000' (region starting from 1,000,000bp) or - `chr2:1,000,000-2,000,000' (region between 1,000,000 and - 2,000,000bp including the end points). The coordinate is + for example, in the following format: `chr2' (the whole + chr2), `chr2:1000000' (region starting from 1,000,000bp) or + `chr2:1,000,000-2,000,000' (region between 1,000,000 and + 2,000,000bp including the end points). The coordinate is 1-based. OPTIONS: -b Output in the BAM format. - -u Output uncompressed BAM. This option saves time spent - on compression/decomprssion and is thus preferred - when the output is piped to another samtools command. + -f INT Only output alignments with all bits in INT present + in the FLAG field. INT can be in hex in the format of + /^0x[0-9A-F]+/ [0] + + -F INT Skip alignments with bits present in INT [0] -h Include the header in the output. -H Output the header only. + -l STR Only output reads in library STR [null] + + -o FILE Output file [stdout] + + -q INT Skip alignments with MAPQ smaller than INT [0] + + -r STR Only output reads in read group STR [null] + + -R FILE Output reads in read groups listed in FILE [null] + -S Input is in SAM. If @SQ header lines are absent, the `-t' option is required. @@ -85,35 +98,24 @@ COMMANDS AND OPTIONS `samtools faidx ', the resultant index file .fai can be used as this file. - -o FILE Output file [stdout] - - -f INT Only output alignments with all bits in INT present - in the FLAG field. INT can be in hex in the format of - /^0x[0-9A-F]+/ [0] - - -F INT Skip alignments with bits present in INT [0] - - -q INT Skip alignments with MAPQ smaller than INT [0] - - -l STR Only output reads in library STR [null] - - -r STR Only output reads in read group STR [null] - - -R FILE Output reads in read groups listed in FILE [null] + -u Output uncompressed BAM. This option saves time spent + on compression/decomprssion and is thus preferred + when the output is piped to another samtools command. tview samtools tview [ref.fasta] - Text alignment viewer (based on the ncurses library). In the - viewer, press `?' for help and press `g' to check the align- - ment start from a region in the format like - `chr10:10,000,000' or `=10,000,000' when viewing the same + Text alignment viewer (based on the ncurses library). In the + viewer, press `?' for help and press `g' to check the align- + ment start from a region in the format like + `chr10:10,000,000' or `=10,000,000' when viewing the same reference sequence. - pileup samtools pileup [-f in.ref.fasta] [-t in.ref_list] [-l - in.site_list] [-iscgS2] [-T theta] [-N nHap] [-r - pairDiffRate] | + pileup samtools pileup [-2sSBicv] [-f in.ref.fasta] [-t in.ref_list] + [-l in.site_list] [-C capMapQ] [-M maxMapQ] [-T theta] [-N + nHap] [-r pairDiffRate] [-m mask] [-d maxIndelDepth] [-G + indelPrior] | Print the alignment in the pileup format. In the pileup for- mat, each line represents a genomic position, consisting of @@ -123,20 +125,18 @@ COMMANDS AND OPTIONS end of a read are all encoded at the read base column. At this column, a dot stands for a match to the reference base on the forward strand, a comma for a match on the reverse - strand, `ACGTN' for a mismatch on the forward strand and - `acgtn' for a mismatch on the reverse strand. A pattern - `\+[0-9]+[ACGTNacgtn]+' indicates there is an insertion - between this reference position and the next reference posi- - tion. The length of the insertion is given by the integer in - the pattern, followed by the inserted sequence. Similarly, a - pattern `-[0-9]+[ACGTNacgtn]+' represents a deletion from the - reference. The deleted bases will be presented as `*' in the - following lines. Also at the read base column, a symbol `^' - marks the start of a read segment which is a contiguous sub- - sequence on the read separated by `N/S/H' CIGAR operations. - The ASCII of the character following `^' minus 33 gives the - mapping quality. A symbol `$' marks the end of a read seg- - ment. + strand, a '>' or '<' for a reference skip, `ACGTN' for a mis- + match on the forward strand and `acgtn' for a mismatch on the + reverse strand. A pattern `\+[0-9]+[ACGTNacgtn]+' indicates + there is an insertion between this reference position and the + next reference position. The length of the insertion is given + by the integer in the pattern, followed by the inserted + sequence. Similarly, a pattern `-[0-9]+[ACGTNacgtn]+' repre- + sents a deletion from the reference. The deleted bases will + be presented as `*' in the following lines. Also at the read + base column, a symbol `^' marks the start of a read. The + ASCII of the character following `^' minus 33 gives the map- + ping quality. A symbol `$' marks the end of a read segment. If option -c is applied, the consensus base, Phred-scaled consensus quality, SNP quality (i.e. the Phred-scaled proba- @@ -155,68 +155,109 @@ COMMANDS AND OPTIONS OPTIONS: - -s Print the mapping quality as the last column. This - option makes the output easier to parse, although - this format is not space efficient. + -B Disable the BAQ computation. See the mpileup com- + mand for details. - -S The input file is in SAM. + -c Call the consensus sequence using SOAPsnp consensus + model. Options -T, -N, -I and -r are only effective + when -c or -g is in use. - -i Only output pileup lines containing indels. + -C INT Coefficient for downgrading the mapping quality of + poorly mapped reads. See the mpileup command for + details. [0] + + -d INT Use the first NUM reads in the pileup for indel + calling for speed up. Zero for unlimited. [1024] -f FILE The reference sequence in the FASTA format. Index file FILE.fai will be created if absent. - -M INT Cap mapping quality at INT [60] + -g Generate genotype likelihood in the binary GLFv3 + format. This option suppresses -c, -i and -s. This + option is deprecated by the mpileup command. - -m INT Filter reads with flag containing bits in INT - [1796] - - -d INT Use the first NUM reads in the pileup for indel - calling for speed up. Zero for unlimited. [0] - - -t FILE List of reference names ane sequence lengths, in - the format described for the import command. If - this option is present, samtools assumes the input - is in SAM format; otherwise it - assumes in BAM format. + -i Only output pileup lines containing indels. - -l FILE List of sites at which pileup is output. This file - is space delimited. The first two columns are - required to be chromosome and 1-based coordinate. - Additional columns are ignored. It is recommended - to use option -s together with -l as in the default - format we may not know the mapping quality. + -I INT Phred probability of an indel in sequencing/prep. + [40] - -c Call the consensus sequence using SOAPsnp consensus - model. Options -T, -N, -I and -r are only effective - when -c or -g is in use. + -l FILE List of sites at which pileup is output. This file + is space delimited. The first two columns are + required to be chromosome and 1-based coordinate. + Additional columns are ignored. It is recommended + to use option - -g Generate genotype likelihood in the binary GLFv3 - format. This option suppresses -c, -i and -s. + -m INT Filter reads with flag containing bits in INT + [1796] - -T FLOAT The theta parameter (error dependency coefficient) - in the maq consensus calling model [0.85] + -M INT Cap mapping quality at INT [60] -N INT Number of haplotypes in the sample (>=2) [2] - -r FLOAT Expected fraction of differences between a pair of + -r FLOAT Expected fraction of differences between a pair of haplotypes [0.001] - -I INT Phred probability of an indel in sequencing/prep. - [40] + -s Print the mapping quality as the last column. This + option makes the output easier to parse, although + this format is not space efficient. + + -S The input file is in SAM. + + -t FILE List of reference names ane sequence lengths, in + the format described for the import command. If + this option is present, samtools assumes the input + is in SAM format; otherwise it + assumes in BAM format. -s together with -l as in + the default format we may not know the mapping + quality. + + -T FLOAT The theta parameter (error dependency coefficient) + in the maq consensus calling model [0.85] - mpileup samtools mpileup [-r reg] [-f in.fa] in.bam [in2.bam [...]] + mpileup samtools mpileup [-Bug] [-C capQcoef] [-r reg] [-f in.fa] [-l + list] [-M capMapQ] [-Q minBaseQ] [-q minMapQ] in.bam [in2.bam + [...]] - Generate pileup for multiple BAM files. Consensus calling is - not implemented. + Generate BCF or pileup for one or multiple BAM files. Align- + ment records are grouped by sample identifiers in @RG header + lines. If sample identifiers are absent, each input file is + regarded as one sample. OPTIONS: - -r STR Only generate pileup in region STR [all sites] + -B Disable probabilistic realignment for the computation + of base alignment quality (BAQ). BAQ is the Phred- + scaled probability of a read base being misaligned. + Applying this option greatly helps to reduce false + SNPs caused by misalignments. + + -C INT Coefficient for downgrading mapping quality for reads + containing excessive mismatches. Given a read with a + phred-scaled probability q of being generated from + the mapped position, the new mapping quality is about + sqrt((INT-q)/INT)*INT. A zero value disables this + functionality; if enabled, the recommended value is + 50. [0] -f FILE The reference file [null] + -g Compute genotype likelihoods and output them in the + binary call format (BCF). + + -u Similar to -g except that the output is uncompressed + BCF, which is preferred for pipeing. + + -l FILE File containing a list of sites where pileup or BCF + is outputted [null] + + -q INT Minimum mapping quality for an alignment to be used + [0] + + -Q INT Minimum base quality for a base to be considered [13] + + -r STR Only generate pileup in region STR [all sites] + reheader samtools reheader @@ -243,8 +284,8 @@ COMMANDS AND OPTIONS [500000000] - merge samtools merge [-h inh.sam] [-nr] - [...] + merge samtools merge [-nur] [-h inh.sam] [-R reg] + [...] Merge multiple sorted alignments. The header reference lists of all the input BAM files, and the @SQ headers of inh.sam, @@ -261,12 +302,16 @@ COMMANDS AND OPTIONS SAM format, though any alignment records it may con- tain are ignored.) + -R STR Merge files in the specified region indicated by STR + -r Attach an RG tag to each alignment. The tag value is inferred from file names. -n The input alignments are sorted by read names rather than by chromosomal coordinates + -u Uncompressed BAM output + index samtools index @@ -315,7 +360,7 @@ COMMANDS AND OPTIONS -S Treat paired-end reads and single-end reads. - calmd samtools calmd [-eubS] + calmd samtools calmd [-eubSr] [-C capQcoef] Generate the MD tag. If the MD tag is already present, this command will give a warning if the MD tag generated is dif- @@ -333,6 +378,12 @@ COMMANDS AND OPTIONS -S The input is SAM with header lines + -C INT Coefficient to cap mapping quality of poorly mapped + reads. See the pileup command for details. [0] + + -r Perform probabilistic realignment to compute BAQ, + which will be used to cap base quality. + SAM FORMAT SAM is TAB-delimited. Apart from the header lines, which are started @@ -375,26 +426,106 @@ SAM FORMAT |0x0400 | d | the read is either a PCR or an optical duplicate | +-------+-----+--------------------------------------------------+ +EXAMPLES + o Import SAM to BAM when @SQ lines are present in the header: + + samtools view -bS aln.sam > aln.bam + + If @SQ lines are absent: + + samtools faidx ref.fa + samtools view -bt ref.fa.fai aln.sam > aln.bam + + where ref.fa.fai is generated automatically by the faidx command. + + + o Attach the RG tag while merging sorted alignments: + + perl -e 'print "@RG\tID:ga\tSM:hs\tLB:ga\tPL:Illu- + mina\n@RG\tID:454\tSM:hs\tLB:454\tPL:454\n"' > rg.txt + samtools merge -rh rg.txt merged.bam ga.bam 454.bam + + The value in a RG tag is determined by the file name the read is com- + ing from. In this example, in the merged.bam, reads from ga.bam will + be attached RG:Z:ga, while reads from 454.bam will be attached + RG:Z:454. + + + o Call SNPs and short indels for one diploid individual: + + samtools pileup -vcf ref.fa aln.bam > var.raw.plp + samtools.pl varFilter -D 100 var.raw.plp > var.flt.plp + awk '($3=="*"&&$6>=50)||($3!="*"&&$6>=20)' var.flt.plp > + var.final.plp + + The -D option of varFilter controls the maximum read depth, which + should be adjusted to about twice the average read depth. One may + consider to add -C50 to pileup if mapping quality is overestimated + for reads containing excessive mismatches. Applying this option usu- + ally helps BWA-short but may not other mappers. It also potentially + increases reference biases. + + + o Call SNPs (not short indels) for multiple diploid individuals: + + samtools mpileup -augf ref.fa *.bam | bcftools view -bcv - > + snp.raw.bcf + bcftools view snp.raw.bcf | vcfutils.pl filter4vcf -D 2000 | bgzip + > snp.flt.vcf.gz + + Individuals are identified from the SM tags in the @RG header lines. + Individuals can be pooled in one alignment file; one individual can + also be separated into multiple files. Similarly, one may consider to + apply -C50 to mpileup. SNP calling in this way also works for single + sample and has the advantage of enabling more powerful filtering. The + drawback is the lack of short indel calling, which may be implemented + in future. + + + o Derive the allele frequency spectrum (AFS) on a list of sites from + multiple individuals: + + samtools mpileup -gf ref.fa *.bam > all.bcf + bcftools view -bl sites.list all.bcf > sites.bcf + bcftools view -cGP cond2 sites.bcf > /dev/null 2> sites.1.afs + bcftools view -cGP sites.1.afs sites.bcf > /dev/null 2> sites.2.afs + bcftools view -cGP sites.2.afs sites.bcf > /dev/null 2> sites.3.afs + ...... + + where sites.list contains the list of sites with each line consisting + of the reference sequence name and position. The following bcftools + commands estimate AFS by EM. + + + o Dump BAQ applied alignment for other SNP callers: + + samtools calmd -br aln.bam > aln.baq.bam + + It adds and corrects the NM and MD tags at the same time. The calmd + command also comes with the -C option, the same as the on in pileup + and mpileup. Apply if it helps. + + LIMITATIONS - o Unaligned words used in bam_import.c, bam_endian.h, bam.c and + o Unaligned words used in bam_import.c, bam_endian.h, bam.c and bam_aux.c. - o In merging, the input files are required to have the same number of - reference sequences. The requirement can be relaxed. In addition, - merging does not reconstruct the header dictionaries automatically. - Endusers have to provide the correct header. Picard is better at + o In merging, the input files are required to have the same number of + reference sequences. The requirement can be relaxed. In addition, + merging does not reconstruct the header dictionaries automatically. + Endusers have to provide the correct header. Picard is better at merging. - o Samtools paired-end rmdup does not work for unpaired reads (e.g. - orphan reads or ends mapped to different chromosomes). If this is a - concern, please use Picard's MarkDuplicate which correctly handles + o Samtools paired-end rmdup does not work for unpaired reads (e.g. + orphan reads or ends mapped to different chromosomes). If this is a + concern, please use Picard's MarkDuplicate which correctly handles these cases, although a little slower. AUTHOR - Heng Li from the Sanger Institute wrote the C version of samtools. Bob + Heng Li from the Sanger Institute wrote the C version of samtools. Bob Handsaker from the Broad Institute implemented the BGZF library and Jue - Ruan from Beijing Genomics Institute wrote the RAZF library. Various + Ruan from Beijing Genomics Institute wrote the RAZF library. Various people in the 1000 Genomes Project contributed to the SAM format speci- fication. @@ -404,4 +535,4 @@ SEE ALSO -samtools-0.1.8 11 July 2010 samtools(1) +samtools-0.1.9 27 October 2010 samtools(1) -- 2.30.2