citation classics

a fun way to pass lockdown time is perusing these little nuggets, where an author of the highly cited work gives personal background and opinion about the work itself

After only a few quick glances, I found Pat O’Farrell’s work about 2-D protein gel electrophoresis… a person very important to me because of a fleeting interaction we had in which he told me to stick with Stavros Lomvardas during my first year of grad school at UCSF when i was trying to decide where to go. Amazing… i had no idea what he had been researching so many years ago, and that it has been cited a bajillion times

download only the SRAs you want, from cmd line

go to the NCBI SRA database site, click the record you are interested in and then click the tab ‘data access’ and copy-paste the https URL into a text file

place this text file of your SRA URLs into the directory where you wish to download the data (no spaces in the list or any trailing characters, etc)

enter the command

for i in `cat srr.list` ; do wget $i ; done

et voila, go have a snack

understanding bisulfite sequencing — C-to-T vs G-to-A

Recently, I have been re-analyzing bisulfite sequencing data from a fungus with significant DNA methylation around its centromeres, but with limited mappability due to high transposon similarity. I ‘discovered’ that diferent bisulfite mapping software produced drastically different summary methylation statistics.

After about a week of comparing the output of various mapping softwares, I was faced with another discovery: I didn’t quite grasp how the bisulfite software works at the conceptual level. What an embarrassment! A… sixth year postdoc?… that doesn’t understand his own bread-and-butter methods. In any case, my initial realization stemmed from reading the perl script of our lab’s ‘in-house’ (i.e. not published and not well documented) bisulfite mapper, which works a lot like published softwares “Bismark” and “BS-seeker” and others (however it is quite different from BSMAP, Pash, and others… perhaps more on that later). With software like Bismark, all sequencing reads and the genome itself are C-to-T converted, whereupon mapping is carried out, and later resolved using read ID matching to calculate C that did not convert in reads, and was therefore methylated. However, a G-to-A converted genome is also generated. This has long bothered me… why do we need a G-to-A converted genome as well? My understanding was we need C-to-T alone because of the directionality strategy built into essentially all library preparation methods. Of course one needs directionality for RNA-seq, since you care very much about the strand that gave rise to a particular single-stranded RNA. But, to review, why directionality for bisulfite libraries? In order to make the libraries, adapters are ligated to the fragmented gDNA and the protolibraries are then denatured. That means the 5′-end of your protolibrary top strand will have the same adaptor sequence as the 5′-end of your bottom strand. For our NextSeq flowcell, these adaptors serve two critical functions: 1.) as the complementary strand that anneals the library to the flowcell, and 2.) as a sequencing primer binding site. Following bridge amplification of the libraries on-flowcell, which generates a complementary strand to your originals you added, extension from the read 1 sequencing primer re-creates the “original top” and “original bottom” — such is the beauty of directionality in the bisulfite context. The sequencing information you get from read 1-based extension exactly replicates the ‘original’ bisulfite-converted genomic DNA. On this strand, C converts to T when it’s not protected by methylation (and its derivatives), and so that is why we map these reads to a C-to-T converted genome. OK…. so why the G-to-A converted genome also?? G-to-A seems relevant, but only because the template strand covalently bound to the flowcell represents your data in that transformation…ah! but what about paired end libraries!!! Duh. You may have seen this coming; if so, my apologies. The punchline to this diatribe is: as long as you are using directional library prep methods (basically any normal kit will produce directional libraries, for instance from Tecan/Nugen, NEB, Illumina, Swift, etc.) paired end bisulfite sequencing requires the G-to-A converted read and genome sequence to map read-2 reads. All of your unmethylated bases in the ‘original strand’ (top or bottom) will register as A on the complementary strand, the ‘complementary to original’ strand, which is synthesized from the read 2 primer (in our case, built in to the NextSeq reagents). The Bismark github page has some documentation on this, but I had really not appreciated the importance of directionality in these libraries and how that translates to single-end mapping requiring only the C-to-T converted genome and reads; and likewise the relevance of G-to-A converted genomes for read 2 in paired-end libraries.

xml update

generate a bigwig (bw) file from a bed (remove non-nuclear genomes first)

for i in ./*bed; do perl -wnl -e ‘/Chr\d/ and print;’ $i | bedtools genomecov -bg -i – -g ~/bsgenome_tair9.Chr.txt > ${i%%bed}bg; done


for i in ./*bg; do bedGraphToBigWig $i ~/bsgenome_tair9.Chr.txt ${i%%bg}bw; done

collect the names of these bw

ls -l | awk ‘{print $9}’ > file_names.txt

perl -wpl -e ‘s|^(.*)$|<Resource name=”$1″ path=”http://yourServerHere/igvdata/chip_seq_jic/$1″/>|g;’ file_names.txt > xml.tmp

paste names from this new file into the field of a new igvdata xml with appropriately descriptive global and category headers,

add this xml to your igv_registry.txt in your /var/www/html/igv directory,
and import the bw files to an appropriately named directory in /var/www/html/igvdata

new data? quickly make XML files for your IGV server

Running IGV from a server (as briefly outlined here) enables you to access data quickly and remotely, but when new data comes in that you want to add to your existing data (‘nodes’), modifying the xml files can really suck the life out of you.

I have been getting lots of new data, but have been putting off uploading to my IGV web server because… I didn’t want to update the xml file! I realized however a relatively quick way to coax your data tracks (for me, mostly bisulfite sequencing) into the xml format to quickly add to the existing IGV xml file with all your other data. There’s almost certainly a better way to do this, but here’s what I came up with in a nutshell: move the data to the directory where you would host tracks that are viewed on your server (/var/www/html/igvdata/….), then open a new IGV window and upload those tracks, save the session (which outputs a session-specific XML), then basically copy-paste that XML file generated to make the new node in your existing data structure.

  1. put all the data tracks you want to add to your IGV server into the appropriate directory (/var/www/html/igvdata/some_new_directory_/ data_tracks_here_:-) /
  2. in a new IGV session, open all the tracks you want to ultimately add to your server
  3. save the session
  4. open the session xml file and copy all the “resources” that are those tracks you are adding.
  5. start a new xml in an IGV-friendly layout as outlined here with <Global> and <Category> root-parent elements.
  6. run this snippet of code over this new xml to add “name” feature to these file resources according to their basename (or you can do this by hand if you have few samples and/or don’t like the basename) otherwise your tracks will be nameless and sad
    1. perl -wpl -e ‘s/(path=some_new_directory\/)(.)(\/>)/name=”$2 $1$2$3/g;’
  7. modify the order and create sub-categories of the files using an xml editor like komodo so you don’t lose your marbles
  8. save this new and righteous XML in your web servin’ directory.
  9. update your registry text file (as outlined on the same webpage as step 5).

good luck!

calculate mappability of a genome

I couldn’t find a simple explanation of how to run gem_mappability as was carried out in Catoni et al. 2017 EMBO (out of the Paszkowski lab). I found enough information here to get me going – below is the simplified version for those that simply want to get a bigwig or bedgraph of these values but don’t care about comparing a bajillion different k-mer sizes.

  1. download the compressed file (your specific version may be different)
  2. tar -xvjf GEM-binaries-Linux-x86_64-core_i3-20130406-045632.tbz2
  3. cd GEM-binaries-Linux-x86_64-core_i3-20130406-045632/
  4. cp all the executables in ./bin to a directory in your path, such as /usr/local/bin
  5. then run the following – change the names of your files accordingly:

gem-indexer -i /path_to_your_genome_fasta -o output_name_you_pick

gem-mappability -I output_from_above -l [you decide how big kmer should be] -o pick_a_gem_output_name

gem-2-wig -I gem_indexer_output -i gem_mappability_output -o your_wig

then convert your wig to bigWig using Jim Kent/UCSC tools

fastq trimming – Illumina truseq2 v truseq3

the 5′ end (13 nt) are identical, with some similarity further 3′ as well

When I’m not sure which adaptors were used for the construction of a sequencing library, but I know they were Illumina, I take the top ~100k reads and run trimmomatic using more-or-less default settings against the 2 different Illumina truseq adaptors that ‘ship’ with that software. Then compare how the two trim-logs look — the one that trimmed off more crap is the one to proceed with. Something like this (these are set for small RNA libraries where the inserts are very short!):

java -jar trimmomatic-0.36.jar SE -threads 5 -trimlog trimlog_test test.fastq out.test.fastq ILLUMINACLIP:TruSeq2-SE.fa:2:30:10 LEADING:3 TRAILING:3 SLIDINGWINDOW:4:15 MINLEN:1

awk ‘$6>0 {print}’ trimlog_test | wc

genotyping in 96-well format

had to split the plates to get them to fit into small thermocycler :-<

Generating Arabidopsis triple and quadruple mutants (and beyond) requires stamina.  the mind-numbing process can be made less time-consuming by doing on-plate DNA extraction with an SDS-free extraction method, as put forth in a Ronan O’Malley et al. paper (“guide to the t-dna insertion collection” approximately). They use steel balls and a paint can shaker to achieve tissue disruption, whereas I use a PCR plate with zirconia beads after deep-freezing the leaf sample (partially submerge plate on metal plate rack in ethanol+dry ice). I then add 300ul TE with 100mM NaCl (roughly per O’Malley), vortex a bit, and spin down the plate at 4000 RPM in an eppendorf 5804. The trick post-spin is take only the very top to avoid contaminating debris. I took ~50ul here, adding it to a new plate where I’d crudely pipetman’d out 200ul cold isopropanol. Once done with this, I store o/n in the freezer, spin down at high speed as before the next day, and then decant as much iso as possible. From here, I do a 70% EtOH wash and spin as before, then dry the samples out. I found ramming air into the plate was the only way that worked for me… I titled the plate at one of our PCR vents and the remaining traces of alcohol slowly evaporated. It took a long time to dry. I resuspend in 50ul EB and proceed with PCR using multichannel pipets and life is good.  

NB: I use adhesive foil for PCR plates to cover the samples for all these steps. The foil dents heavily during the vortex, but doesn’t break open.

Once ready to run the samples on a DNA gel, you can cast the gel with an appropriately-sized comb to further streamline the process. I use a 26-well format from Biorad in their sub-cell electrophoresis get-up, then load WT in even positions, with mutant in odd positions (something I realize the O’Malley paper also recommends!)

make a freeNAS file server with old parts: I

 when my lab departed for the John Innes Centre about a year ago, there was a large quantity of raw computer parts left behind for the salvage heap.  I wanted a dedicated place to back up my laptop and home computers remotely, and had heard about freely-available network attached storage (NAS) systems, but that was where my knowledge ended.  Our lab had been using the somewhat pricey synology NAS system, and I wanted to learn if there was an open source alternative. 

FreeBSD-based FreeNAS fit the bill.  There is a TON of info and support for how FreeNAS works and how to get it going.  What I envisioned was taking some old HDDs laying around, reformatting them, and using them as the storage core of the system.  I would boot to a usb drive on which I’ve installed FreeNAS, and format things from there.   The amazing thing is… it basically worked and now I have a server for a lot of previously unsecure stuff.  Booting off of a USB loaded with the OS is actually the recommended way to go: see this simple instructional video from FreeNAS 

One detail that is worth noting is that FreeNAS storage is memory intensive.  I am not 100% clear on why, but I believe it has to do with the ZFS-based architecture (NB: zfs does not stand for anything!).  For each TB of hard drive, they recommend 1GB of RAM, starting with at least 4GB RAM as a base.  Luckily I had a bunch of DDR3 RAM laying around and a capable Asus X99 motherboard with an i7 cpu.  Once I got the thing running, it had 64GB RAM supporting 8TB of hard storage; I’ve gone about 2TB into this thing after about a year. 

the web interface for FreeNAS with some basic system specs.

Make an IGV server out of your computer

this makes it possible to remotely view your data on any integrated genome viewer (IGV) session with an internet connection.  Make sure your computer’s web server enabled — for me on ubuntu 14 and 16, it meant installing the so-called LAMP stack (I used the slick little installer package called tacksel as outlined here)

  1. become loosely familiar with xml format – you use xml to write the structure of your tracks and features, how they’re colored, and lots of other details
  2. there is a hierarchy that you will adhere to:  a.) a registry at the top (information about where your xml files are) , b.) the xml files stating where your data files are (I put them in the directory ./igvdata/, and c.) a directory full of those data in their precise location as specified by xml files in b (put these below the xmls within ./igvdata)
  3. how i did it: in /var/www/html/igv, make your registry e.g. igv_registry.txt.  it’s a simple list (of so-called infolinks) comprised of  http://your_server_url/igvdata/_some_xml_file.xml   ….  This registry file points to the location of your secondary xml files; my files are in subdirectories within /var/www/html/igvdata/ so my registry file reads like: http://__ip___/igvdata/some_cool_set_of_experiments.xml
  4. and the xml file “some_cool_set_of_experiments.xml” reads like:
<?xml version="1.0" encoding="UTF-8"?>
<Global name="bsseq_sex_cells"  infolink="http://ip_addr/igv/" version="1">
    <Category name="bsseq_sperm">
        <Category name="wt.sp.f4">
         <Resource name="wt.sp.CG.f4"
                  path="http://ip_addr/igvdata/bsseq.sperm/" />
        <Resource name="wt.sp.CHG.f4"
                  path="http://ip_addr/igvdata/bsseq.sperm/" />
         <Resource name="wt.sp.CHH.f4"
                  path="http://ip_addr/igvdata/bsseq.sperm/" />


here I have specified the “global name” as bsseq_sex_cells, so this will be an expandable menu item in IGV, with the CG, CHG, and CHH tracks available as items within that menu. 

As you can see above in the xml snippet, under ‘Resource’ the path shows my tracks for this set of experiments are all within a sub-directory in ./igvdata called bsseq.sperm — it seemed a good way for me to organize my data.  Also cool is that each sub-directory corresponds to a different menu heading, like so

a menu of my data sets after all the xml’ing and set up

6. to enable this, set your IGV session to look for your server at your IP address:

click ‘Edit Server properties’ and enter the appropriate URL; change name of registry accordingly