The calculation based on this definition is also simple: sum up the pairwise differences (in nucleotides) for all possible pairs, then divide by the total number of such pairs, which yields the desired quantity.

However, sometimes it is easier to perform the calculation on a per base pair basis, i.e. imagine you march along a multiple alignment, where your unit of calculation is a column of aligned sequences. This is particularly true when one deals with a long stretch of sequences, or when the coverage differs at different locations, as is true for today’s short reads generated from NGS.

I did some search on the internet and didn’t find a simple formula for this. Well, in fact, there is one, from this paper. However it takes a bit of time to digest it. Here is a digested version, or more importantly, a way to think of the formula:

Think of the very original definition as two levels of summation: first sum over the length of the sequence for any random pair of DNA from the population, calculate the total number of differences; then sum that over all pairs. The grand total is then divided by the total number of pairs. Imagine if we expand the summations: every unit is just one bp difference at position X between sequence x and y, which contributed ONE to the summation in the numerator. Now let’s ask how many “ONE”s does position X contribute in total to the numerator. That depends on the frequency of either the minor or major allele. Let’s say there are n sequences in total and the minor allele at X occurs j times out of n. Then the total number of “ONE”s is easy to calculate: it is just (j choose 1) * (n-j choose 1), which equals j*(n-j). The shared denominator is (n choose 2), equaling n(n-1)/2. Therefore the following formula:

sum ( 2j(n-j) / n(n-1) )

Note that this formula does a bp-by-bp calculation and thus it seems that one can incorporate the coverage differences into the calculation by varying n for different sites. Indeed, with some simple algebra, one can show that this way of calculation equals the coverage adjusted nucleotide diversity estimate proposed by Begun et al in the paper I mentioned above. This calculation is then fairly straightforward.

The title is a bit deceiving, as what is really reported in this article is a failure of in-vivo validation of in-silico predicted muscle enhancers. OK. straight to what they have done:

The essence of the results is summarized (and hidden) in this supplementary table 2:

One can see that (1) the testing process has fairly high failure rate, such that less than half of the samples could successfully make it to the final tests; (2) among those tested, because the method has high background noise, they setup a high stringency to call a “muscle specific enhancer”. As a result, a very small fraction (<10%) were called. Surprisingly, the positive hits is not at all enriched among the apriori “muscle specific gene set”. If anything, there is a deficit of positive hits in this set (although the difference is not significant according to Fisher’s Exact Test).

So how are these enhancer elements predicted? They used three programs, the Logistic Regression Analysis (LRA), which is the initial idea of using clusters of binding sites (1998), two later variations–MSCAN and ClusterBuster. Note that all of them were based on this idea of clustering of transcription factor binding sites. Five key TF believed to be involved in muscle development were included with their PWM taken from JASPAR. — Q: how good are these PWMs?

They first assembled three sets of genomic regions: muscle gene regulatory regions; non-muscle gene regulatory regions and background conserved sequences. For the first two, they took literature curated (based on microarray of gene expression profile or individual studies) muscle genes and a random set of the non-muscle genes. They conducted their searches effectively in a +/- 10kb region of a gene, that is, the gene locus plus 10kb upstream and downstream, all non-coding regions included. — Q: is 10kb enough to capture most of the enhancers? Will expand this range include more authentic ones and thus raise their heuristic threshold for calling a significant enrichment for enhancers?

One detail worth noticing is that the constructs they tested are very short, at most 400 bp long (1st. and 3rd. quartiles are 259bp and 365 bp). This is surprising to me as even in Drosophila where the enhancers are known to be quite compact, a 300bp enhancer is considered extremely short. — Q: is this short length a result of the prediction algorithms or artificially trimmed to make them amenable to cloning and testing? Critically, if we believe that minimal functional enhancers in human are on average much longer than 400bp (which is not unreasonable given our knowledge about minimal enhancer length in Drosophila and the comparison between human and flies), could this short length be a key contributor to the surprisingly low validation rate???

There is more details pertaining to their methodologies, such as how they determine whether an amplicon drives “muscle specific gene expression” or not, involving the use of three different cell lines. I’m not sure whether the cell line systems are good enough to capture the essential biologies. Naively, suppose we know the truth functional enhancers for muscle development in-vivo, how many of them can be detected in this cell line system? But since those technical details are well beyond my expertise, I’ll leave them to someone who is more qualified at to critique.

Overall, I find the short length of the inserts the most concerning, and to me that may explain the low success rate of validation. But I’m also not surprised if truly the predictions don’t work that well, which suggests additional rules necessary for predicting enhancer elements.

http://www.unixwiz.net/techtips/ssh-agent-forwarding.html

http://mah.everybody.org/docs/ssh

[Notes]

1. use ssh-keygen to generate a public-private key pair.
2. The public key is stored in id_rsa.pub (or id_dsa.pub). This need to be copied to the server YOU ARE CONNECTING TO, into the ~/.ssh/autherized_keys
3. the private key is stored in id_rsa (or id_dsa). To connect to the same server from another remote machine, copy this file to that machine’s ~/.ssh/ folder
4. The security mechanism of the public-private key pairs are still unclear to me (I get some part by reading online). Somehow the security aspect is related to the fact that during the process of the authentification, the private key itself is NEVER sent via the network. Instead, the local ssh program use the private key to compute a message, sometimes called digital signature, which is sent to the server. The signature is related to the key pair such that anyone who knows the public key can use the signature to verify the identity of the other party. But because the private key itself is never sent, and that the signature is only one-time generated, will change the next time, no one should be able to guess your private key based on the signature and/or the public key.

   I found this site gives a relatively clear explanation:

   [ref]

   http://www.unixwiz.net/techtips/ssh-agent-forwarding.html

   How Key Challenges Work

   

   One of the more clever aspects of the agent is how it can verify a user’s identity (or more precisely, possession of a private key) without revealing that private key to anybody. This, like so many other things in modern secure communications, uses public key encryption.

   When a user wishes access to an ssh server, he presents his username to the server with a request to set up a key session. This username helps locate the list of public keys allowed access to that server: typically it’s found in the $HOME/.ssh/authorized_keys file.

   The server creates a “challenge” which can only be answered by one in possession of the corresponding private key: it creates and remembers a large random number, then encrypts it with the user’s public key. This creates a buffer of binary data which is sent to the user requesting access. To anybody without the private key, it’s just a pile of bits.

   

   When the agent receives the challenge, it decrypts it with the private key. If this key is the “other half” of the public key on the server, the decryption will be successful, revealing the original random number generated by the server. Only the holder of the private key could ever extract this random number, so this constitutes proof that the user is the holder of the private key.

   The agent takes this random number, appends the SSH session ID (which varies from connection to connection), and creates an MD5 hash value of the resultant string: this result is sent back to the server as the key response.

   The server computes the same MD5 hash (random number + session ID) and compares it with the key response from the agent: if they match, the user must have been in possession of the private key, and access is granted. If not, the next key in the list (of any) is tried in succession until a valid key is found, or no more authorized keys are available. At that point, access is denied.

   Curiously, the actual random number is never exposed in the client/agent exchange – it’s sent encrypted tothe agent, and included in an MD5 hash from the agent. It’s likely that this is a security precaution designed to make it harder to characterize the properties of the random number generator on the server by looking at the the client/agent exchange.

   More information on MD5 hashes can be found in An Illustrated Guide to Cryptographic Hashes, also on this server.

http://flysnp.imp.ac.at/methods.html#isolation

Here is a copy of the pdf file content:

Single fly genomic DNA preparation
This is a short protocol for the isolation of genomic DNA from a
single fly for PCR applications. Of the DNA yield, 1-5µl were
used to perform PCR reactions.
Squishing Buffer (SB):
 10mM Tris-HCl pH8.2
 1mM EDTA
 25mM NaCl
 200µg/ml Proteinase K (store 4mg/ml stock solution in –80˚C and
  add 1/20 to the SB just before use)
1. Add Proteinase K to the Squishing Buffer (SB) freshly from the
stock.
2. Place each single fly in wells of a 96 well plate and mash the
fly for 5-10 seconds with the tips of a multi-pipette
containing 50µl of SB each, without expelling any liquid
(sufficient liquid escapes from the tip). Expel the remaining
SB.
3. Incubate at 37˚C for 30 minutes.
4. Inactivate the Proteinase K by heating to 95˚C for 5 minutes.
It is convenient to use the PCR machine.
5. Add 50µl of TE or H2O. Use 1µl for PLP-PCR (~200bp), or 2-5µl
for normal PCR (~1kb).

[Motivation]

What is the evolutionary force that favors the clustering of genes functioning in the same pathway?

[Hypothesis]

1. coordinated expression, perhaps to facilitate “channeling” of toxic intermediates in metabolic pathways
2. to utilize the genetic linkage to preserve co-adapted genes.
3. reflect a history with frequent horizontal transfers.

……

Perhaps no order of mammals presents us with so extraordinary a series of gradations as this–leading us insensibly from the crown and summit of the animal creation down to creatures, from which there is but a step, as it seems, to the lowest, smallest, and least intelligent of the placental Mammalia. It is as if nature herself had foreseen the arrogance of man, and with Roman severity had provided that his intellect by its very triumphs, should call into prominence the slaves, admonishing the conqueror that he is but dust.

These are the chief facts, this the immediate conclusion from them to which I adverted in the commencement of this Essay. The facts, I believe, cannot be disputed; and if so, the conclusion appears to me to be inevitable.

But if Man be separated by no greater structural barrier from the brutes than they are from one another–then it seems to follow that if any process of physical causation can be discovered by which the genera and families of ordinary animals have been produced, that process of causation is [147] amply sufficient to account for the origin of Man. In other words, if it could be shown that the Marmosets, for example, have arisen by gradual modification of the ordinary Platyrhini, or that both Marmosets and Platyrhini are modified ramifications of a primitive stock–then, there would be no rational ground for doubting that man might have originated, in the one case, by the gradual modification of a man-like ape; or, in the other case, as a ramification of the same primitive stock as those apes.

At the present moment, but one such process of physical causation has any evidence in its favour; or, in other words, there is but one hypothesis regarding the origin of species of animals in general which has any scientific existence–that propounded by Mr. Darwin. For Lamarck, sagacious as many of his views were, mingled them with so much that was crude and even absurd, as to neutralize the benefit which his originality might have effected, had he been a more sober and cautious thinker; and though I have heard of the announcement of a formula touching “the ordained continuous becoming of organic forms,” it is obvious that it is the first duty of a hypothesis to be intelligible, and that a qua-quâ-versal proposition of this kind, which may be read backwards, or forwards, or sideways, with exactly the same amount of signification, does not really exist, though it may seem to do so.

[148] At the present moment, therefore, the question of the relation of man to the lower animals resolves itself, in the end, into the larger question of the tenability, or untenability, of Mr. Darwin’s views. But here we enter upon difficult ground, and it behoves us to define our exact position with the greatest care.

It cannot be doubted, I think, that Mr. Darwin has satisfactorily proved that what he terms selection, or selective modification, must occur, and does occur, in nature; and he has also proved to superfluity that such selection is competent to produce forms as distinct, structurally, as some genera even are. If the animated world presented us with none but structural differences, I should have no hesitation in saying that Mr. Darwin had demonstrated the existence of a true physical cause, amply competent to account for the origin of living species, and of man among the rest.

……

On all sides I shall hear the cry–”We are men [152] and women, not a mere better sort of apes, a little longer in the leg, more compact in the foot, and bigger in brain than your brutal Chimpanzees and Gorillas. The power of knowledge–the conscience of good and evil–the pitiful tenderness of human affections, raise us out of all real fellowship with the brutes, however closely they may seem to approximate us.”

To this I can only reply that the exclamation would be most just and would have my own entire sympathy, if it were only relevant. But, it is not I who seek to base Man’s dignity upon his great toe, or insinuate that we are lost if an Ape has a hippocampus minor. On the contrary, I have done my best to sweep away this vanity. I have endeavoured to show that no absolute structural line of demarcation, wider than that between the animals which immediately succeed us in the scale, can be drawn between the animal world and ourselves; and I may add the expression of my belief that the attempt to draw a psychical distinction is equally futile, and that even the highest faculties of feeling and of intellect begin to germinate in lower forms of life.⁸ At the same [153] time, no one is more strongly convinced than I am of the vastness of the gulf between civilized man and the brutes; or is more certain that whether from them or not, he is assuredly not of them. No one is less disposed to think lightly of the present dignity, or desparingly of the future hopes, of the only consciously intelligent denizen of this world.

• a technical note
  the expression level was measured by fluorescence to cell volume ratio for 25,000 cells in flowcytometry and take the mean. The reason to use the ratio is that the number of transcripts, i.e. expression level, depends on not only the promoter strength but also the cell’s “developmental stage”. Naively, one would expect that a cell grows its size and linearly increases its transcript number. Therefore the logic behind this ratio measure.
• 1st result — Mig1 binds cooperatively
  

  Hill equation with a Hill coefficient of 3.4 and K 5 1.8 (red) fits the observed data (blue) well, compared to a Hill equation with a Hill coefficient of 1 (green).

• Role of weak binding sites
  It’s a bit different than what I used to understand. Here is my current (I think correct) understanding: weak sites, when nearby a strong site, doesn’t get promoted to be a strong site, at least according to the model. This is what I used to believe what they suggest. But in fact, they estimated a free parameter that measures the ratio of affinity of MIG1 protein to the weak site to its affinity to a strong site, which gives 6.7. They claim that this is in the 95% CI of the PWM prediction, which suggests 9 fold lower affinity for the weaker site. The reason they suggest a weak site is useful is through its cooperative binding with the strong site. ( how is this the case? ) See figure below:
  

  Plots of expression for pairs of promoters that are almost identical except that either one strong Mig1 site or two strong Mig1 sites replace one strong and one weak Mig1 site. A blue circle represents one promoter pair and the red line represents equal expression.

• Role of weak binding sites in vivo
  They have two strategies to link their findings from the synthetic libraries to the in vivo biology
  (1) compare the number of co-occurrence of weak site in promoters that harbor a strong site with that of shuffled promoters
  (2) they did a slightly trickier comparison: they found 359 promoters that contain at least one strong Mig1 site from all yeast promoters. They then applied their thermodynamic model through all the promoters, rank them by the predicted Mig1 repression and take the top ranking 359. As a test, they had 136 documented Mig1 target genes (perhaps by some microarray assays). They use this as a reference set and ask how much could the two 359 promoters sets overlap with the 136 biologically defined targets. In the end, their thermodynamic model predicted set predicts 8 more promoters out of the 136 (41 vs 33). This demonstrates that the thermodynamic model captures more features of repression than a simple look for strong hit.

The conditional fixation time can be longer on average for a weakly selected mutation (positively) than a neutral mutation. (Martin Novak comment: this might be because we are calculating the “conditional fixation time”. In such cases, most of the neutral mutations don’t make it to fixation. Those that do make it have a certain average time between arise and fixation. For weakly selected mutations, the effect of selection is weak and therefore the difference in time is not large. However, when mutations are weakly selected, they are more likely to go to fixation, however also hang around for a longer time.)

How To Compare Directories in Unix

June 9th, 2008 | Basic stuff, Unix

ubuntu$ find /tmp/dir1 /tmp/dir2
/tmp/dir1
/tmp/dir1/file1
/tmp/dir1/file2
/tmp/dir1/dir11
/tmp/dir1/dir11/file11
/tmp/dir1/dir11/file12
/tmp/dir2
/tmp/dir2/file1
/tmp/dir2/dir11
/tmp/dir2/dir11/file11
/tmp/dir2/dir11/file12
/tmp/dir2/file3

ubuntu$ diff /tmp/dir1 /tmp/dir2
Common subdirectories: /tmp/dir1/dir11 and /tmp/dir2/dir11
diff /tmp/dir1/file1 /tmp/dir2/file1
1d0
< New line
Only in /tmp/dir1: file2
Only in /tmp/dir2: file3

ubuntu$ diff /tmp/dir1 /tmp/dir2 | grep Only
Only in /tmp/dir1: file2
Only in /tmp/dir2: file3

ubuntu$ diff --brief /tmp/dir1 /tmp/dir2
Common subdirectories: /tmp/dir1/dir11 and /tmp/dir2/dir11
Files /tmp/dir1/file1 and /tmp/dir2/file1 differ
Only in /tmp/dir1: file2
Only in /tmp/dir2: file3

ubuntu$ diff --recursive --brief /tmp/dir1 /tmp/dir2
Files /tmp/dir1/dir11/file12 and /tmp/dir2/dir11/file12 differ
Files /tmp/dir1/file1 and /tmp/dir2/file1 differ
Only in /tmp/dir1: file2
Only in /tmp/dir2: file3