EVA measures for secondary structure prediction accuracy

Author: Burkhard Rost rost@columbia.edu

Contents

Standard-of-truth (conversation of observed secondary structure)
Per-residue accuracy
Per-segment accuracy
Accuracy of predicting structural class and secondary structure content
References

Standard-of-truth

PDB -> secondary structure
Observed secondary structure is taken primarily from the program DSSP (reference). However, we shall (soon) run STRIDE additionally (reference).
Conversion of DSSP secondary structure
The following chart illustrates, how the 8 states from DSSP are converted to three secondary structure states:

DSSP H G I E B T S ' '

used H H H E E L L L

Per-residue accuracy

The following scores are used to measure per-residue accuracy (reference, note: most tables contain only a subset of these scores):

Prediction accuracy matrix:

note: the total number of residues observed in state i is:

note: the total number of residues predicted in state i is (helix, strand, other)

and the total number of residues is simply:
Three-state prediction accuracy: Q₃

Thus, the three-state per residue accuracy Q₃ becomes:
Per-state percentages:

To define accuracy for a particular state (helix, strand, other), there are two possible variants. These answer to the following questions.
- How many of the observed helix residues (strand|other) were correctly predicted?
  Given are the correctly predicted residues as percentage of all residues OBSERVED in a particular state ('%obs').
- How many of the predicted helix (strand|other) residues were correctly predicted?
  Given are the correctly predicted residues as percentage of all residues PREDICTED in a particular state ('%prd)
Information index:

The information index is given by:

where P_obs describes the probability for finding one particular string of N_res residues with obs_i residues being in structure i out of all combinatorial possible ones, and P_prd is the probability for a particular realisation of the prediction matrix {M}. The resulting information index is:
Matthew's correlation coefficients are defined by the following formulas:

Per-segment accuracy (SOV)

The Segment OVerlap measure for prediction accuracy is defined by (for more details):

Per-stage segment overlap:

with the following definitions:

S1 and S2       are the observed and predicted secondary structure segments (in state i, which can be either H, E or C)
LEN(S1)         is the number of residues in the segments S1
MINOV(S1;S2)    is the length of actual overlap of S1 and S2, i.e. the extent for which both segments have residues in state i, for example H
MAXOV(S1;S2)    is the length of the total extent for which either of the segments S1 or S2 has a residue in state i
DELTA(S1;S2)    is the integer value defined as being equal to the following

		

THE SUM         is taken over S, all the pairs of segments {S1;S2}, where S1 and S2 have at least one residue in state i in common
N(i)            is the number of residues in state i defined as follows: 

		

		The two sums are taken over S and S':
S(i)            is the number of all the pairs of segments {S1;S2}, where S1 and S2 have at least one residue in state i in common
S'(i)           is the number of segments S1 that do not produce any segment pair

Segment OVerlap quantity measure for all three states:


where the normalization value N is a sum of N(i) over all three conformational states (i = HELIX, STRAND, COIL)

Accuracy of predicting structural class and secondary structure content

Structural classes derived from secondary structure content:

Proteins can be classified into 'structural' classes according to their secondary structure content (reference). To evaluate how well one of the four major classes:

all-alpha, all-beta, alpha-beta, other
is predicted, the following classification scheme is used (based on modifications of papers by Zhang & Chou and Kneller et al. ):

class protein length percentage H percentage E

all-alpha > 60 > 45% < 5%

all-beta > 60 < 5% > 45%

alpha-beta > 60 > 30% > 20%

other other other other

NOTE: the table is to be read as an if alpha, elsif beta, elsif alpha-beta, else mix.
For these classes the overall four-class accuracy is reported (number of proteins correctly predicted in one of the four classes / total number of protein).
Four-state structural class accuracy:
The percentage of proteins correctly predicted in one of the above four classes is simply given by:
Difference between observed and predicted secondary structure content:
Because of technical constraints (build up of averages on flight), the accuracy of predicting the secondary structure content is measured by the simple difference between observed and predicted (reference):

Note: the abbreviations
- contDH
- contDH
are used on the EVA pages (for omega_i; w)
Pearson correlation coefficient for secondary structure content:
A more reasonable score to measure the accuracy of predicting content is the Pearson coefficient (note: this score may be added at some later stage):