EVA: evaluation of protein structure prediction servers

Ingrid Y. Y. Koh ^1,*, Volker A. Eyrich ², Marc A. Marti-Renom ³, Dariusz Przybylski ^2,4, Mallur S. Madhusudhan ³, Narayanan Eswar ³, Osvaldo Gra�a ⁵, Florencio Pazos ⁵, Alfonso Valencia ⁵, Andrej Sali ³ & Burkhard Rost ^{1, 2, 6, *}

This article is published in (Nucleic Acids Research, 31, issue, date and pages) � copyright Oxford University Press (2003). OUP is the only authorised source. All copying of this article including placing on another website requires the written permission of the copyright owner.

Abstract

EVA (http://cubic.bioc.columbia.edu/eva/) is a web server for evaluation of the accuracy of automated protein structure prediction methods. The evaluation is updated automatically each week, to cope with the large number of existing prediction servers and the constant changes in the prediction methods. EVA currently assesses servers for secondary structure prediction, contact prediction, comparative protein structure modelling, and threading/fold recognition. Every day, sequences of newly available protein structures in Protein Data Bank are sent to the servers and their predictions are collected. The predictions are then compared to the experimental structures once a week; the results are published on the EVA web pages. Over time, EVA has accumulated prediction results for a large number of proteins, ranging from hundreds to thousands, depending on the prediction method. This large sample assures that methods are compared reliably. As a result, EVA provides useful information to developers as well as users of prediction methods.

Key words: protein structure prediction, comparative modelling, inter-residue contacts, inter-residue distances, secondary structure, threading.

Abbreviations used

Introduction

Continuous, automated, large data sets, statistical significance.The goal of EVA is to evaluate the sustained performance of protein structure prediction servers through a battery of objective measures for prediction accuracy. While the bi-annual CASP meetings address the question �how well can experts predict protein structures with the help of machines?�, the question addressed by EVA is �how well can automatic servers predict protein structures?�. Conceptually, this is similar to CAFASP, but there is a major difference: EVA provides a continuous, fully automatic and statistically more significant analysis of structure prediction servers, whereas CAFASP only covers a limited number of proteins determined in a period of about four months in every two years: fewer than 10 proteins were available for the non-homology category at CAFASP3 in 2002. This implies that it is - at best - extremely difficult to infer differences of statistical significance from the CAFASP/CASP data sets. For example, the assessor for secondary structure prediction in 2002 concluded that there was no improvement in secondary structure predictions with respect to the CAFASP/CASP in 2000 although the numerical values differed by over six percentage points.

A tool for developers of prediction methods.EVA facilitates developers of structure prediction methods to improve their approaches and users of prediction servers to apply methods judiciously. The ranking of each prediction method is analyzed and updated on the web every week. Ranking is a non-trivial task because of the non-uniformity in data sets and in the measures for accuracy. Another complication is that methods are compared most reliably when they are tested under identical conditions, i.e. with identical sets of proteins [1, 2, 3] . Here, we sketch the EVA mechanisms that enable such large-scale assessment of prediction servers automatically and continuously.

Design and Implementation

Five steps from sequence to results.The analysis of prediction methods involves the following steps ( Fig. 1 ): (1) select a set of suitable test sequences, (2) apply prediction methods to those sequences, (3) assess prediction methods by measuring prediction quality using certain scoring functions, (4) determine criteria for statistically significant differences, and rank the methods accordingly, (5) merge results of the current week with those accumulated in the past, publish results on the web, and communicate with and gather results from the EVA satellites (at Centro Nacional de Biotecnologia (CNB) in Madrid, Spain, and at University of California, San Francisco (UCSF).).

Fig. 1. : Flowchart of EVA. Every day, EVA downloads the newest protein structures from PDB⁴. The structures are added to mySQL databases, sequences are extracted for every protein chain, and are sent to each prediction server by META-PredictProtein⁵ (except for threading in which only novel structures are sent). META-PP collects the results and sends them to EVA. Every week, predictions of secondary structure, threading/fold recognition, comparative modelling and inter-residue contacts are evaluated at the EVA satellites at Columbia University, University of California, San Francisco, and CNB Madrid. The central EVA site at Columbia collects all the assessments from the satellites and the results from the database searches, and publishes the updated web pages. Finally, all web pages are mirrored at the satellites.

(2) Collection of predictions. Once a day, META-PP [5, 6] submits sequences to prediction servers and collects the results. Once a week, these results are sent to EVA satellites for evaluation, namely to Columbia University for secondary structure prediction and fold recognition/threading, to UCSF for comparative modelling and to CNB for inter-residue distances/contacts.

(3) Evaluation of sustained performance. Prediction quality is evaluated using a battery of scoring functions sketched below for all four categories.

(4) Ranking prediction methods. Ranking is most reliable when prediction methods are tested under identical circumstances. The best way to rank two methods is to assess their performance based on the identical test sets. Two ranking methods are currently available in EVA. The first one is based on sub-sets of all proteins that are common to all methods. The limitations of this approach are that (i) not all methods exist at the same time and (ii) not all sequences are predicted by all methods at any given time, due to server downtime and errors. In practice, these two effects reduce the size of the common sub-sets dramatically. The second ranking approach relies on pairwise method comparisons that depend on the sub-set of proteins common to the two compared methods [3] . This pairwise ranking approach determines for each pair of participating servers whether or not it is possible to discriminate their accuracies, given the size of the test set and the particular accuracy measure used. The downside of this approach is that the overall ranking list obtained by averaging the pairwise results may be �frustrated� due to the different testing sets for the different pairs of methods.

(5) Results presented on the EVA web sites. The central EVA site at Columbia University collects either the assessments or the html pages with assessments from the satellites every week, and presents them on the web. The central EVA site is mirrored at all EVA satellites ( Fig. 1 ).

Categories of evaluation

EVA currently addresses the following protein structure prediction categories ( Table 1 ): comparative modelling (EVA-CM), inter-residue contact prediction (EVA-con), secondary structure prediction (EVA-sec) and threading (EVA-FR). In the following, we sketch the measures for accuracy employed for each category. Note that the detailed definitions of the scores are available through the EVA web sites.

 

Method	URL	Main developer(s)	Quote
Comparative modeling
3D-Jigsaw	http://www.bmm.icnet.uk/servers/3djigsaw/	PA Bates, P Fitzjohn & BC Moreira	[26, 27]
CPHModels	http://www.cbs.dtu.dk/services/CPHmodels/	S Brunak et al.	[28]
ESyPred3D	http://www.fundp.ac.be/urbm/bioinfo/esypred/	C Lambert	[29]
SDSC1	http://cl.sdsc.edu/hm.html	Shindyalov & PE Bourne	[22]
SwissModel	http://www.expasy.org/swissmod/	T Schwede, MC Peitsch & N Guex	[30]
Threading/ Fold recognition
3D-PSSM	http://www.sbg.bio.ic.ac.uk/~3dpssm	L Kelley, B Maccallum & MSternberg	[31]
BLAST	http://www.ncbi.nlm.nih.gov/BLAST	S Karlin & S Altschul	[32]
FUGUE	http://www-cryst.bioc.cam.ac.uk/~fugue/	K Mizuguchi	-
Libellula	http://www.pdg.cnb.uam.es:8081/libellula.html	-	-
Prospect	http://www.aber.ac.uk/~phiwww/prof/	Y Xu	[33]
PSI-BLAST	http://www.ncbi.nlm.nih.gov/BLAST/	S Altschul et al	[9]
SAMt99	http://www.cse.ucsc.edu/research/compbio/HMM-apps/model-library-search.html	K Karplus, C Barrett & R Hughey	[34, 35]
Superfamily	http://supfam.mrc-lmb.cam.ac.uk/SUPERFAMILY/	J Gough	[36]
Inter-residuecontacts
CORNET	http://prion.biocomp.unibo.it/cornet.html	P Fariselli, O Olmea & A Valencia,R Casadio	[37, 38, 10]
PDGCON	http://www.pdg.cnb.uam.es:8081/	F Pazos, O Olmea & A Valencia	[13]
CONcons/ CONhydro	http://www.pdg.cnb.uam.es:8081/	F Pazos, O Olmea & A Valencia	[37, 39, 38]
Secondarystructure
APSSP2	http://www.imtech.res.in/raghava/apssp2/	G Raghava	[40]
Jpred	http://jura.ebi.ac.uk:8888/	JA Cuff & GJ Barton	[41]
PHDsec	http://cubic.bioc.columbia.edu/predictprotein	B Rost & C Sander	[42]
PHDpsi	http://cubic.bioc.columbia.edu/predictprotein	D Przybylski & B Rost	[43]
PROF_king	http://www.aber.ac.uk/~phiwww/prof/	M Ouali & R King	[44]
PROFsec	http://cubic.bioc.columbia.edu/predictprotein	B Rost	[45]
PSIpred	http://insulin.brunel.ac.uk/psiform.html	D Jones	[46, 47]
SAM-T99sec	http://www.cse.ucsc.edu/research/compbio/HMM-apps/T99-query.html	K Karplus, C Barrett & R Hughey	[48, 34]
SSpro2	http://promoter.ics.uci.edu/BRNN-PRED/	G Pollastri & P Baldi	[49]

EVA-CM implements a small number of criteria - arranged hierarchically from coarser to finer - that measure the accuracy of a comparative model. The assessed aspects of a model include fold type, alignment, whole structure, core structure, loops, and side-chains. Final ranking is reported using the �pairwise� comparison of prediction servers [3] . From May 2000 to January 2003, predictions were collected from 5 different servers, resulting in 20,957 submitted models for 9,050 different PDB chains. On average, 2.3 models were predicted per chain.

EVA-con evaluates inter-residue contact/distance predictions. A number of servers predict contacts directly, using neural networks of different kinds trained on contact maps [10, 11] . There are also predictions of contacts based on assembled structures [12] . The current evaluation criteria implemented in EVA-con include: (1) accuracy - the number of the correctly predicted contacts divided by the total number of predicted contacts [13] ; (2) improvement over random - the calculated accuracy divided by the random accuracy [13] ; (3) distance distribution of the predicted contacts - the weighted harmonic average difference between the predicted contact distance distribution and the all-pairs distance distribution [14] ; and (4) delta evaluation -the percentage of correctly predicted contacts that are within a certain number (delta) of residues of the experimental contact, measured along the sequence [15] . EVA-con may also be used to evaluate ab initio, fold recognition and comparative modelling servers by transforming models into intra-molecular contacts between the corresponding C-beta atoms (C-alpha for Gly) with a 8� cut off.

EVA-sec evaluates protein secondary structure predictions. Secondary structures are assigned from 3D structures through DSSP [16] and STRIDE [17] . EVA-sec measures accuracy by: (1) per-residue accuracy [18] (Q₃) - percentage of residues correctly predicted in one of the three states (helix, strand or other), (2) per-segment accuracy [18, 19] (SOV) - average overlap between segments (methods that get most of the segment cores right are generally more useful than those that get some of the entire segments right), and (3) accuracy of predicting structural class - percentage of proteins correctly predicted in one of the following classes: all-alpha, all-beta, alpha/beta and others [20, 21] . Rankings are presented using both the �common subset� and �pairwise� comparison approaches.

EVA-FR currently evaluates models only for novel sequences (i.e., proteins for which PSI-BLAST searches do not reveal similarity to a known structure). Since there is no single measure that can comprehensively assess the quality of threading models, EVA-FR employs an array of alignment dependent and alignment independent measures [22, 23, 24] . For most of the measures used, two aspects of server performance are considered: (1) the ability to produce good models for each target (rank analysis), and (2) the ability to assign reliable scores to its models, measured through Receiver Operator Characteristics curves (ROC; note this aspect is often referred to with 'fold recognition'). Methods are ranked through both the �common subset� and �pairwise� comparison approaches.

Discussion

EVA provides an automated and continuous evaluation. Every week, test sequences are automatically submitted to prediction servers and results are evaluated and posted on the EVA web sites. The test sets are constructed so that methods could not have been trained based on the sequences in the test sets. Moreover, the test sets are as large as possible. In addition, the reliability of the comparisons between methods is maximised by using only test sets common to the methods assessed.

EVA provides supplemental information to CASP. Since 1994, the development of structure prediction methods has been influenced by the CASP meetings. While EVA uses well-defined numerical criteria to evaluate sustained performance, expert evaluations are still needed to learn what measures are most useful. However, human assessors are not likely to be able to handle many more test sequences than those at CASP. At the same time, there are problems with ranking methods based on test sets that are too small [1, 2, 3] . EVA rankings are statistically more significant than those at CASP, because EVA assesses prediction methods continuously on as many proteins every month as CASP in two years [1] . We believe that CASP needs to be supplemented by a large-scale, automated and continuous assessment, such as that by LiveBench [25] (assessment for threading methods only) and EVA. In fact, EVA may replace certain CASP categories in the future. For example, it was proposed at the last 2002 CASP meeting to eliminate secondary structure predictions from CASP. Instead, EVA-sec will replace CASP/CAFASP for users interested in those methods. This decision was partially influenced by the fact that the evaluation of secondary structure prediction methods has matured and this matured analysis has demonstrated beyond doubt that the set of proteins at CASP5 (2002) was not representative and too small.

EVA allows developers to focus on developing better methods. The best secondary structure prediction methods have reached a sustained level of 76% accuracy for the last two years [2] which indicates a substantial improvement in secondary structure prediction over the last four years. While it is always difficult to choose an appropriate set of measures, EVA uses standard criteria that have been largely used by experts in the area. For secondary structure prediction, these criteria are well established. For all other categories, we are currently experimenting with new criteria, others will be incorporated into EVA upon request from users. The precise definitions of the criteria are available on the web. (While we can make our original scripts available upon request, we currently do not have the resources to cast the whole EVA code into a form that guarantees portability or ease-of-use.) Overall, EVA allows developers to focus on the development of better methods, rather than on the generally time-consuming evaluation.

Extension of the EVA framework to other prediction categories. In principle, the concepts implemented in EVA could and should be generalised to evaluating a larger variety of prediction methods. Often, the problem is the availability of new high-resolution data. We intend to explore extensions that cover the predictions of protein-protein interactions, membrane regions, signal peptides, cleavage sites, structural/functional motifs, and sub-cellular localisation.

Acknowledgements

Thanks to Jinfeng Liu and Megan Restuccia (Columbia) for computer assistance. We are grateful to members of the Protein Design Group. The contribution of the PDG is supported in part by a grant from the Spanish Ministry of Science and Technology (PDG, CNB-CSIC). IK was supported by the grant 5-P20-LM7276 from the National Institute of Health (NIH); DP was supported by the NIH grant RO1-GM63029-01, AS, MAMR, MSM and NE by the NIH grants R01 GM54762 and P50 GM62529, BR by the NIH grant 1-P50-GM62413-01 and the NSF grant DBI-0131168. Thanks to Phil Bourne (UCSD) and the RCBS crews for maintaining an excellent PDB and to all experimentalists who enabled this analysis by making their data publicly available. Last, not least, thanks to all those developers who support EVA by going through the trouble of making their methods publicly available.

References