Abbreviations used

Static benchmarking of membrane helix predictions

Andrew Kernytsky ^1,* & Burkhard Rost ^{1, 2, 3}

This article is published in (Nucleic Acids Research, issue, date and pages) © copyright Oxford University Press (2003). OUP is the only authorised source. All copying of this article including placing on another website requires the written permission of the copyright owner.

Abstract

Prediction of trans-membrane helices continues to be a difficult task with a few prediction methods clearly taking the lead; none of these is clearly best on all accounts. Recently, we have carefully set up protocols for benchmarking the most relevant aspects of prediction accuracy and have applied it to over 30 prediction methods. Here, we present the extension of that analysis to the level of an automatic web server evaluating new methods (http://cubic.bioc.columbia.edu/tmh_benchmark/). The most important achievements of the tool are: (1) any new method is compared to the battery of well-established tools. (2) The battery of measures explored allows spotting strengths in methods that may not be 'best' overall. In particular, we report per-residue and per-segment scores for accuracy, and the error-rates for confusing membrane helices with globular proteins or signal peptides. An additional feature is that developers can directly investigate any hydrophobicity scale for its potential in predicting membrane helices.

Key words: genome sequence analysis, predicting globularity, protein domains, protein structure prediction, solvent accessibility, multiple alignments, transmembrane helices, bioinformatics.

Introduction

Membrane spanning proteins are vital for cells to function [1, 2] . However, it is very difficult to experimentally determine high-resolution three-dimensional structures for these proteins: fewer than 50 structures are currently deposited in PDB [3, 4] . C-terminal fusions with indicator proteins [5, 6] and from antibody-binding studies [7, 8] reveal the location of the helices and the orientation with respect to the protein termini. We refer to these data - slightly incorrectly - as 'low-resolution structures'. Mšller, Apweiler, and colleagues at the EBI have carefully hand-selected the results from low-resolution experiments for about 500 proteins [9] . We have taken the high-resolution set from PDB, added the low-resolution set from the EBI and have filtered the noise from redundancy of very similar proteins by creating the largest possible sub-set that is chosen such that we cannot infer structural similarity between any pair of proteins in that set from sequence alone ('sequence-unique subset [10, 11] ). Our previous work gives a comprehensive evaluation of transmembrane helix prediction methods based on these sets [11, 12] . Our automatic benchmarking server accomplishes this by using several different evaluation criteria for evaluating accuracy of the prediction method against proteins of known high and low resolution structure. False positives are estimated by applying the method to signal peptides and proteins without trans-membrane helices.

Methods

Input. The server accepts two types of input from users: (1) Simple scales reflecting the propensity of residues to form membrane helices, e.g. hydrophobicity scales. Such scales for each amino acid are either uploaded in text format or entered into a form. (2) Results from novel prediction methods: developers can benefit from the benchmark server by following the following steps: (i) download the data sets from our web site (about 2200 proteins, some of which contain membrane helices, some do not), (ii) run your method on all proteins, (iii) upload the predictions to our server in either of two commonly accepted formats. The upload is checked for possible problems that are immediately communicated to the developer. For example, if the predictions contains three-state Ð helix / non-helix / possible-helix Ð predictions rather than two state Ð helix / non-helix Ð predictions, the user is give a choice to abort the operation, convert all possible-helix residues to non-helix, or convert all possible-helix residues to helix.

Algorithm. When we test hydrophobicity scales, we simply apply the Wimley-White algorithm turn such scales into predictions of membrane helices [13] . New prediction methods (or predictions) are evaluated directly from the data uploaded by the users. A detailed description of the particular scores and schemes explored to measure performance is beyond the scope of this manuscript; they are available in our original publications or on our web site [11, 12] .

Output. Submissions are tracked through identifiers (IDs) that are shown on all result pages. After the request is queued, the user can either refresh the results page to check the status of the request, or follow the link that is e-mailed to the provided e-mail address when the request is completed. When the results are ready (approximately 3-5 minutes), the user is presented with several tables showing how well the tested method/scale performs in comparison to established methods (Table). Results are given separately for (1) high- and (2) low-resolution membrane proteins, as well as for the discrimination against (3) globular proteins and against (4) signal peptides ( Fig. 1 ). In all four resulting tables, several columns show different measures for prediction accuracy and discrimination. Clicking on the column headers will resort the given table by that metric. Clicking on the question mark (?) in the column header names will give a description of the metric. Clicking on the other prediction servers in the row header will give the full name of the server as well as the citation for the source as well as a web link to the server if available. Although the primary format used is the interactive web document described here, the results can also be obtained in non-interactive format. If desired, the results will be e-mailed in text format along with the link to the interactive results. Additionally, a permanent, non-interactive web document can be generated on the server by clicking a link on the interactive web page. A link to the document is then provided which the user can use to reference the serverÕs results.

Fig. 1. : Sample for server output. Example of one of the four output tables from server showing where the tested method falls in the ranking of existing methods. Table shown is for accuracy of predicting helices as tested against known high resolution structures. The method named YOU is highlighted in the output and shows the result for tested method. Hyperlinks re-sort by the given column and also lead to column (scoring metric) and row (prediction method) descriptions.

>>>Table 1<<<

Discussion

Standard point of reference. The primary goal of this server is to provide users, developers, and referees with a standard benchmark evaluation for helical trans-membrane prediction methods in a format that is publicly available and as convenient as possible. The tool may help all not to over-estimate performance, and/or to spot strengths and weaknesses of particular methods. The battery of measure for performance that we use encompasses almost all the scores that found applied in the literature. With the web server, any new algorithm can be tested instantly and seamlessly.

Downside of static benchmark: over-fit to do well on this set only. We might argue that a possible problem with such an easily available and comprehensive method is that someone with enough time on their hands could write a program searching the space of all possible hydrophobicity-like scales in order to optimise the performance on our sets, more generally, developers may over-fit their models. In fact, to some extent, this is a principle problem of any standard data set accepted in the community. However, we challenge that if the scale/method really does consistently better than all methods in respect to all scores, it may indeed capture important aspects of helical membrane proteins. Perhaps more probable is the possibility that one may accidentally overfit to the benchmark by testing against the benchmark several times during development. To that end, developers can at least reduce the risk of fooling themselves by first testing their final or nearly final method on their own data sets and by then investigating to what extent their results are confirmed by ours.

Ultimate solution: go dynamic. Nevertheless, there is only one way to completely solve the problem, namely test on proteins that could not have been used to develop methods since their experimental structures arrived after the method. This is the concept that we explore through our EVA server evaluating the performance of structure prediction for globular proteins [14, 15, 16] . However, for globular proteins every week tens of new structures appear in PDB. Although, this will not become reality for membrane proteins in the foreseeable future, we are currently investigating ways of embedding some dynamic system for the evaluation of membrane predictions into EVA.

Acknowledgements

Thanks to Jinfeng Liu and Megan Restuccia (Columbia) for computer assistance; to Chien Peter Chen (Columbia) for his in-depth analysis of membrane helix predictions. Particular thanks to Volker Eyrich for his crucial help with setting up the META-PP and EVA servers without which most of the results presented here would not exist. Thanks to the anonymous reviewer for very detailed, constructive comments. This work was supported by the grants RO1-GM63029-01 from the National Institute of Health (NIH) and 1-R01-LM07329-01 from the National Library of Medicine (NLM). Last, not least, thanks to Amos Bairoch (SIB, Geneva), Rolf Apweiler (EBI, Hinxton), Phil Bourne (San Diego Univ.), and their crews for maintaining excellent databases and to all experimentalists who enabled this tool by making their data publicly available.

References