We discuss the outcomes of the 2019 cryo-EM Model Metrics Challenge (https://challenges.emdataresource.org). The goals of this challenge were to evaluate the quality of models that can be produced using current modelling software, and to assess the performance of metrics currently in use to evaluate cryo-EM models. Four maps in the resolution range of 1.8-3.1 Å were suggested for generating atomic models. Three of the four maps formed a resolution series from the same imaging experiment /specimen (apoferritin).
Sixty three models were submitted by 16 different modeller teams. Majority of the models (51) were built using ab initio techniques, the remaining using optimization approaches. Models were assessed using a wide panel of metrics covering geometric quality, fit-to-map, and similarity to reference structures. A web-base infrastructure was developed to present the evaluation results and facilitate the analysis of submitted models. Reassuringly, all represented modelling approaches were able to produce largely correct coordinates for each of the four map targets. Most submitted models were as good or better than the high-quality reference models chosen for comparison with each target. Automated methods were able to generate fairly complete models, needing only small amounts of manual intervention to be finalized (but some is always needed). This outcome represents a great advance over the previous challenges.
Comparing performance across different targets and metrics enabled systematic assessment of the evaluation metrics, particularly with regards to fit to the experimental map and model geometry. The results permit several specific recommendations to be made about validating near-atomic cryo-EM structures. In particular, our preliminary analysis revealed that some measures are sensitive to the presence of hydrogen atoms and realistic atom displacement parameters (B-factors) in models. By calculating pairwise correlation between all measures and clustering the results, we identified groups of measures that tend to return similar results or are most divergent from others. All evaluation scores and correlation coefficients are posted on the model evaluation system website (https://model-compare.emdataresource.org) as interactive tables and plots.
Future challenges are planned for the year 2020. There we plan to challenge participants with reconstructing protein structures from low-resolution maps. Stay tuned.