For a substantial investment of this nature, program evaluation is obligatory.  The criterion for success is simple: how many meaningful breakthroughs or scientific revolutions has it spawned? This should be the sole criterion for success, and if the answer is few or zero, then the program should be seriously revamped.

 

How can one determine the number of revolutions? Certainly, this cannot be done by the usual standards of counting papers. A more effective approach is to compare the prevailing paradigm at the time of initiation of thematic funding with the prevailing paradigm at the time of evaluation, thereby determining the extent of shift. This could be done through examination of the literature and through interviews with members of the relevant scientific community. In some cases the shift should be obvious by inspection. The discoveries of prions and ulcer-causing bacteria H. pylorii for example resulted in phase-transition-like shifts in thinking, which would be relatively easy to confirm.

 

At what stage should the evaluation be conducted? Considering the duration of the ramp-up period, and the time required for revolutions to unfold, a reasonable time might be five to ten years. By that time many of the projects will have matured, and ample time will have elapsed for determining whether the program has succeeded. At that point, decisions could be made about desirable changes in the Institute’s approach.