working paper: RISK ASSESSMENTS BY EXPERT PURCHASING PROFESSIONALS: PROCESS AND PERFORMANCE
[Interested in the full version? Let me know]
abstract:
We consider the judgments by professional managers and analyze:
-their performance relative to managers with less experience
-whether the way in which they judge differs from that of less experienced managers
-whether different ways of judgment lead to better performance
Combining survey-data and conjoint-analysis in an experiment with 367 managers we find that experienced managers do not perform better than amateurs (and worse than a computer model). Experienced managers are more certain about their judgments, but not faster or otherwise different. Performance increases with certainty. We suggest that in this management context there is no process-performance paradox.
Long abstract:
“[E]xpert judgments in most clinical and medical domains are no more accurate than those of lightly trained novices. […] And expert judgments have been worse than those of the simplest statistical models” (Camerer and Johnson, 1991: 203). The mediocre performance of experts can perhaps be explained by their decision-making strategies: there are signs that these really are different. Experts search less, and apply more knowledge and more “configural rules” (ibid. 203-204). However, in many fields, including management, the direct performance of expert decision-makers is not easily ascertained. These are precisely the fields in which researchers have also focused on the characteristics that typify those we call experts, and their strategies for making assessments and decisions (Glaser & Chi, 1988; Shanteau, 1988; Shanteau & Peters, 1989; Ericsson & Smith, 1991; Shanteau & Stewart, 1992; Ericsson & Charness, 1994; Ericsson & Lehmann, 1996). For example, Shanteau, Weiss, Thomas &
Pounds (2002) report that besides the characteristics mentioned earlier, experts are more self-assured, assess and decide with less difficulty (relying for a large part on their intuition), are faster and more consistent, and generally demonstrate a greater degree of consensus. It is even suggested in some publications that in the absence of objective performance criteria, these characteristics ascribed to experts in fact define the expert. We therefore see that the great majority of studies into the performance of experts is actually based on research within clinical and medical domains. We expand this research in the direction of risk assessments in the management of transactions.
Our contribution addresses the following questions:
[Performance] How good are expert judgments and decisions compared with those of amateurs and elementary statistical models, if we focus on risk assessments in a management context?
[Process] How much difference is there between experts and amateurs in their strategies for making assessments and decisions (degree of certainty, degree of reliance on intuition, use of information, speed, consistency, consensus), if we focus on risk assessments in a management context?
[Process and performance] To what degree do observed differences in the way experts assess and decide correlate with the performance of the experts?
To answer these questions, we first performed a series of five experiments. These entailed presenting concrete cases in the field of IT product procurement to professional managers as well as novices and amateurs. Each time, the participant was asked to predict how problematic the case would be, and exactly what types of problems could be expected to occur (N=367 participants; k=2,936 assessed cases). The difference with standard conjoint analysis is that in this instance, these are all ‘real-life’ transactions: every case submitted is drawn from an existing database that was compiled during an earlier survey study (Buskens & Batenburg, 2000). This means that for each of the transactions submitted for assessment, we know in advance whether there really were problems, what type of problem(s), the severity of the problem(s), etc. (Rooks 2002, Rooks, Raub & Tazelaar 2006). In these assessment experiments we have access to a criterion variable that is known to a high level of det
ail, something that is regularly absent from research into the decisions of experts in a management context. This enables us to establish the capability of each of the various types of subjects in making risk assessments, and to compare these to the performance of a simple computer model based on an identical set of data (Dawes, Faust & Meehl, 1993: 356).
We continued to build on this method of data collection in a sixth experiment. An important difference is that in this experiment (N=244 participants; k=3,904 cases) we have subjected all participants to two assessment tests. One based on 8 cases per person, with a fixed number of attributes per case (either 7 or 14), and one based on 8 cases but with the participant able to decide for himself how much extra information per case he needs to make a judgment (max 14; MouseLab-type method). Every participant in this experiment was further asked to state how much relative importance he assigned to each of the 14 case attributes presented. In addition, this sixth experiment incorporated time registration for every participant.
Our results suggest the following:
[Performance] Our results on the performance of experienced managers are roughly in line with those in other fields of expertise and robust across our experiments. We find that (a) judgments of professional managers are meagre at best, and (b) certainly not better than the judgments by less experienced managers or even amateurs. Furthermore, (c) neither general nor specific human capital of managers has an impact on their performance, and (d) a simple formula outperforms the average (and the above average) manager even when the formula only has half of the information as compared to the manager (cf. Meehl, 1954, 1986; Dawes, 1971; 1979; Dawes, Faust & Meehl, 1993; Grove & Meehl, 1996; Grove, Zald, Lebow, Snitz & Nelson, 2000, or our own Snijders, Tazelaar & Batenburg, 2003; Tazelaar & Snijders, 2004).
[Process] We do find some differences in the way in which managers make their judgments. Experienced managers are more certain about their judgments, both before and during the experiment. Some managers claim to base themselves to a large extent on intuition and gut feeling, but this is not related to the experience managers have. These ‘gut-decision-makers’ suggest that they base themselves on less information before they judge or decide, but if we take a look at whether this actually happens, we see that this is not the case. Contrary to what is sometimes found in the literature, we do not find that experienced managers (or gut-decision-makers) are faster or more efficient judges. The only correlation we find with speed is certainty of the judge: those who are more certain at the time of judgment, are the ones who have taken more time. Managers with a higher level of general education show somewhat higher internal consistency on our tests. Years of experience or specific hu
man capital do not correlate with internal consistency. It seems that of the process characteristics generally ascribed to experienced managers, the only one that holds up in the expected direction under close scrutiny, is certainty.
[Process and performance] Performance on our test depends to a large extent on three main effects: certainty, speed, and ‘model-consistency’, where ‘model-consistency’ is defined as the correlation between the model of the expert and the expert judgments. Those who are more certain receive higher test-scores, just like the managers who are more model-consistent, and the managers who use more time for the test. An interesting finding is that the negative effect of a high speed on the performance only holds for those who are uncertain or have a relatively low general education level. A second conspicuous finding is that the effect of being a gut-decision-maker has a slightly negative effect on average, but a positive effect for those with a lot of specific expertise. Taken together, this implies that if expertise has an effect on performance, in our case this most likely runs through an increased certainty (expertise -> more certain -> better performance). One other way in whi
ch this might occur, and worthy of further study, is through the usefulness of intuitive judgments with increasing expertise (expertise -> intuitive judgments are better -> better performance).
|
