Psicothema, 2002. Vol. Vol. 14 (nº 2). 233-238

Hilda Gambara and Orfelio G. León

The aim of this work is to study the effects of training in analysis and decision making on the way a multiattribute decision task is carried out. We analyzed the quantity of information subjects used and the strategies they followed. In addition, we studied whether a particular initial key or marker produced pre-decisional bias in decision makers. Furthermore, special effort was made to control the different importance assigned by decision makers to attributes, through the use of a «personalized conflict» task. The main results were: subjects with training analyze significantly more information than subjects without it; they devote more attention to the alternative eventually selected, but they do so more as a process of consolidation than as pre-decisional bias, and use compensatory strategies. Manipulation of the marker did not result in significant effects on the dependent variables.

Formación y Sesgo Pre-decisional en una tarea de decisión multiatributiva. El objetivo de este trabajo es el estudio de la formación en análisis y toma de decisiones sobre la forma de abordar una tarea de decisión entre alternativas multiatributivas. Se analizó la cantidad de información utilizada y las estrategias seguidas. Adicionalmente, se estudió si una determinada clave inicial provocaba un sesgo pre-decisional en los decisores. Se realizó un especial esfuerzo por controlar la importancia que los decisores asignan a los atributos, elaborando una tarea de «conflicto personalizado». Los resultados principales fueron: los sujetos con formación analizan significativamnte más información que los sujetos sin ella; atienden más a la alternativa finalmente elegida, pero lo hacen más como un proceso de consolidación que como un sesgo pre-decisional y utilizan estrategias compensatorias. La variable marcador no reflejó efectos significativos.

It is currently assumed that decision behaviors are highly sensitive to the task and to contextual factors, and it is also accepted that decision makers use strategies in a way contingent in the decision problem, attempting to compensate for precision in the decision by means of effort. This is the main thesis of Payne, Bettman and Johnson (1993) in their now classic book «The Adaptative Decision Maker». In this adaptative process decision makers may ignore information that is potentially important for the problem, use irrelevant information or distort the information toward a particular alternative; Payne et al. (1993) propose that in order to improve decisions it is necessary to favor the degree of fit between task demands and subjects’ information-processing capacities. This can be achieved either by reducing the effort required or by helping the decision maker to increase the precision of his/her choice, avoiding bias. The first alternative implies reducing the tasks’s cognitive effort demands; the second implies either using external aids or training decision makers. There is abundant literature on how modifications in a task that reduce effort lead to an improvement in decisions (e.g., Iglesias, de la Fuente and Martín , 2000; Russo and Shoemaker, 1989; Yates and Estin, 1996). There is also a wealth of research on the use of external aids (e.g., Gambara and León, 1988; Humphreys and Mc Fadden, 1980; Todd and Benbasat, 2000; von Winterfeldt and Edwards, 1986). However, researchers have paid less attention to the specific effects of training for improving decisions (Klein, 1997), despite the fact that important work has been carried out on the effectiveness of training in the solution of problems in general (Nisbett, 1993). As far as decision making is concerned there are two types of training. The first of these emerges from the tradition of heuristics and biases, and attempts to teach methods of elimination of biases, or «debiasing». Thus, for example, in order to avoid overconfidence, subjects are forced to consider all the alternatives (Koriat, Lichtentein and Fischhoff, 1980). The main weak point that has been pointed out for this strategy is its poor potential for generalization to real situations. A second type of training has focused on normative models or on decision analysis. This type of training has, at least in certain conditions (structured tasks), demonstrated its effectiveness (Means, Salas, Crandall and Jacobs, 1993; Zakay and Wooler, 1984). For example, in a decision task with uncertainty, Hansen and Helgeson (1996) compared the decisions of subjects with and without training in statistics/probability. This comparison was based on the strategies followed, the information used and the time devoted to the task, and was made using the Mouse-Lab program (Johnson, Payne, Schkade and Bettman, 1986), which permits monitoring information processsing. The results indicated that subjects without training made riskier decisions, were more likely to use non-compensatory strategies and acquired more information on items irrelevant to the task.

The main objective of this work is to analyze the incidence of training on subjects’ decision making process in a multiattribute decision task. In contrast to the studies cited above, we shall not focus on training in specific debiasing strategies or on the training of some component of the decision making process (e.g., training in probability), but rather on the possible effect of a more general training in decision making and decision analysis. Specifically, this training consists in a 40-hour Analysis Decision course for first-degree students. For our purposes, it would be relevant to establish whether students are capable of generalizing their knowledge beyond the classroom context, and whether, having acquired the tools for analyzing a decision, they change their way of dealing with prototypical decision tasks.

Thus, if the course has fulfilled its objectives, if it has been effective, the way subjects with training approach a multiattribute decision task should be different from the way decision makers without training deal with it. How do subjects normally approach a decision task? Svenson (1992,1999) describes decision making as a process of «Differentation-Consolidation». From very early stages of the decision process the decision maker begins to differentiate between a possible candidate and the rest of the alternatives. This leads to different treatment as regards the attention paid and the information gathered in relation to that candidate; the subject attempts to «create an alternative that is sufficiently superior in comparison to its competitors» (Svenson, 1996, p. 254). Svenson (Op.Cit.) proposes this process for all decision tasks, but especially for those where there is a conflict between alternatives, as some attributes favor one alternative and others favor another. The decision maker begins by identifying a series of markers that indicate where to begin the solution of the problem and that can convert one particular alternative into a candidate. The existence of a candidate from the beginning of the processing will mean that the structuring of the information is focused on that alternative, which is gradually tested to see whether it can serve as the final choice. On the basis of this differentiation, the decision maker consolidates his/her «pre-decision», increasing the distance between his/her candidate and the remainder of the alternatives. Consequently, the probability of an alternative eventually being chosen will be greater if it has been selected as a candidate in previous stages. The fact that in the differentiation phase the candidate can be supported, favoring it in a biased way, is congruent with the phenomenon that decision literature has identified as pre-decisional bias or candidate heuristic (Gambara and León, 1988; Montgomery, 1983, 1989; Montgomery and Willem, 1999; Russo, Medvec and Meloy, 1996). This heuristic is shown with the eventual choice of an alternative that was considered as «possible» or «candidate» from the earliest stages of the decision process, and may produce distortion of the information assessed and lead to analysis of more information from this candidate to the detriment of the other alternatives. Another consequence of this entire process is that decision makers do not need to examine all the available information in order to reach their final decision (e.g., Dahlstrand and Montgomery, 1984; Elstein and Bordage, 1979; Payne et al., 1993).

Thus, our first hypothesis is that, in a multiattribute decision task, subjects with training will perform differently from subjects without training, in that they will analyze all the information available.

A second hypothesis is that subjects that have taken training in decision analysis will not present an attention bias toward one alternative from the initial stages of the decision process, and will thus avoid biasing the information.

Lastly, given that one of the keys to this differentiation-consolidation process is the existence of a marker that acts as a guide from the initial stages of decision, we shall analyze whether subjects with training, as against subjects without training, are resistant to biasing the information of a candidate if we induce the existence of a marker. Thus, we shall manipulate this variable, attempting to cause the decision maker to become more involved with a certain alternative from the early stages of the task -that is, favoring the existence of a marker from the start of the decision process.

Considerations with regard to the task: personalized conflict. We used as experimental task a prototypical decision problem, represented by a matrix of alternatives by attributes, in which one attribute favors one alternative and another favors a different one. When we present a matrix of attributes x alternatives, for example for choosing a flat, we try to achieve that the information is compensated between the different values of attributes; thus, a small flat would be traded off by a low price, while a high price would be traded off by more space. However, although these types of task are defined by the conflict inherent in them, a problem that should not be ignored is that they fail to take account of the «individual differences» in such a conflict. Starting out from the same decision structure for all decision makers implies considering that the task has the same level of conflict for all, and that therefore the attributes have the same importance for all subjects; we know, of course, that this is not the case. In order to overcome this problem, we need to know for each subject his or her order of importance with regard to the attributes involved in the task, thus enabling us to construct a task with personalized conflict (León, 1997), as explained in the Procedure section.

In sum, our question is whether subjects that have been trained on a decision analysis will follow the same pattern of decision making as lay subjects in a multiattribute decision task. To this end we shall analyze the differences between decision makers with and without training in: the quantity of information analyzed, the attention paid to the different alternatives, the candidate heuristic and the strategies followed.

Method

Students from the Autónoma University of Madrid. In the first phase of the study a total of 69 students participated: 40 without training in decision analysis and 29 with such training. In the second phase we worked with 39 of the 69 participants from the first phase: 17 with training and 22 without. The students without training were all in their first year of a Psychology degree. The proportion of women in this group was 70%. Those trained were enrolled on a variety of degree courses (Law, Economics, Psychology, Philosophy, Computer Engineering, Tourism) and the proportion of women was 61%.

The first independent variable is «training», with two levels: with and without training in decision analysis. It is a quasi-experimental variable. Training consisted in a 40-hour Decision Analysis course for first-degree students, in which participants begin by studying the characteristics of the decision maker (heuristics and biases), moving on to focus on the Decision Analysis procedure, with and without uncertainty (construction of decision trees or value trees). A course of this type, as well as contributing to students’ academic training, should improve their decision making abilities. All the participants in the training groups passed the final exam in this subject, thus confirming that the independent variable was effectively manipulated. The content of the subject studied by participants can be found in León (2001).

The second independent variable was «marker»; also with two levels: with and without marker. This variable was manipulated by means of the instructions. Half of the subjects, chosen at random, were given instructions that included an alternative of special relevance for them. The aim of this manipulation was to induce a marker that would guide them towards the candidate alternative.

The dependent variables studied were: 1. Quantity of information examined (number of items considered before reaching a decision) 2. Attention paid to the alternatives (number of information items seen for the chosen alternative and for the non-chosen ones) 3. The candidate heuristic (percentage of subjects in which the alternative that is the candidate in the first phase coincides with that eventually chosen, this first phase being defined as covering the first 10 dimensions seen).

The task consisted in the selection of a student, from a group of six, to whom a grant would be awarded. The information was presented in a matrix of alternatives x attributes (6x6). Presentation was made by means of the program Mouse Lab 6.0 (Johnson, Payne, Schkade and Bettman, 1996). In order to take into account the fact that the attributes do not have the same importance for all subjects, a personalized conflict task, as described below, was used.

In this first phase subjects indicated the importance for them of a set of attributes, as selection criteria for potential recipients of a grant. The procedure in this first phase was as follows: subjects were asked collectively (but in two separate groups: those with and those without training) to put six attributes in order of importance. In the instructions they were told to imagine themselves in the role of a member of the postgraduate grants selection committee of a prestigious United States university, and place in order of importance the six attributes normally considered: Grades, Research project, Level of English, Experience abroad, Research experience and Head of department’s report. For each subject, a set of 6 profiles of equally attractive candidates was prepared. This was incorporated in the Mouse-Lab program, and a diskette produced for each decision maker with the Mouse-Lab and his or her personalized matrix (see table 1).

Two weeks after the first phase we proceeded to the actual experimental task: the choice of a candidate for a postgraduate grant, using the Mouse-Lab program. The task was carried out collectively in the faculty’s computer rooms, but once again in two groups, one of subjects with training and one of those without. Each subject was given his/her diskette with the program and his or her personalized matrix. Subjects were also given the corresponding instructions, according to the experimental condition to which they had been assigned (with or without marker).

The instructions stated the following: «Imagine you are a member of the committee that awards scholarships to a prestigious United States university. In recent years the majority of candidates have been selected according to faculties: sometimes Philosophy students have been selected, sometimes Law students, sometimes Medical students, and so on. The idea is to select the best candidate independently of the faculty to which he or she belongs. In the final phase of the selection process there will be six candidates remaining. What we want you to do is analyze information on each student until you reach a decision about who should receive the grant». For the situation «with marker» the following was added to the instructions, if the subject was from Psychology: «The objective is to select the candidate independently of the faculty to which s/he belongs; nevertheless, in recent years, coincidentally, no candidates from Psychology have been selected». This comment was modified according to the faculty from which the subject came. In this way, we aimed to induce the decision maker to begin by focusing his or her decision on his/her colleague. This manipulation is similar to that used by Russo, Medvec and Meloy (1996) in their work on distortion of the information in a decision task between products.

Before beginning the task, and in order to familiarize subjects with the program, they practiced with the Mouse-Lab in a decision task with flats. During the task, after each ten items of information seen, subjects indicated the candidate they would select based on the information they had so far. The task was considered to have finished when the subject chose a candidate to receive the grant. Mean duration was approximately 45 minutes.

Results

For the data analysis, of the 17 subjects with training one was eliminated as an atypical case; of the 22 subjects without training, two were eliminated: one because of data recording problems and another because he did not understand the task.

Because groups with and without training were not random groups, we first checked that there were no differences in the dependent variables between psychology students and the rest of the students in the training group.

A. Quantity of information examined. On checking the assumptions before proceeding with a 2x2 ANOVA (training x marker) for analyzing the data, we detected that the assumption of homogeneity was not fulfilled. Due to the great variability of this dependent variable, the most appropriate measure of central tendency is the median (see Table 2 and Figure 1). We found that the training group from the decision analysis course analyzed more information (Mdn = 77.00) than the untraining group (Mdn = 50). Comparing the groups using the non-parametric Mann-Whitney U Test, we obtained, with a significance level of 0.05, significant differences for the variable training U = 126, p = .020. These data support our first hypothesis, that subjects with training analyze more information.

With regard to the variable marker, no statistically significant differences were found, U = 162, p = 0.210.

B. Quantity of information examined by alternatives. We next analyzed the difference in quantity of information examined by participants for the chosen alternative and for the non-chosen alternatives. Once again, given the non-fulfillment of the homogeneity assumption, we applied non-parametric tests.

Table 3 and Figure 2 show the quantity of information analyzed for the alternative eventually chosen and the most-attended non-chosen alternatives, for each group. Differences between the means of the two groups (with and without training) for attention paid (number of dimensions seen) to the alternative eventually chosen was statistically significant, U = 87.50, p <.05; this was not the case when comparing the training and untraining group for the rest of the non chosen alternatives.

Statistically significant differences were also found in attention paid to each one of the alternatives for the group with training: Friedman test, X² (5) = 70.550, p <.001, and in attention paid to each one of the alternatives for the group without training: X² (5) = 85.520, p<.001.

As it can be seen, the alternative eventually chosen is that to which most attention is paid by groups with and without training, to a greater extent in the case of the former group. In order to ascertain whether this difference in attention paid leads to an attention bias in the decision process, we must investigate whether it already occurs in the early stages of the process. That is, we must observe whether the candidate heuristic is produced.

For the variable marker, no significant differences were found between groups, for any case. Table 4 shows means and medians of attention for each alternative.

C. Analysis of candidate heuristic. For this analysis we calculated the coincidence between the candidate alternative after the first 10 trials (when 28% of the information had been seen) and the alternative eventually selected. For all subjects the candidate alternative coincided with the chosen alternative in 45% of cases. By groups, we observed similar percentages, with no significant differences. Given the variability observed in the quantity of information seen by subjects (between 32 and 130 items), we decided to calculate the percentage of the amount of information seen by each subject that corresponded to the first coincidence with the eventually-chosen alternative. This coincidence, in the total of subjects, occurred when subjects had seen 30% of the information (median). No statistically significant differences were found by groups.

Given that no candidate heuristic was found, the above-described differences between the group with training and the group without must be located at another point in the decision process. With the aim of exploring this aspect, we proceeded to a more qualitative analysis of the strategies followed by the subjects.

D. Strategies followed by the decision makers. The majority of subjects with training observed a precise order in their analysis of the matrix information. 78% of subjects analyzed all the attributes for all of the alternatives. Moreover, the majority of them combined this inspection of the information with a strategy by alternatives (63%). Finally, many of them made a pairwise comparison in order to confirm their final choice (72%). Only one appeared to follow a non-compensatory strategy. For subjects without training, we found that 40% followed strategies by attributes, analyzing, first, all attributes for all alternatives; 45% of subjects followed a strategy by alternatives, and only one subject completely combined the two types of strategy. There was a group of subjects (54.5%) that appeared at one point to have become lost, using no clear strategy until they began to follow a certain order.

Discussion

The main objective of this research was to study the effect of training in decision analysis, comparing the performance of subjects with and without such training. The study was carried out using a multiattribute decision task, in which participants had to select the best candidate for receiving a grant, in accordance with a series of criteria or attributes. We controlled individual differences due to the conflict inherent in this type of task through the construction of what we have called matrix with personalized conflict.

Summarizing the results, first of all, it was verified that subjects with training analyze more information than subjects without training; though all participants examined all the information available. The form of presentation of the task, by means of a computer program, will certainly have made it easier to discover the information than with an information board. Secondly, the data show that subjects with training pay more attention to all the alternatives. Moreover, the alternative eventually chosen is that to which most attention is paid, with statistically significant differences between attention paid to this alternative and attention paid to the others. Thirdly, this attention paid to the alternative eventually chosen does not reflect the candidate heuristic, either for the group with training or the group without. A more qualitative analysis of the data reveals that subjects with training follow more ordered strategies than those without training. Finally, with regard to the manipulation of the variable marker made with the intention of guiding the decision process and observing whether subjects with training resisted its effect, no significant differences were found in relation to any dependent variable.

Considering these results as a whole, we can conclude that subjects with training use more information than subjects without it, following a precise order in their examination of this information through the application of a clear strategy during the decision process, and even combining several strategies, which leads them to pay more attention to the candidate they eventually choose. Subjects without training examine a smaller quantity of information, and follow more disorganized strategies. Like subjects with training, they too pay more attention to the alternative eventually selected. However, we consider that this result should not be interpreted in the same way for the two groups. Returning to Svenson’s (ops.cit.) theory of Differentiation-Consolidation, this greater attention is produced in subjects with training in the final stages (that is, in the consolidation phase), more than in the initial stages (as a process of differentiation), so that it would not be reflecting a candidate heuristic or pre-decisional bias. Rather, it appears that what subjects do is attempt to make sure of what they are doing. Although we did not find evidence of a candidate heuristic in subjects without training either, we did observe that when greater attention is paid it is not due to a process of consolidation, as it does not occur at the end. On the other hand, in relation to the candidate heuristic, in similar tasks, in which participants had to choose an apartment, a coincidence between the candidate alternative and that eventually chosen was found in a percentage considerably higher than in our task when more than 25% of the information had been seen (54% in Dahlstrand and Montgomery, 1984, and 84.61% in Gambara and Léon, 1988). In our study this phenomenon was not found. This discrepancy in results may be due to the fact that it was easy to examine the information with the computer program, and to the type of information presented in the dimensions (more difficult to retain, more quantitative), which complicates the discrimination of one candidate alternative from the others. It was already pointed out that the way the information is presented affects judgment and choice (Payne et al. 1993).

As far as the lack of initial differentiation between alternatives is concerned, this was not modified by the manipulation of the variable marker, which we thought may guide the decision making process. Although a slight manipulation for involving the subject more with a given candidate leads to the analysis of more information in general, and specifically in relation to the alternative eventually chosen, these differences do not reach statistical significance. Despite basing our manipulation of the variable marker on that of Russo et al. (1996), it was not sufficient to produce significant differences among the groups. One line of research that lies open would involve the study of keys or markers that may or may not act as guides in the decision process.

Finally, some brief comments on the role of training in decision making; Klein (1997) suggests that training based on the methods of decision analysis (normative methods) may be useful in certain conditions: when the elements of the decision problem are well-specified, when time-pressure is low, when decision makers are not experts and, particularly, in tasks with conflict. All of these characteristics are present in the task we have used. However, according to Klein (0p. Cit.), this type of training has its limitations when the task is not well-structured, in complex situations, with time-pressure and uncertainty, in changing conditions and in a context in which the group or organization must be taken into account.

In brief, in this study we have found that superior knowledge of decision analysis leads subjects to deal with the decision task in a way that differs from the approach of those with less knowledge. This knowledge was reflected in their different approach to the task of selecting candidates, which was more systematic and organized, in an analysis of all the information available, and in a «consolidation» of the process, as though they were trying to «ensure» that what they had decided, they had decided well.

Acknowledgements

This research was supported by grant PB97-0041 from Programa Sectorial de Promoción del Conocimiento (M.E.C).

Dahlstrand, U. and Montgomery, H. (1984). Information search and evaluative processes in decision making: A computer based process tracing study. Acta Psychologica, 56, 113-123.

Elstein, A.J. and Bordage, G. (1979). Psychology of Clinical Reasoning. En G. Stone y N. Alder (Eds.), Health Psychology, San Francisco, Jossey-Bass.

Gambara, H. and León, O.G. (1987). Uso de la información en el proceso de toma de decisiones. Boletín de Psicología, 17, 83-99.

Gambara, H. and León, O.G. (1988). Decisiones óptimas vs. decisiones espontáneas: la toma de decisiones con alternativas multiatributivas. Cognitiva, 1, 229-310.

Hansen, D.E. and Helgeson, J.G. (1996). The effects of statistical trianing on choice heuristics in choice under uncertainty. Journal of Behavioral Decision Making, 9, 41-57.

Humphreys, P.C. and Mc Fadden, W. (1980). Experiencies with MAUD: Aiding decisions structuring through reordering versus automating the composition rule. Acta Psychologica, 45, 51-69. Trad. Castellana: Experiencias con el MAUD: ayudas al decisor mediante técnicas de estructuración versus técnicas autodozimantes. Estudios de Psicología, 1987, 30, 125-139.

Iglesias, S., De la Fuente, E.I. y Martín, I. (2000). Efecto de las estrategias de decisión sobre el esfuerzo cognitivo. Psicothema, 12, 267-272.

Johnson, E.J., Payne, J.W., Schkade, D.A. and Bettman, J.R. (1986). Monitoring information processing and decision: the Mouse-Lab system. Unpublished manuscript, Center or Decision Studies, Fuqua School of Business, Duke University.

Johnson, E.J., Payne, J.W., Schkade, D.A. and Bettman, J.R. (1996). A mouse based sotware system for doing consumer behavior and decision making reserach. Unpublished manuscript, Center or Decision Studies, Fuqua School of Business, Duke University

Klein, G. (1997). Developing expertise in decision making. Thinking and Reasoning, 3, 337-352.

Koriat, A., Lichtestein, S. and Fischhoff, B. (1980). Training for calibration. Organizational Behavior and Human Performance, 26, 149-171.

León, O.G. (1997). On the death of SMART and the birth of GRAPA. Organizational Behavior and Human Decision Processes, 71, 249-262.

León, O.G. (2001). Tomar decisiones difíciles (2ª ed.). Madrid: Mc Graw-Hill.

Means, B., Salas, E., Crandall, B. and Jacobs, O. (1993). Training decision makers for the real world. In G.A. Klein, J. Orsanau, R. Calderwood and C.E. Zsambok (Eds.), Decision making in action. Models and methods (pp. 306-326). Norwood, NJ: Ablex Publishing Corporation.

Montgomery, H. (1983). Decision rules and the search for dominance structure: Towards a process model of decision making. In P.C. Humphreys, O. Svenson and A. Vari (Eds.), Analyzing and aiding decision processes (pp. 343-369). North Holland: Amsterdam.

Montgomery, H. (1989). From cognition to action: the search for dominance in decision making. En H. Montgomery and O. Svenson (Eds), Process and structure in human decision making (pp. 23-49). Chichester, England: Wiley.

Montgmery, H. and Willen, H. (1999). Decision making and action: The search for a good structure. In P. Juslin and H. Montgomery (Eds.), Judgment and decision making. Neo-brunswikian and process-tracing approaches. Londres: LEA.

Nisbett, R.E. (1993). Rules for reasoning. Hillsdale, NJ: Lawrence Erlbaum Associates Inc.

Payne, J.W., Bettman, J.R. and Johnson, E.J. (1993). The adaptative decision maker. Cambridge: Cambridge University Press.

Russo, J.E., Medvec, V.H. and Meloy, M.G. (1996). The distorted of information during decisions. Organizational Behavior and Human Decision Processes, 66, 102-110.

Russo, J.E. and Shoemaker, P.J.H. (1989). Decision traps (ten barriers to brilliant decision making). New York: Dobleday.

Svenson, O. (1992). Differentiation and consolidation theory of human decision making: A frame of reference for the study of pre and post-decision processes. Acta Psychologica, 80, 143-168.

Svenson, O. (1996). Decision making and the search for fundamental psychological regularities: what can be learned from a process perspective? Organizational Behavior and human decision processess, 65 (3), 252-267.

Svenson, O. (1999). Differentiation and consolidation theroy: Decision making processes. Before and after a choice. In H. Montgomery and O. Svenson (Eds), Process and structure in human decision making (pp. 23-49). Chichester, England: Wiley.

Todd, P.T. and Benbasat, I. (2000). Inducing compensatory information processing through decision aids that facilate effort reduction: An experimental assessment. Journal of Behavioral Decision Making, 13, 91-106.

von Winterfeldt, D. and Edwards, W. (1986). Decision analysis and behavioral research. Cambridge: Cambridge University Press.

Yates, J.F. and Estin, P. (1996). Training good judgement. Paper presented at the annual meetings of the Society for Judgement and Decision Making. Chicago, IL. November.

Zakay, G. and Wooler, S. (1984). Time pressure, training and decision effectiveness. Ergonomics, 27, 273-284.