Psicothema, 2006. Vol. Vol. 18 (nº 1). 123-129

Roser Sala, Rosario Granero and Lourdes Ezpeleta

The aim of this study is to obtain dimensions from a categorical diagnostic interview. 512 children aged 8 to 17 attending public centers for children’s mental health and presenting some form of psychological disorder were interviewed with the «Diagnostic Interview for Children and Adolescents»(DICA-IV). Analysis of the principal components indicate that a two-dimensional model (externalizing-internalizing) satisfactorily explain the data collected from the children and their parents. The data from the adolescents are better represented by a three-dimensional model (attention problems-internalizing-antisocial behavior). The factor scales show good internal consistency and significantly relate to other criteria of psychopathology and incapacity (concurrent validity). These results show that it is possible to use DICA-IV to obtain categorical and dimensional indicators simultaneously. The latter are particularly sensitive for measuring changes over the course of disorders.

Análisis dimensional de una entrevista diagnóstica categorial: la DICA-IV. El objetivo de este estudio es obtener dimensiones a partir de una entrevista diagnóstica categorial. Se entrevistó con la Entrevista Diagnóstica para Niños y Adolescentes a 512 participantes de entre 8 y 17 años de centros de asistencia primaria pediátrica o psiquiátrica que presentaban algún tipo de psicopatología. El análisis de componentes principales indica que un modelo bidimensional (exteriorizado-interiorizado) explica los datos recogidos de los niños y sus padres. Los datos de los adolescentes quedan mejor representados por un modelo de tres dimensiones (problemas de atención-interiorizados-comportamiento antisocial). Las escalas factoriales muestran una buena consistencia interna y se relacionan de forma significativa con otros criterios de psicopatología e incapacidad (validez concurrente). Los resultados indican que es posible obtener simultáneamente indicadores categoriales y dimensionales de esta entrevista. Estos últimos pueden ser especialmente útiles para medir cambios en el curso de los trastornos.

Traditionally, conceptualization of child psychopathology has been dominated by a categorical perspective based on clinical practice. This position is reflected in the Diagnostic and Statistic Manual of Mental Disorders-Fourth Edition (American Psychological Association, 1994) and the International Classification of Disease-Tenth Edition (World Health Organization, 1992). At the same time, there is a psychometric perspective which enjoys considerable success (Krueger, Caspi, Moffit and Silva, 1998). These two approaches have today provided two different taxonomies of child disorders: «clinical syndromes», originating from hypothesis on the covariation of symptoms deriving from clinical observations of the patients (Wakefield, 1999), and «empirical syndromes» generated from statistical covariation between symptoms but without a priori conceptions of their grouping (Achenbach, 1991).

The categorical model has been very useful in providing the profession with a common language. However, one of the major criticisms made is that there is little empirical evidence for it, as it is based on the consensus of the experts.

The dimensional system provides a classification for psychological problems based on the quantification of attributes (Costello, 1994) that indicates the degree of intensity of the characteristics the subjects present in various dimensions. Their flexibility should be remarked as the main advantage of dimensional models, as they allow different cut-off points to be defined depending on the objectives of the evaluation (diagnosis, prediction, prevention or treatment). They also enhance and facilitate the representation of their subject’s idiosyncratic character rather than forcing them to fit into predefined categories, which makes them more coherent with what is observed in clinical practice (Haynes and O’Brien, 1990). They are more parsimonious than categorical systems and solve the critical factor of severity. Quantitative scales make it possible to identify much more homogeneous groups of subjects as regards their clinical-psychological state.

Although these two approaches reflect different traditions (clinical-categorical versus empirical-dimensional), they have not developed in isolation. In recent years a number of researchers have remarked that the integration of clinical and psychometric perspectives could help improve the classification and assessment of psychopathology (Cantwell, 1996; Clark, Watson and Reynolds, 1995; Kamphaus and Frick, 1996; Waldman, Lilienfield and Lahey, 1995). Certainly, there are examples such as Child Behavior Checklist (Achenbach, 1991) where the symptoms and indicators of both approaches have been successfully linked.

Nowadays the structured interview is considered the point of reference for psychopathological evaluation both in the clinic and in research, but it is often not used to its full potential because it is applied only to evaluate the presence or absence of disorders. A few authors have derived dimensional scales on the basis of the information collected from structured diagnostic interviews in order to integrate the two approaches, as using symptom counts in structured interviews to represent psychopathology makes considerable sense both from clinical and research perspectives. Cerel and Fristad (2001) identify two methods for creating dimensional scales from the Diagnostic Interview for Children and Adolescents-Revised (DICA-R). The first method, the simplest, allows a scale to be obtained, the «DICA-SUM», which is the sum total of the symptoms present in the subjects. The second scale, «Behavior, Anxiety, Mood and Other», produces a general weighted score, and consists of dividing the number of symptoms present in the subject by the total of symptoms possible for each disorder. The said authors also provide convergent and discriminant validity for the scales obtained. Lucas et al. (2001) derive two scales (parents and children) from the Diagnostic Interview Schedule for Children (DISC 2.3), from the most predictive questions for each specific diagnosis. Rubio-Stipec et al. (1996) obtained, via confirmatory factor analysis, 4 scales (depression, attention deficit hyperactivity disorder, oppositional defiant disorder and behavior symptoms) for the items that evaluate these 4 disorders in the DISC 2.3. They show that these continuous measures have high test-retest reliability, good internal consistency and a strong relationship with established categorical diagnosis.

The objective of this work is the dimensional analysis of the information collected with a semistructured categorical interview, the Diagnostic Interview for Children and Adolescents (DICA-IV; Reich, Leacock and Shanfeld, 1997) and the evaluation of internal consistency and validity of the factors obtained. The possibility of using this instrument for simultaneously obtaining categorical and dimensional classifications will allow us to benefit from the advantages of both approaches in research as well as in clinical practice.

Method

Participants

The sample comprised 380 children between the age of 8 and 17, from the Barcelona area, who were attending public outpatient primary care services for mental health and 132 subjects attending outpatient pediatric consultation that were representative of children attending public health services in educational, social and cultural variables. Mental health patients were interviewed as a part of their own diagnostic process. Outpatient pediatric patients were volunteers treated for minor physical problems, not suffering any chronic disease. All of these were included in the study because they presented at least one psychological syndrome as evaluated with DICA-IV. The mean age was 13.2 years (SD= 2.6) and 53.5% were male (n= 274). Following the Hollingshead’s (1975) classification, 58.7% were from middle to lower middle, 33.3% lower and only 8% upper middle and upper socio-economic backgrounds. 97.9% of the subjects were of European Mediterranean ethnic origin. Subjects evidencing mental retardation or other generalized development disorders were excluded from the study.

Most prevalent disorders in the sample were oppositional defiant disorder (ODD) (46.3%), major depression (MD) (44.5%), attention deficit hyperactivity disorder (ADHD) (35.9%) and general anxiety disorder (33.4%). 14.3% presented conduct disorder (CD). 77% presented comorbidity; 23% of the children presented a single DSM-IV disorder, 21.9% presented two and the remaining 55.1% presented three or more disorders.

Material

To evaluate the presence of psychopathology the DICA-IV (Reich et al., 1997) was used, adapted for Spain and computerized (Granero and Ezpeleta, 1997; Ezpeleta, de la Osa, Doménech, Navarro and Losilla, 1997; Ezpeleta, de la Osa, Júdez, Doménech, Navarro and Losilla, 1997; de la Osa, Ezpeleta, Doménech, Navarro and Losilla, 1997; Ezpeleta, Granero, Osa, Doménech and Guillamón, 2002). The DICA-IV is a semi-structured diagnostic interview which evaluates the disorders most prevalent in children and adolescents, following DSM-IV criteria (APA, 1994). It comprises three versions: DICA-C for children aged 8 to 12, DICA-A for adolescents aged 13 to 17 and DICA-P for parents. To create the dimensional scales four sections of the DICA-IV interview were chosen: ADHD, ODD, CD and MD. These disorders were selected for two reasons: in the first place, disorders that were highly prevalent in the sample were required, since for analysis it was necessary for the items to have variability. Secondly, the size of the available sample and the characteristics of the statistical analysis that was to be applied limited the study of further items. In precise terms, 64 direct questions of the interview were included (21 on ADHD, 8 on ODD, 15 on CD and 20 on MD) which collected information of symptoms during the last year. Some of the questions on conduct disorder were excluded from some analyses as insufficient affirmative answers were available.

The «Children’s Global Assessment Scale» (CGAS, Shaffer et al., 1983) was used to evaluate the concurrent validity of the dimensions found in the DICA-IV. The scale, which evaluates the functional impairment the symptoms produce, was scored on a scale of 1 (maximum impairment) to 100 (excellent functioning). The CGAS score is assigned at the end of the interview, and considers all the diagnostic information available. Scores above 70 indicate normal adjustment. The CGAS has obtained, in different studies, good interrater and test-retest reliability, as well as significant correlations with other psychopathological measures (Ezpeleta, Granero and de la Osa, 1999). For this study the lowest level of functioning in the last 12 months was chosen.

Conduct problems empirical scales of the CBCL (Achenbach and Rescorla, 2001) were also used as a criterion with which to evaluate the concurrent validity of the factors obtained with the DICA-IV.

Procedure

Having obtained the informed written consent of the parents and the verbal consent of the child to participating in the study, two different interviewers previously trained in the use of the instruments, interviewed the parents and the child separately and simultaneously. Once the information had been obtained from the DICA-IV, each interviewer assigned a CGAS impairment score, evaluating the lowest degree of functional adjustment in the previous 12 months. At the same time the parents were asked to respond to the CBCL questionnaire. Interviews with each of the informants in the sample had a mean duration of 80 minutes.

The subject’s answers in the DICA-IV interview were treated dichotomously (problem present vs. absent).

Three different exploratory principal components analyses were made: one for the answers from children (8 to 12 years), another for the answers from adolescents (13 to 17 years) and a third for the information provided by parents.

In order to create the dimensions, the Categorical Principal Components Analysis (CATPCA) was used with the SPSS program, version 11.0 for Windows. To retain an item in a factor two criteria were applied: a) the absolute value of the weighting equal to or higher than 0.30; and b) a minimum relative difference between two weightings of 0.10 if the item weighted higher than 0.30 in more than one factor. If criteria a and b were not fulfilled, the item was excluded from all dimensions.

Only those dimensions that were clinically interpretable and contained a minimum of six items with factor weights above 0.30 were considered acceptable. It was also considered important that eigenvalues were higher than 1, that the solution chosen should explain a sufficiently high percentage of variability and that the model should be as parsimonious as possible.

To measure concurrent validity, Pearson’s correlation coefficient was calculated between the dimensions appearing in factoring and the CGAS and CBCL scores.

Results

In the three analyses made, the four- and five-component solutions were immediately rejected because they had very low factor weightings and were difficult to interpret from a clinical point of view. In the one-dimension solution there were notable saturations for most of the questions included in the analysis. However, in all cases, the two- and three-dimension solutions provided additional information that was clinically relevant.

In the analysis of children the two-dimension solution explained the 28.7% of the total variability (Table 1). The first dimension, termed Externalizing Problems, contains 37 items and includes ADHD and ODD items, and some CD items. The second component, Internalizing Problems, contains 14 items and includes questions relating to MD, except two, «increase in appetite» and «early waking insomnia», which weighted more clearly in the first dimension. As 5 items did not weight significantly in any dimension and 3 others weighted in both dimensions, these 8 questions were not retained for any factor.

In the analysis of the adolescents (Table 1) the three-dimension solution is the one that best represents the data. The first component groups the MD items and some ODD items that are generally related with depressive irritability: «loses temper», «argues with adults», «easily annoyed by others» and «often angry and resentful». It also includes a single ADHD item («loses things»), which could also be related to depression because it has to do with a lack of attention and concentration, and two CD items, which correspond to behavior that is quite common in adolescents («steel without confrontation» and «truant from school»). In all, 23 items are included under Internalizing Problems. The second component brings together 11 questions, principally lack of attention ADHD items. It also includes one ODD and one CD item. The factor thus brings together Attention Problems. Finally, in the third dimension there are significant factor weightings for 6 items (which were not weighted in any of the previous components) that referred to Antisocial behavior. This three-dimensional solution explains 33.3% of the total variability. In this case, 6 items had no weight in any dimension and 17 weighted in more than one, so they were not retained for any factor.

In the analysis of principal components with information provided by the parents (Table 1) a first component of Externalizing Problems was obtained, containing high saturations for the ADHD and ODD items, and for three CD items. The second component, which contains Internalizing Problems, has notable factor weightings for the MD items (except «increase in appetite»). This two-dimensional model explains 29.4% of the variability of the original data and includes 51 questions. It is a very stable model as no item weights significantly for more than one factor. The items with no weight in any of the dimensions were those referring to serious CD problems.

From the non-weighted sum of the items dichotomously coded 0 or 1 (0: Problem absent; 1: Problem present) the subjects’ scores on each of the scales obtained from factor analysis have been calculated. Their internal consistency has been analyzed using Cronbach’s Alpha, and the results are from very good to excellent (Table 1).

Table 2 contains the Pearson correlation coefficient (together with the corresponding 95% confidence interval) between the scores on the DICA-IV and CGAS scales. All the scales associate in a way that is statistically significant with the CGAS scores. In addition, all the dimensions maintain a negative linear association with impairment, as was foreseeable: the highest CGAS scores correspond to the best adjusted individuals (with lower scores on the dimensions and, thus, less psychopathology). In general the correlations are moderate. The scale that most notably associates with the CGAS is Externalizing Problems for children, with a value r= -0.56.

The DICA-IV scales are associated positively and significantly with most CBCL scores (Table 3). In general, the Externalizing Problems, Attention Problems and Antisocial Behavior scales on the DICA-IV relate to the externalizing score on the CBCL. Similarly, the DICA-IV Internalizing Problems scales relate to the internalizing score on the CBCL. Nevertheless, there is one exception: Internalizing Problems for children has no linear association with any of the scales in the broad CBCL bands. Most of the scales have a significant relationship with the total CBCL score.

As regards the current CBCL scales, except for children’s analyses, the DICA-IV Internalizing Problems factor is positively and significantly associated with the CBCL’s Anxiety/Depression scale. It is related with Somatic complaints in adolescents and parents, and with the Withdrawn/Depressed scale in parents. In children’s analyses, Internalizing Problems is not related with any of CBCL scales.

In all cases, the Externalizing Problems (Attention Problems with adolescents) factor relates with Attention problems and Aggressive behavior, with the highest correlation always being on the Attention problems scale. For adolescents and parents Externalizing problems also correlates with Rule-breaking behavior. For parents Externalizing problems are also related with Thinking problems, Social problems and with the Anxiety/Depression scale, although with lower correlation indices.

The Antisocial behavior scale (for adolescents) relates particularly with the Rule-breaking behavior scale on the CBCL. This scale correlates positively with the Aggressive behavior score and with Attention problems.

Discussion

By applying factor analysis to the answers provided by children and parents in the DICA-IV interview, two and three dimensions were obtained (internalizing, externalizing or attention problems and antisocial behavior) which have good internal consistency (reliability) and are valid as they significantly relate with other clinical criteria (CGAS and CBCL).

The factor groupings obtained are consistent with the results obtained by other authors (Achenbach and Edelbrock, 1978, 1984; Cantwell, 1996; Krueger et al., 1998), who demonstrate the existence of two general dimensions for explaining psychological problems in children: externalizing and internalizing. Although both dimensions have shown moderate stability over time (Ollendick and King, 1994), suggesting the possibility that this two-dimensional structure is maintained until adulthood, the results obtained in the present study indicate that, with the sections analyzed, for adolescents, a three-dimensional structure that separates attention problems and antisocial behavior would fit the data better. The distinction between ADHD and CD has previously been documented by such authors as Fergusson, Horwood and Lloyd (1991), who stress that despite being different dimensions they are closely co-related and possibly share etiological bases.

The fact that the Antisocial Behavior dimension should emerge from the information from adolescents and not in that from children or parents could be due, for one thing, to the fact that the most reliable informants on such acts of aggression or law-breaking are the adolescents; clinical practice shows that parents tend to ignore acts of vandalism their children are responsible outside the home. Apart from that, adolescence is the age at which such behavior tends to occur and, absent during childhood, it thus does not emerge in factor analysis.

The models obtained are fairly stable as regards factor weightings; there is, however, a series of items that weight for more than one component. This is explained by the fact that the factors are not independent, so that the items with secondary weight in other dimensions are unspecific indicators for the factors and constitute the shared part of the dimensions. This lack of independence of the factors is a reflection of psychopathological reality. In fact, one of the most controversial issues in psychopathology is understanding comorbidity, which is, on the other hand, the rule rather than the exception.

The items not significantly weighting in any dimension are questions with little variability in their answers, either because they are not common in the sample («uses weapons», «physical cruelty to people», «physical cruelty to animals», «destroys others’ property», «increase in appetite») or because they are very prevalent items («blames others of own mistakes»). Hartman et al. (2001) also report this in their factor analysis to test the construct validity of DSM-IV, and indicate that these are evident symptoms of CD and that the low saturations for the said items in the factors is due, simply, to the lack of variability in the data.

The associations between the scales obtained by factoring and the CGAS are statistically significant, but in general moderate. It should be remembered that scoring the CGAS takes into account the general psychopathology of the child or adolescent and, in particular, it evaluates how this interferes in daily life. Thus the CGAS score considers a much larger number of problems than have formed part of our analysis. Even though CGAS is not an independent observation because it is based partially on the responses to some of the items that went into forming the factor scales, it is possible to use the CGAS score as a validation criterion, since impairment is a different construct to that of psychopathology (Angold, Costello, Farmer, Burns and Erkanli, 1999; Pickles et al., 2001).

Likewise, the relationship between the DICA-IV scales and the CBCL scores is significant but moderate. The CBCL is an inventory collected from parents so that, in some cases, informer bias may appear in correlating the information obtained from children with that provided by their guardians. Additionally, the CBCL is also a broad questionnaire on child psychopathology which evaluates a larger number of problems than those that have been included in analysis with the DICA-IV.

The results obtained are generalizable to children with psychopathology attending public health services in Barcelona. This project is a first attempt to derive reliable, valid dimensional scales from the DICA-IV. There are several practical uses of these psychometrically defined scales. As their main advantage, it should be noted first that they are a way of making better use of the information collected in structured interviews. They mean that the DICA-IV can be used simultaneously to obtain categorical and dimensional indicators. The latter are particular sensitive for measuring changes during the course of disorders, consequently they would be of particular use in longitudinal studies in order to measure precise changes in psychopathology (as a result, for example, of therapy or the natural evolution of psychological states). They also allow for the initial identification of children with sub-clinical symptoms who may be at risk of suffering impairment (Cerel and Fristad, 2001). Jensen and Watanabe (1999) believe that subjects scoring above a cut-off point on a scale, although they may reveal no categorical diagnosis, are at greater risk of problems in school, concentration difficulties, inability to learn, or of needing mental health services. Although we could use the total items counts in the disorder symptoms, defined in DSM-IV, as continuous measures, the purpose of this study was to obtain empirical information that could only be provided by dimensional analyses. Important advantages were obtained through carrying out these empirical analyses, although we restricted ourselves to a limited number of items, some of which may not have appeared in the factor analyses, due to their low frequency in the sample. We reduced information exclusively to the important items, and obtained information about the gathering of psychological problems not grouped together in DSM-IV. Dimensional indicators must be derived from empirical analyses, not from counts of what is rationally created. DICA-IV scales therefore add important information to categorical diagnoses such as severity criteria, as well as information on sub-clinical problems.

It has to be said that the principal limitation of this study is the size of the sample which, although it is adequate for the disorders included, has prevented the inclusion of a larger part of the DICA-IV interview in the psychometric analysis. Nor did it permit separate analysis according to sex, which would have allowed to value if the factor dimensions for boys and girls are identical. Finally, to facilitate the use of the dimensions obtained, further studies are needed of the general population to permit norms and/or cut-off points to be obtained.

Acknowledgements

This work was supported by grant DGICYT PM98-173 of the Ministry of Education and Culture and grant BS02002-3850 of the Ministry of Science and Technology, Spain.

Achenbach, T. M. (1991). Manual for the Child Behavior Checklist/4-18 and 1991 profile. Burlington, VT: University of Vermont, Department of Psychiatry.

Achenbach, T. M. and Edelbrock, C. S. (1978). The classification of child psychopathology: a review of analysis of empirical efforts. Psychological Bulletin, 85(6), 1.275-1.301.

Achenbach, T. M. and Edelbrock, C. S. (1984). Psychopathology of childhood. Annual Review of Psychology, 35, 227-256.

Achenbach, T. M. and Rescorla, L. A. (2001). Manual for the ASEBA School-Age Forms & Profiles. Burlington, V.T.: University of Vermont, Research Center for Children, Youth & Families.

Angold, A., Costello, E. J., Farmer, E., Burns, B. J. and Erkanli, A. (1999). Impaired but undiagnosed. Journal of the American Academy of Child and Adolescent Psychiatry, 38, 129-137.

APA (American Psychiatric Association) (1994). Diagnostic and Statistical Manual of Mental Disorders (4th ed.). Washington, DC: Author.

Cantwell, D. P. (1996). Classification of child and adolescent psychopathology. Journal of Child Psychology and Psychiatry, 37, 3-12.

Cerel, J. and Fristad, M. A. (2001). Scaling structured interview data: a comparison of two methods. Journal of the American Academy of Child and Adolescent Psychiatry, 40, 341-346.

Clark, L. A., Watson, D. and Reynolds, S. (1995). Diagnosis and classification of psychopathology: challenges to the current system and future directions. Annual Review of Psychology, 46, 121-153.

Costello, C. G. (1994). Two dimensional views of psychopathology. Behaviour Research and Therapy, 32, 391-402.

Ezpeleta, L., Granero, R. and Osa, N. de la. (1999). Evaluación del deterioro en niños y adolescentes a través de la Children’s Global Assessment Scale (CGAS). Revista de Psiquiatría Infanto-Juvenil, 1, 20-26.

Ezpeleta, L., Granero, R., Osa N. de la, Doménech, J. M. and Guillamón, N. (2002). Perception of need of health and use services in children and adolescents. Do they share the same predictors? Psicothema, 14, 532-539.

Ezpeleta, L., Osa, N. de la, Doménech, J. M., Navarro, J. B. and Losilla, J. M. (1997). Fiabilidad test-retest de la adaptación española de la Diagnostic Interview for Children and Adolescents – DICA-R. Psicothema, 9, 529-539.

Ezpeleta, L., Osa, N. de la, Júdez, J., Doménech, J. M., Navarro, J. B. and Losilla, J. M. (1997). Diagnostic agreement between clinician and the Diagnostic Interview for Children and Adolescents – DICA-R in a Spanish outpatient sample. Journal of Child Psychology and Psychiatry, 38, 431-440.

Fergusson, D. M., Horwood, L. J. and Lloyd, M. (1991). Confirmatory factor models of attention deficit and conduct disorder. Journal of Child Psychology and Psychiatry, 32(2), 257-274.

Granero, R. and Ezpeleta, L. (1997). Entrevista diagnóstica para niños y adolescentes (EDNA-IV) [Computer program]. Universidad Autónoma de Barcelona.

Hartman, C. A., Hox, J., Mellenbergh, G. J., Boyle, M. H., Offord, D. R., Racine, Y. et al. (2001). DSM-IV Internal construct validity: when a taxonomy meets data. Journal of Child Psychology and Psychiatry, 42(6), 817-836.

Haynes, S. N. and O’Brien, W. H. (1990). Functional analysis in behaviour therapy. Clinical Psychology Review, 10, 649-668.

Hodges, K. (1993). Structured interviews for assessing children. Journal of Child Psychology and Psychiatry, 34(1), 49-68.

Hollingshead, A. B. (1975). Four factor index of social status. Unpublished manuscript,Yale University, Department of Sociology.

Jensen, P. S. and Watanabe, H. (1999). Sherlock Holmes and child psychopathology assessment approaches: the case of the false-positive. Journal of the American Academy of Child and Adolescent Psychiatry, 38(2),138-146.

Kamphaus, R. W. and Frick, P. J. (1996). Clinical assessment of child and adolescent personality behaviour. Needham Heights, MA: Simon & Schuster.

Krueger, R. F., Caspi, A., Moffit, T. E. and Silva, P. A. (1998). The structure and stability of common mental disorders (DSM-III-R): a longitudinal-epidemiological study. Journal of Abnormal Psychology, 107, 216-227.

Lucas, C. P., Zhang, H., Fisher, P. W., Shaffer, D., Regier, D. A., Narrow, W. E. et al. (2001). The DISC predictive scales (DPS): efficiently screening for diagnoses. Journal of the American Academy of Child and Adolescent Psychiatry, 40, 443-449.

Ollendick, T. H. and King, N. J. (1994). Diagnosis, assessment and treatment of internalizing problems in children: The role of longitudinal data. Journal of Consulting and Clinical Psychology, 62, 918-927.

Osa, N. de la, Ezpeleta, L., Doménech, J. M., Navarro, J. B. and Losilla, J. M. (1996). Fiabilidad entre entrevistadores de la entrevista diagnóstica estructurada para niños y adolescentes (DICA-R). Psicothema, 8, 359-368.

Pickles, A., Rowe, R., Simonoff, E., Foley, D., Rutter, M. and Silberg, J. (2001). Child psychiatric symptoms and psychosocial impairment: relationship and prognostic significance. British Journal of Psychiatry, 179, 230-235.

Reich, W., Leacock, N. and Shanfeld, K. (1997). DICA-IV Diagnostic Interview for children and Adolescents-IV [Computer software]. Toronto, Ontario: Multi-Health Systems, Inc.

Rubio-Stipec, M., Shrout, P. E., Canino, G., Bird, H. R., Jensen, P., Dulcan, M. et al. (1996). Empirically defined symptom scales using the DISC 2.3. Journal of Abnormal Child Psychology, 24(1), 67-83.

Shaffer, D., Gould, M. S., Brasic, J., Ambrosini, P., Fisher, P., Bird, H. et al. (1983). A Children’s Global Assessment Scale (CGAS). Archives of General Psychiatry, 40, 1.228-1.231.

Waldman, I. D., Lilienfield, S. O. and Lahey, B. B. (1995). Toward construct validity in the childhood disruptive behaviour disorders: classification and diagnosis in DSM-IV and beyond. Advances in Clinical Child Psychology, 17, 323-363.

WHO (World Health Organization) (1992). ICD-10: International Classification of Diseases and related health problems (10th revision). Geneva: Author.