Knowledge Base

Occupational Test Administration and Use

Eatwell, J and Direen, K (1997). Occupational Test Administration and Use. In H. Love & W. Whittaker (Eds.) Practice issues for clinical and applied psychologists in New Zealand. Wellington: The New Zealand Psychological Society.

Psychometric assessments are used to enhance the quality and quantity of information available for selection, development and training decisions and as an aid to organisational change. The highest standards of practice in the use of all psychometric tools are needed in order to maximise the benefit to organisations and the individuals assessed, and to promote fairness and equality of opportunity for all.

To facilitate this the following guidelines have been produced for practitioners utilising psychometric assessment instruments. The guidelines aim to cover when, and who should use, psychometric tools, the principles in choosing appropriate tools, the preparation of candidate and administration of tests, and the issues surrounding confidentiality and storage of materials and results.

When Should Assessment Instruments Be Used?

Where competencies have been identified as required for effective job performance, psychometric assessments may be used to measure these for selection either for short listing candidates or final selection decision making, placement or promotional decisions, development, team building, counselling, outplacement and organisational development. In each case the situation must be evaluated to decide whether the use of assessment instruments would be appropriate to help achieve the desired human resource objectives.

There are some occasions where it is not usually appropriate to use psychometric assessment. For instance, it is inappropriate that such tools be used for making redundancy decisions, since direct information on job performance should already be available (such tools can be valuable in making re-deployment decisions or in outplacement counselling however). Similarly, an organisation may want to restrict the use of some assessments to counselling or development applications.

Assessment tools are best used in decision making in conjunction with other relevant information. In a promotion decision, the results from psychometric assessments may be integrated with interview performance, the candidate’s track record and a manager’s recommendations to provide the best information about an individual’s suitability. Use of a single assessment result alone should be avoided whenever possible.

Who Should Use Psychometric Instruments?

Knowledge and experience are required to use psychometric tools effectively. It is recognised throughout the world that the use of psychometric instruments by unsuitably qualified individuals can have very detrimental effects. To access materials requires both formal training in their use and often specialist training for the specific instruments in question. Qualified users should ensure that materials are only used appropriately and are not used by untrained people or for a purpose for which they were not intended. It is also their responsibility to work within the confines of their own expertise and to recognise when refresher training, skills updates or expert advice is needed.

Using Psychometric Instruments

Choice of Instrument

It is important that whenever instruments are chosen there is written documentation of the reasoning behind the choice. This may include copies of job analysis reports, job descriptions, person specifications, validation studies, etc. If the relevance of the particular measure is challenged, such evidence supports the instrument choice, shows the care taken and helps ensure users do not take inappropriate short cuts. The importance of job analysis in determining the knowledge, skills and abilities required for effective on the job performance is magnified by the Human Rights Act 1993 which makes it unlawful to make a selection decision on the basis of information that may be discriminatory or is non-job relevant.

There are three main aspects to consider when choosing a psychometric instrument: the content of the instrument; the level the instrument is aimed at; and its psycho-metric qualities (Cronbach, 1984).

i) Instrument Content:

Whenever psychometric assessments are used it is vital that there is a match between the skills and characteristics measured and the job and organisational demands. This is reinforced in the Pre-employment Guidelines Based on the Human Rights Act 1993, Section 3, where it is stated that “Employers should only request information that is clearly relevant to the requirements of the employment and to the applicant’s ability to do the job”. For instance, the instrument should not require understanding of complex vocabulary or performance at speed unless these are relevant to the job. This is particularly important when selection or promotion decisions are based on assessment results.

Objective Job Analysis is the best way to determine the skills and attributes required for a particular job. These skills are then matched to appropriate assessment tools. The more detailed the analysis of the job and the closer the match between the attribute required on the job and that measured by the instrument, the higher the content validity of the tool. Wherever possible this approach should be supported by empirical research relating assessment results to actual job performance. This information will allow users to refine their use of psychometric tools and answer the question - how relevant are they? In large scale assessment procedures it may be appropriate to perform criterion related validation studies before utilising a psychometric instrument.

The second aspect of instrument content that we should be aware of when matching assessment materials to jobs is the context in which the skill is measured. This should, as far as possible, reflect the type of content found in the job. For example, a typing test should require the typing of material similar to that required on the job. However, care must be taken not to include material requiring knowledge specific to the organisation that would, for instance, put external applicants at an unfair disadvantage. Content of an instrument that is of a more general nature should be equally accessible to all applicant groups, men and women and ethnic minorities. For instance, in the typing test, if a job relevant text is too technical for an external applicant to deal with before training, more general tests should be used.

ii) Instrument Level:

The level of difficulty at which the skill or attribute is measured should be appropriate to the job. An instrument which is too easy or difficult will not differentiate between individuals with good and poor potential. In selection the level of the instrument used should also be appropriate for the likely applicant pool. If the general level of applicants is below the level of standard required for the job, employers should consider what they can do to attract better applicants. Training or job redesign options also need to be considered. If there is a tendency for individuals from one particular ethnic group or gender to fail to meet the required standard, Section 73 of the Human Rights Act 1993 sometimes allows positive action targeted at this group in order that they may “achieve an equal place with other members of the community”. This may take the form of special training programmes.

iii) Psychometric Qualities:

Assessment instruments should be psychometrically sound. The relevant information and statistics for judging the instruments should appear in the user manual. These should include:

  • Specification of the skill or attribute the instrument measures - we need to know that the instrument we are using is actually assessing the type and level of skill or attribute required on the job.

  • Description of groups for which the instrument is appropriate (educational background, work experience etc) - we need to ensure that the instruments we use are appropriate for the individuals being assessed. This can be done by referring to the biographical details of the norm group chosen to make comparisons. For example, we would not give applicants to a clerical position a verbal reasoning test designed for university graduates.

  • Details of the development process - it is important to find out whether the instruments we use have been developed in a rigorous manner. All instruments should be developed on the basis of a thorough job analysis to ensure that they are assessing skills, knowledge or abilities necessary for effective on the job performance in the occupational area for which they are designed. For instance, each question within an aptitude test needs to have been carefully reviewed and trailed to ensure that it is interpreted in the same manner by all candidates and is non-biased. To this end, the developers of psychometric tools should have reviewed every single item within each instrument to eliminate those that do not appear to be rationally related to other items, those that are ambiguous in any manner or those that are judged unsuitable in any other way. The developers should also conduct a number of trials for each tool to ensure that they are both reliable and valid

  • Instrument reliability statistics - when interpreting assessment results, it is very important to know how consistent and therefore, how accurate the scores are. Psychometric instruments are measuring devices in their own right. They have their own errors of measurement which may cause scores to vary from one session to another. The assessee’s effort and attention may vary from time to time and changes in experience or health may have an effect. The way the instrument is administered can also influence the individual(s) being assessed. The content of the instrument itself can also be a contributor to error, although this can be reduced by eliminating poor items and ambiguous questions or statements. It is important to know how much inconsistency there is in any instrument so that this can be allowed for in the interpretation of the results.

  • Evidence of instrument validity - the validity of an instrument is concerned with the extent to which it actually measures what it has been designed to measure. For example, when thinking about the validity of a personality questionnaire for measuring say, customer service style, we would be addressing the question “how well does this questionnaire actually measure the style required on the job?” We also need to know that the instruments we use for an occupational group are assessing skills related to successful on the job performance for that group. The stronger the relationship between instrument performance and on the job performance in a particular skill or aptitude area, the greater the utility of the instrument. The validity of an instrument should be the primary consideration of any user when choosing an assessment tool.

  • Steps to guard against ethnic and gender bias - thorough analysis of an instrument’s content should take place in order to ensure that questions are free from sexist language and are equally meaningful to all ethnic groups to whom they may be administered. Research shows that for some occupational assessments differences are observed in the performance of gender and ethnic groups. The reasons for these differences are extremely complex; differences in applicants socio-economic backgrounds, their educational experience and training opportunities are likely to be reflected in their assessment results. To ensure that an instrument is not unfairly discriminating between ethnic or gender groups an assessment of on-the-job differences in performance should be conducted. If there are differences in assessment, but not on-the-job performance for each of the groups, then the use of the instrument could be unlawful (Hunter, Schmidt, and Hunter, 1979).

  • Representative and relevant norm groups

Do not judge an instrument solely by how widely it is used. An instrument can become out-dated and companies sometimes use inappropriate and out-dated assessment tools.

Preparation of Candidates and Administration of Assessment Tools

Standardisation is the fundamental concept in the utility of tests (Mischel, 1986). It is the standardised conditions, instructions, time, content, scoring and interpretation which make psychometric tools (or any assessment tool or methodology) objective.

i) Use Of Practice Materials:

Some candidates may be unfamiliar with assessment processes so it may be difficult for them to perform at their best. Others may find the assessment situation very stressful. Ethnic minority candidates in particular, may perhaps, under-perform because of the effects of educational disadvantage or race discrimination. Older candidates and those with less educational experience are also likely to suffer these sorts of problems. Practice items at the beginning of an assessment procedure can reduce the bias that may arise from differential assessment sophistication, helping some people but not others. They can also reduce nervousness by allowing a candidate to gain confidence in his/her ability to perform well in the assessment. If possible, candidates should be notified a week in advance that they will be assessed. Examples or descriptions of what assessment instruments will be like (practice leaflets) should be provided, so that candidates can familiarise themselves with the type of tasks involved (Kellett, Fletcher, Callen, & Geary1994). Such practice increases the effectiveness of the assessment proper by giving an accurate measure of a candidate’s style or ability.


ii) Administration of Psychometric Instruments:

The administration instructions are extremely important and must always be strictly adhered to. Only qualified persons should administer assessment tools. Abuse of procedures described in the instrument manual can lead to bias and possible unlawful discrimination. Special care should be taken with people whose first language is not English to ensure that they have understood the administration instructions properly. Some assessments which are fair for native English speakers will present problems for people with a lesser command of the English language. Instruments requiring reading skills when these are not an integral part of the job are particularly likely to be unfair. Where possible, such candidates should be assessed in their native language. There are a number of aspects to instrument administration that should be standardised, this is illustrated below:

Standardisation:

An encouraging attitude on the part of the administrator is always desirable, but it is particularly important to establish rapport with individuals who might lack confidence or who feel anxious about the assessment. The introduction to the assessment session is an important part of the administration procedure and instructions should be clear and not rushed. It allows the establishment of this rapport and should be conducted in a serious yet friendly manner. Information should be provided during instrument administration on the following:

  • Why instruments are being used and how they fit into the assessment procedure.
  • The intended recipients of the information.
  • The name and address of the organisation responsible for conducting the assessments.
  • The name and address of the organisation who will hold the assessment results.
  • That the completion of the assessment is not compulsory.
  • The implications, if any, should the candidate choose not to complete the assessment.
  • That they will receive meaningful feedback relating to their performance on all assessments conducted.

There should be an opportunity for candidates to ask general questions before the formal assessment procedure starts.

Confidentiality and Storage

The Role of Feedback

The Privacy Act 1993 requires that whenever assessment results are used assessors should be honest and open with candidates about why the instruments are being used and what will happen to the results. Members of the Psychological Society are also bound by the Code of Ethics to obtain the informed consent of the individuals to be assessed when undertaking a psychological assessment. As such, individuals must be informed of the right to know the content of psychological assessment reports and in reporting findings, psychologists must endeavour to ensure that appropriate explanations of the findings and their interpretations are given. All individuals assessed should thus, be offered meaningful feedback of their results as soon after the assessment process as possible. Personality and motivation questionnaire feedback is critical and will often enhance the interpreters own understanding of assessment results.

Feedback does not need to be lengthy, indeed with a large number of applicants this might be very time consuming. A face to face interview is preferred, but telephone feedback may be the only option in some circumstances. Feedback should be given by qualified users and should be an open and accurate two-way process. The NZPS guidelines do not sanction the release of uninterpreted data from assessments to persons untrained in their use and interpretation, profile charts may be shown to respondents but they should not be given copies to take away. A short narrative summary may be provided if desired. This is particularly useful where assessment is for counselling and development purposes.

Computer generated or narrative reports can support, but should not replace, the feedback interview. Some may be suitable to give to respondents, but many are intended as aids to interpretation to the trained instrument user and could easily be misinterpreted by others. Users should follow guidelines provided by the author or publisher of such systems.

Under Information Privacy Principle 8 of the Privacy Act 1993, personal information gathered and held about an individual should not be used without ensuring that the information is accurate, up to date, complete, relevant and not misleading. Feedback interviews are an important part of this validation process.

Assessment Results

Assessment results, like all personal information, should be stored with the strictest regard to confidentiality. Access should be restricted to those with a need to know and in accordance with what has been agreed with the respondent during administration and feedback. Persons who are untrained should not be allowed access to raw data from assessments, but only to clear interpretations of those results.

Individuals do change and develop and so psychometric data can become less accurate over time. Instrument scores should therefore not be kept on file indefinitely. The time period for which scores are valid will differ depending on the nature, the measures and the particular use made of them. Care should be taken with the use of results over 6 to 12 months old for selection purposes. Little reliance should be put on results over 2 years old for any purpose. The Privacy Act 1993, requires that all personal information, including assessment results, should be protected by such security safeguards as it is reasonable to take to ensure against (1) loss and (2) unauthorised access, use, modification or disclosure.

Materials

The security of materials is paramount. Free circulation leads to over familiarity and devalues psychometric instruments. Responsible instrument publishers only supply materials to trained users who, in turn, must ensure untrained users do not gain access to them. Within an organisation decisions should be taken about who should hold assessment materials and who should have access. It may not be desirable for all users to have access to all materials. Central storage can help prevent unnecessary duplication of materials but may not be practical in decentralised organisations. An organisation must supply instrument users with appropriate storage space where the materials can be kept under lock and key. It is highly desirable that all materials are logged in and out of storage when used. This helps ensure materials are not carelessly left lying around or misplaced. Failure to keep track of materials can be expensive where replacements have to be purchased or annual lease fees paid on missing booklets.

Summary and Conclusion

Research and legislative precedent is firmly entrenching the importance of rigorous job analysis to justify the use and choice of tests. Our ethical and professional obligations as psychologists are clearly laid out and oblige us to only use tools for which we are trained. Validity and reliability and appropriateness of the items should underlie our evaluation of tests.

Standardisation is the key principle behind the application of psychometric tools. As practitioners this does not always come easily to us, but being rigorous must be foremost in our minds.


References

Cronbach, L.J. (1984). Essentials of Psychological Testing (4th ed.). Harper & Row New York.

Hunter, J.E., Schmidt, F.L. & Hunter, K. (1979). Different validity of employment tests by race: A comprehensive review and analysis. Psychological Bulletin, 86, 721-735.

Kellett, D., Fletcher, S., Callen, A., & Geary, B. (1994). Fair Testing: The Case of Bristish Rail. The Psychologist, January, 26 - 29.

Mischel, W. (1986). An Introduction to Personality. CBS College Publishing. Japan.

Uniform Guidelines on Employee Selection Procedures. (1985). Equal Employment Opportunities Commission, Department of Labour & The Office of Personnel Management. 29CFR, Section 1607, Washington DC.