Staffing Policies and Programs: What information does a selection decision maker need to collect in making staffing decisions? What are the ways in which this information can be collected?

Part 4
Staffing Activities: Selection
Chapter 7: Measurement
Chapter 8: External Selection I
Chapter 9: External Selection II
Chapter 10: Internal Selection
McGraw-Hill/Irwin
Copyright © 2012 by The McGraw-Hill Companies, Inc., All Rights Reserved.

Part 4
Staffing Activities: Selection
Chapter 7:
Measurement

Staffing Policies and Programs
Staffing System and Retention Management
Support Activities
Legal compliance
Planning
Job analysis
Core Staffing Activities
Recruitment: External, internal
Selection:
Measurement, external, internal
Employment:
Decision making, final match
Staffing Organizations Model
7-*

7-*
Chapter Outline

Importance and Use of Measures
Key Concepts
Measurement
Scores
Correlation Between Scores
Quality of Measures
Reliability of Measures
Validity of Measures
Validation of Measures in Staffing
Validity Generalization
Staffing Metrics and Benchmarks

Collection of Assessment Data
Testing Procedures
Acquisition of Tests and Test Manuals
Professional Standards
Legal Issues
Determining Adverse Impact
Standardization
Best Practices

7-*
Learning Objectives for This Chapter

Define measurement and understand its use and importance in staffing decisions
Understand the concept of reliability and review the different ways reliability of measures can be assessed
Define validity and consider the relationship between reliability and validity
Compare and contrast the two types of validation studies typically conducted
Consider how validity generalization affects and informs validation of measures in staffing
Review the primary ways assessment data can be collected

7-*
Discussion Questions for This Chapter

Imagine and describe a staffing system for a job in which there are no measures used.
Describe how you might go about determining scores for applicants’ responses to (a) interview questions, (b) letters of recommendation, and (c) questions about previous work experience.
Give examples of when you would want the following for a written job knowledge test
a low coefficient alpha (e.g., α = .35)
a low test–retest reliability.
Assume you gave a general ability test, measuring both verbal and computational skills, to a group of applicants for a specific job. Also assume that because of severe hiring pressures, you hired all of the applicants, regardless of their test scores.
How would you investigate the criterion-related validity of the test?
How would you go about investigating the content validity of the test?
What information does a selection decision maker need to collect in making staffing decisions? What are the ways in which this information can be collected?

7-*
Key Concepts

Measurement
the process of assigning numbers to objects to represent quantities of an attribute of the objects
Scores
the amount of the attribute being assessed
Correlation between scores
a statistical measure of the relation between the two sets of scores

7-*
Importance and Use of Measures

Measures
Methods or techniques for describing and assessing attributes of objects
Examples
Tests of applicant KSAOs
Job performance ratings
of employees
Applicants’ ratings of their
preferences for various types
of job rewards

7-*
Importance and Use of Measures
(continued)

Summary of measurement process
(a) Choose an attribute of interest
(b) Develop operational definition of attribute
(c) Construct a measure of attribute as operationally
defined
(d) Use measure to actually gauge attribute
Results of measurement process
Scores become indicators of attribute
Initial attribute and its operational definition are transformed into a numerical expression of attribute

7-*
Measurement: Definition

Process of assigning numbers to objects to represent quantities of an attribute of the objects
Attribute/Construct – Knowledge of mechanical principles
Objects – Job applicants

Ex. 7.1 Use of Measures in Staffing
7-*

7-*
Measurement: Standardization

Involves
Controlling influence of extraneous factors
on scores generated by a measure and
Ensuring scores obtained reflect the attribute measured
Properties of a standardized measure
Content is identical for all objects measured
Administration of measure is identical for all objects
Rules for assigning numbers are clearly specified and agreed on in advance

7-*
Measurement: Levels

Nominal
A given attribute is categorized and numbers are assigned to categories
No order or level implied among categories
Ordinal
Objects are rank-ordered according to how much of attribute they possess
Represents relative differences among objects

Interval
Objects are rank-ordered
Differences between adjacent points on measurement scale are equal in terms of attribute
Ratio
Similar to interval scales – equal differences between scale points for attribute being measured
Have a logical or absolute zero point

7-*
Measurement: Differences in
Objective and Subjective Measures

Objective measures
Rules used to assign numbers to attribute are predetermined, communicated, and applied
through a system
Subjective measures
Scoring system is more elusive, often involving a rater who assigns the numbers
Research shows these may not be strongly related, but purely objective measures can miss important parts of job performance

7-*
Scores

Definition
Measures provide scores to represent
amount of attribute being assessed
Scores are the numerical indicator of attribute
Central tendency and variability
Exh. 7.2: Central Tendency and Variability: Summary Statistics
Percentiles
Percentage of people scoring below an individual in a distribution of scores
Standard scores

7-*
Discussion questions

Imagine and describe a staffing system for a job in which there are no measures used.
Describe how you might go about determining scores for applicants’ responses to (a) interview questions, (b) letters of recommendation, and (c) questions about previous work experience.

7-*
Correlation Between Scores

Scatter diagrams
Used to plot the joint distribution of the two sets of scores
Exh. 7.3: Scatter Diagrams and Corresponding Correlations
Correlation coefficient
Value of r summarizes both
Strength of relationship between two sets of scores and
Direction of relationship
Values can range from r = -1.0 to r = 1.0
Interpretation – Correlation between two variables does not imply causation between them
Exh. 7.4: Calculation of Product-Movement Correlation Coefficient

7-*
Exh. 7.3: Scatter Diagrams and
Corresponding Correlations

7-*
Significance of the Correlation Coefficient

Practical significance
Refers to size of correlation coefficient
The greater the degree of common variation
between two variables, the more one variable
can be used to understand another variable
Statistical significance
Refers to likelihood a correlation exists in a population, based on knowledge of the actual value of r in a sample from that population
Significance level is expressed as p < value
Interpretation — If p < .05, there are fewer than 5 chances in 100 of concluding there is a relationship in the population when, in fact, there is not

7-*
Quality of Measures

Reliability of measures
Validity of measures
Validity of measures in staffing
Validity generalization

7-*
Quality of Measures: Reliability

Definition: Consistency of measurement of an attribute
A measure is reliable to the extent it provides a consistent set of scores to represent an attribute
Reliability of measurement is of concern
Both within a single time period and between time periods
For both objective and subjective measures
Exh. 7.6: Summary of Types of Reliability

7-*
Ex. 7.6: Summary of Types of Reliability

7-*
Quality of Measures: Reliability

Measurement error
Actual score = true score + error
Deficiency error
Failure to measure some aspect of attribute assessed
Contamination error
Occurrence of unwanted or undesirable influence on the measure and on individuals being measured

7-*
Ex. 7.7 – Sources of Contamination Error and Suggestions for Control

7-*
Quality of Measures: Reliability

Procedures to calculate reliability estimates
Coefficient alpha
Should be least .80 for a measure to have an acceptable degree of reliability
Interrater agreement
Minimum level of interrater agreement – 75% or higher
Test-Retest reliability
Concerned with stability of measurement
Level of r should range between r = .50 to r = .90
Intrarater agreement
For short time intervals between measures, a fairly high relationship is expected – r = .80 or 90%

7-*
Quality of Measures: Reliability

Implications of reliability
Standard error of measurement
Since only one score is obtained from an applicant, the critical issue is how accurate the score is as an indicator of an applicant’s true level of knowledge
Relationship to validity
Reliability of a measure places an upper limit on the possible validity of a measure
A highly reliable measure is not necessarily valid
Reliability does not guarantee validity – it only makes it possible

7-*
Quality of Measures: Validity

Definition: Degree to which a measure truly measures the attribute it is intended to measure
Accuracy of measurement
Exh. 7.9: Accuracy of Measurement
Accuracy of prediction
Exh. 7.10: Accuracy of Prediction

7-*
Ex. 7.9: Accuracy of Measurement

7-*
Discussion questions

Give examples of when you would want the following for a written job knowledge test
a low coefficient alpha (e.g., α = .35)
a low test–retest reliability.

7-*
Exh. 7.12: Accuracy of Prediction

7-*
Validity of Measures in Staffing

Importance of validity to staffing process
Predictors must be accurate representations of KSAOs to be measured
Predictors must be accurate in predicting job success
Validity of predictors explored through validation studies
Two types of validation studies
Criterion-related validation
Content validation

Ex. 7.13: Criterion-Related Validation
Criterion Measures: measures of performance on tasks and task dimensions
Predictor Measure: it taps into one or more of the KSAOs identified in job analysis
Predictor–Criterion Scores: must be gathered from a sample of current employees or job applicants
Predictor–Criterion Relationship: the correlation must be calculated.
7-*

7-*
Ex. 7.14: Concurrent and Predictive
Validation Designs

7-*
Content Validation

Content validation involves
Demonstrating the questions/problems (predictor scores) are a representative sample of the kinds of situations occurring on the job
Criterion measures are not used
A judgment is made about the probable correlation between predictors and criterion measures
Used in two situations
When there are too few people to form a sample for criterion-related validation
When criterion measures are not available
Exh. 7.16: Content Validation

7-*
Validity Generalization

Degree to which validity can be extended to other contexts
Contexts include different situations, samples of people and time periods
Situation-specific validity vs. validity generalization
Exh. 7.18: Hypothetical Validity Generalization Example
Distinction is important because
Validity generalization allows greater latitude than situation specificity
More convenient and less costly not to have to conduct a separate validation study for every situation

Exhibit 7.18 Hypothetical Validity Generalization Example
7-*

7-*
Discussion questions

Assume you gave a general ability test, measuring both verbal and computational skills, to a group of applicants for a specific job. Also assume that because of severe hiring pressures, you hired all of the applicants, regardless of their test scores.
How would you investigate the criterion-related validity of the test?
How would you go about investigating the content validity of the test?
What information does a selection decision maker need to collect in making staffing decisions? What are the ways in which this information can be collected?

7-*
Staffing Metrics and Benchmarks

Metrics
quantifiable measures that demonstrate the effectiveness (or ineffectiveness) of a particular practice or procedure
Staffing metrics
job analysis
validation
Measurement
Benchmarking as a means of developing metrics

7-*
Collection of Assessment Data

Testing procedures
Paper and pencil measures
PC- and Web-based approaches
Applicant reactions
Acquisition of tests and test manuals
Paper and pencil measures
PC- and Web-based approaches
Professional standards

7-*
Legal Issues

Disparate impact statistics
Applicant flow statistics
Applicant stock statistics
Standardization
Lack of consistency in treatment of applicants is
a major factor contributing to discrimination
Example: Gathering different types of background information from protected vs. non-protected groups
Example: Different evaluations of information for protected vs. non-protected groups
Validation
If adverse impact exists, a company must either eliminate it or justify it exists for job-related reasons (validity evidence)

7-*
Ethical Issues

Issue 1
Do individuals making staffing decisions have an ethical responsibility to know measurement issues? Why or why not?
Issue 2
Is it unethical for an employer to use a selection measure that has high empirical validity but lacks content validity? Explain.