TRAINING AND EMPLOYMENT GUIDANCE LETTER NO. 6-00, CHANGE 1

TO:         ALL STATE WORKFORCE LIAISONS
            ALL STATE EMPLOYMENT SECURITY AGENCIES
            ALL STATE ONE-STOP CENTER SYSTEM LEADS
            ALL STATE WORKER ADJUSTMENT LIAISONS                   
FROM:	    EMILY STOVER DEROCCO
            Assistant Secretary  
SUBJECT :   Changes to Guidance on Capturing Customer Satisfaction Results 
            for Participants and Employers Receiving Services Under Title I 
            of the Workforce Investment Act

1. Purpose. To provide revisions to technical guidance for states to implement the core and customer satisfaction performance measures and calculate performance levels required under the Workforce Investment Act in order that the guidance contained in Training and Employment Guidance Letter (TEGL) No. 6-00 conform to the requirements contained in TEGL No. 14-00.

2. References. Workforce Investment Act of 1998 (WIA), section 136; WIA Final Rules, 20 CFR part 666; TEGL No. 7-99; TEGL No. 8-99; TEGL No. 6-00; TEGL No. 14-00; Planning Guidance and Instructions for Submission of the Strategic Five-Year Plan for Title I of WIA and the Wagner-Peyser Act; WIA: Final Unified Planning Guidance; Notice of Office of Management and Budget (OMB) Action No. 1205-0420, dated March 1, 2001.

3. Background.. Section 136 of WIA specifies core indicators of performance and their application to workforce investment activities in adult, dislocated worker, and youth programs. In TEGL No. 7-99, the Department issued technical guidance to states to use when calculating outcomes on the 15 core performance measures and the two customer satisfaction performance measures. Four core measures apply separately to the adult, dislocated worker and older youth participants; three measures apply to younger youth; and two measures of customer satisfaction apply across the three funding streams.

TEGL No. 6-00 outlined the parameters states are to use in conducting customer satisfaction surveys at the state level and detailed the parameters states are to use in conducting surveys and reporting results for local workforce investment boards (WIBs). Formal instructions for reporting performance outcomes (including customer satisfaction results) to the Department were released in TEGL No. 14-00 as part of the overall WIA performance accountability reporting system approved by OMB on March 1, 2001. The approved performance reporting system contained several changes to the customer satisfaction survey methodology in response to comments forwarded to OMB by states. These changes increase the methodological rigor of the customer satisfaction surveys.

This guidance letter outlines changes to the parameters detailed in TEGL No. 6-00 that are needed to conform to the increased methodological rigor published in TEGL No. 14-00.

4. Changes to the Customer Satisfaction Survey Methodology

A. Revisions to Original Guidance on Conducting Surveys to Obtain State Level Results
• For both the participant and employer surveys, states must achieve at least 500 completed interviews per group and meet the minimum required response rates. For the first year of data collection only, there is a minimum response rate of 50% for the employer survey and 50% for the participant survey for determining the usefulness of the data for performance measurement purposes. Response levels below 50% during the first year will invalidate the results for performance measurement purposes as a significant bias in the results is likely. For the following years of data collection, states are required to achieve response rates of 70% for both the participant survey and the employer survey.

  [Affected item(s) in TEGL No. 6-00: Section 5, Parameters for Conducting Surveys to Obtain State Level Results, items D and F on page 4.]

• States may randomly sample participants and employers to obtain the desired number of completed surveys. States are required to determine the appropriate sample sizes and sampling percentages using the required response rate and the required number of completed interviews. Original guidance required states to draw minimum sample sizes of 1,000 for the participant survey and 1,000 for the employer survey.

  [Affected item(s) in TEGL No. 6-00: Section 5, item E on page 4.]

• Respondents must be told that responding to the survey is voluntary and that the information they provide will be kept confidential. States must not make any assurances regarding confidentiality without ensuring that they have the legal authority to make such assurances. Original guidance stated that sharing information on the voluntary nature of the survey and the confidentiality of results with respondents was optional for states.

  [Affected item(s) in TEGL No. 6-00: Section 5, item C on page 4.]

• States must create a process for creating an up-to-date customer list, capturing information on each customer's address and telephone number. Original guidance from the Department encouraged, but did not require, states to create customer lists in a timely fashion prior to selecting samples for the surveys.

  [Affected item(s) in TEGL No. 6-00: Section 5, item H on page 5.]

• The sampling frame used to obtain state-level results and the sampling frame used to obtain local area results includes all exiters and employers eligible for the survey, with the exception of those who are deceased, institutionalized, incarcerated, or unable to complete the survey due to illness or injury. Original guidance from the Department stated the sampling frame only included participants and employers eligible for the surveys who had valid contact information.

  The response rate is calculated as the number of respondents with complete customer satisfaction information divided by the total number in the sample size. If sampling is not used, the response rate is calculated as the number of respondents with complete customer satisfaction information divided by the total number eligible for the survey (also known as the sampling frame). A survey is considered complete when valid answers are provided by the respondents for each of the core American Customer Satisfaction Index (ACSI) questions. Valid answers are responses 1 through 10 for each of the three ACSI questions. Responses 11 (don't know) and 12 (refused) are considered missing values and are not considered valid responses for computation purposes.

  The calculation of the ACSI includes only the results from complete surveys.

  [Affected item(s) in TEGL No. 6-00: Section 5, item F on page 4.]

• The introductory statements for both the participant survey and the employer survey have been revised to better set the context for the interviews. These revised introductory statements are to be read to each respondent at the beginning of the interviews. The wording of an introductory statement may be modified to meet the needs of the state. In addition to contact information, survey administrators will need to know the service(s) provided to each customer, the date(s) of service, and the firm(s) providing the service(s) in order to complete the interviews. The service(s) should be worded in terms recognizable to the customer.

  The following introductory script replaces the introductory statements for both the participant and employer surveys published in the original guidance:

  [Introductory Script]

  My name is (interviewer) with (firm or agency) and I am conducting a survey for the (program name). I would like to speak to Ms./Mr. (name of participant or name of employer contact).

  Are you the Ms./Mr. (name of participant or name of employer contact) who received a (name of service(s)) from the on (date or date range)? [If the respondent confirms the receipt of services, proceed with the interview. If no, end the interview and thank the respondent for his or her time.]

  I would like to ask you some questions about your recent experience with the services provided to you [substitute "you" with "your business" for employer respondents]. Our purpose is to learn from you how to improve programs and services offered to people [substitute "people" with "area businesses" for employer respondents]. The survey was approved by the Office of Management and Budget to collect information on your experience with services we provide to individuals [substitute "individuals" with "employers" for employer respondents]. The questionnaire is voluntary and should take no longer than XX minutes to complete.

  Please know the answers you provide to my questions will be kept confidential. Do you have any questions for me before we start the interview? [If no, continue. If yes, probe to understand the questions, provide answers and restate the importance of the respondent's participation in the interview.]

  First, I want to know your overall experience with the services. I will read to you three statements and ask you to rate your experience. [Proceed with the core ACSI questions.]

[Affected item(s) in TEGL No. 6-00: Section 5, item J on pages 5 through 7]

B. Additions to Original Guidance on Conducting Surveys to Obtain State Level Results

• Local program staff must inform the customer during the registration process about the importance of satisfying customers and the possibility of being contacted for information on his or her experience with the services.

• The interview should be limited to 15 minutes or less. Shorter questionnaires produce better response rates when compared to longer questionnaires.

• A minimum of five follow-up attempts is required, including contact at various times of the day before closing the record.

• Local programs should collect alternate contact information for a person known by the customer who would know the whereabouts of the customer in the event the customer cannot be reached at the address and telephone number recorded for the customer.

• A letter in advance of the survey should be sent out informing the customer that he or she can expect to receive a telephone call about his or her satisfaction with the services. States should place a phone number on the letter, suggesting the customer call if he or she needs help or call if his or her home phone number changed.

  [The above items should be added to section 5, Parameters for Conducting Surveys to Obtain State Level Results, in TEGL No. 6-00.]

C. Updated Table of Weights to Use in Calculating the ACSI

Updated weights were calculated for use in PY 2001 by a statistical algorithm in order to minimize random survey noise or measurement error that exists in all survey data. Please refer to TEGL No. 6-00 for a description of the formulas for calculating the ACSI scores.

Specific weighting factors were developed for each state for PY 2001 and can be found attached to this TEGL. New weighting factors will be published annually. The weights listed in the attachment are to be used by each state in computing performance on the ACSI for employers and participants.

5. Action Required. States shall distribute this guidance letter to all officials within the state who need such information to implement WIA core and customer satisfaction measures.

6. Inquiries. Questions concerning this issuance may be directed to your appropriate Regional Office.

7. Attachments. ACSI Formula Weights.

 

 

 

 

 

Weights for Use in Calculating
State-Level ACSI Scores
September 14, 2001

	Weights
State Name	Overall Satisfaction	Confirm Expectations	Compared to Ideal
AK	0.3621	0.2650	0.3729
AL	0.4411	0.3096	0.2493
AR	0.4366	0.3213	0.2421
AZ	0.4047	0.3198	0.2755
CA	0.4244	0.3238	0.2518
CO	0.3974	0.3304	0.2722
CT	0.4328	0.3407	0.2265
DC	0.4879	0.3208	0.1912
DE	0.4430	0.3076	0.2494
FL	0.4331	0.3054	0.2615
GA	0.4252	0.3076	0.2672
HI	0.1958	0.6136	0.1906
IA	0.3960	0.3190	0.2850
ID	0.3769	0.3236	0.2995
IL	0.4023	0.3222	0.2755
IN	0.4274	0.3360	0.2366
KS	0.4092	0.3426	0.2482
KY	0.4263	0.3030	0.2707
LA	0.4350	0.3078	0.2573
MA	0.4188	0.3283	0.2529
MD	0.4215	0.3375	0.2410
ME	0.4526	0.2804	0.2670
MI	0.4223	0.3237	0.2539
MN	0.4232	0.3425	0.2342
MO	0.4288	0.3300	0.2412
MS	0.4240	0.3171	0.2589
MT	0.3908	0.3488	0.2604
NC	0.4113	0.3311	0.2577
ND	0.4297	0.2790	0.2913
NE	0.4046	0.3375	0.2579
NH	0.4266	0.3247	0.2487
NJ	0.4309	0.3371	0.2320
NM	0.4312	0.3193	0.2495
NV	0.3810	0.3493	0.2698
NY	0.4129	0.3336	0.2535
OH	0.4217	0.3251	0.2532
OK	0.4382	0.3111	0.2507
OR	0.4247	0.3274	0.2479
PA	0.4314	0.3287	0.2399
RI	0.3932	0.3498	0.2570
SC	0.4551	0.2986	0.2464
SD	0.4039	0.3263	0.2698
TN	0.4314	0.3177	0.2509
TX	0.4186	0.3313	0.2501
UT	0.4063	0.3323	0.2615
VA	0.4313	0.3111	0.2576
VT	0.4225	0.3266	0.2509
WA	0.4059	0.3244	0.2697
WI	0.4282	0.3133	0.2585
WV	0.4408	0.3145	0.2447
WY	0.4488	0.3272	0.2240