[Federal Register: August 5, 1999 (Volume 64, Number 150)] [Notices] [Page 42817-42822] From the Federal Register Online via GPO Access [wais.access.gpo.gov] [DOCID:fr05au99-160] _______________________________________________________________________ Part VI Department of Labor _______________________________________________________________________ Employment and Training Administration _______________________________________________________________________ Consultation Papers on Performance Accountability Under Title I of the Workforce Investment Act of 1998 (WIA); Notice DEPARTMENT OF LABOR Employment and Training Administration, Labor Consultation Papers on Performance Accountability Under Title I of the Workforce Investment Act of 1998 (WIA) AGENCY: Employment and Training Administration, Labor. ACTION: Notice. ----------------------------------------------------------------------- SUMMARY: The purpose of this notice is to disseminate consultation papers for interested parties on the performance accountability system for title I of the Workforce Investment Act. There are two papers. The first paper provides a framework regarding the approach and processes for continuous improvement under title I of the Workforce Investment Act of 1998. The second paper provides a framework for the approach and processes for customer satisfaction measures under title I of the Workforce Investment Act. Interested parties have 30 days to provide comments on these papers. FOR FURTHER INFORMATION CONTACT: Mr. Eric Johnson, Workforce Implementation Taskforce Office, U.S. Department of Labor, 200 Constitution Avenue, NW, Room S-5513, Washington, D.C., Telephone: (202) 219-0316 (voice) (This is not a toll free number), or 1-800-326- 2577 (TDD). Information may also be found, or comments provided, at the website--http://usworkforce.org. SUPPLEMENTARY INFORMATION: The Workforce Investment Act (WIA or Act) Pub. L. 105-220 (August 7, 1998) provides the framework for a reformed national workforce investment system designed to meet the needs of the nation's employers, job seekers and those who want to further their careers. One of the key reforms contained in the Act is the establishment of a comprehensive accountability system to assess the effectiveness of State and local areas in providing employment and training services. The Act requires: - A focus on results defined by core indicators of performance; - Measures of customer satisfaction with programs and services; - A strong emphasis on continuous improvement; - Annual performance levels developed as a result of negotiations among Federal, State and local partners; - Incentive awards and financial sanctions based on State performance; and - Reporting and dissemination of performance results. The two papers contained in this notice focus on two of these requirements--continuous improvement and customer satisfaction. The Department is approaching the development of this new performance accountability system on two tracks. First, definitions of the core measures of performance and temporary reporting instructions have been developed and disseminated for those States who are implementing WIA in Program Year (PY) 1999. Second, the Department is working with States and local governments to develop definitions and reporting requirements for use in PY 2000 and beyond. Part of this process will include using the lessons learned from the early implementing States and working with the Department of Education and other Federal agencies to develop common definitions for performance measures across programs. In general, the Department is considering PY 1999 to be a transition year. The comments received on these two papers will be used in developing the performance accountability system for PY 2000 and beyond. Comments are solicited on the overall framework and approaches being proposed for customer satisfaction and continuous improvement under title I of WIA. Signed at Washington, D.C., this 30th day of July 1999. Raymond L. Bramucci, Assistant Secretary of Labor, Employment and Training Administration. Attachment 1--Continuous Improvement Under Title I of the Workforce Investment Act of 1998 I. Introduction A. Legal Framework The Workforce Investment Act of 1998 envisions a high performance workforce investment system in this country--a system that is customer- driven, results-oriented, flexible, and continuously improving. The Act's purpose is clearly stated as: To provide workforce investment activities that increase participants' employment, retention, earnings, and skill attainment and as a result: - Improve the quality of the workforce; - Reduce welfare dependency; and - Enhance the productivity and competitiveness of the nation. The Act envisions a workforce investment system that strives for high performance rather than settling for compliance levels of performance, and that delivers unparalleled levels of services to customers--job seekers, workers, and employers. Although WIA has numerous references to continuous improvement, this consultation paper focuses on three major provisions contained in Section 136 of the legislation: - A comprehensive performance accountability system will include an assessment of the effectiveness of state and local areas in achieving continuous improvement of workforce investment activities. Section 136(a). - The Governor/Secretary agreement on State adjusted levels of performance must take into account the extent to which those levels promote continuous improvement in performance. Section 136(b)(3)(A)(iv)(III). - States must conduct ongoing evaluations of workforce investment activities to promote and implement methods for continuously improving them. Section 136(e)(1). B. Guiding Principles The U.S. Department of Labor (DOL) is using the following as guiding principles in designing a system-wide approach for continuous improvement. - DOL's role in continuous improvement is primarily one of leadership carried out through an effective technical assistance effort. - For the workforce investment system to strive toward performance excellence, continuous improvement practices must be embraced at all levels--local, State, and DOL Regional and National Offices. - DOL will integrate existing quality initiatives to drive continuous improvement through a technical assistance strategy that includes award and recognition efforts, access to information on best practices, and the availability of a variety of tools. - The Malcolm Baldrige Criteria for Performance Excellence will be used as the framework for continuously improving performance in the workforce investment system. C. Malcolm Baldrige Criteria The Malcolm Baldrige Criteria for Performance Excellence and the Baldrige Scoring Guidelines are proposed as the framework for enabling organizations within the workforce investment system to advance toward high performance. This framework is widely accepted as the standard for defining performance excellence in public and private organizations. The Criteria and Scoring Guidelines are excellent diagnostic instruments that can help leaders identify organizational strengths and key areas for improvement and work to achieve higher levels of performance. DOL will provide resources and technical assistance to state and local organizations that are interested in using the Baldrige Criteria to help improve performance. The following is excerpted from the 1999 ``Criteria for Performance Excellence,'' and includes for informational purposes the relative point value assigned to each category: ------------------------------------------------------------------------ Point Award category value ------------------------------------------------------------------------ Leadership................................................... 125 Strategic Planning........................................... 85 Customer and Market Focus.................................... 85 Information and Analysis..................................... 85 Human Resource Focus......................................... 85 Process Management........................................... 85 Business Results............................................. 450 ---------- Total.................................................. 1,000 ------------------------------------------------------------------------ II. Approach to Continuous Improvement A. Overview of the Approach According to leading Baldrige experts, continuous improvement is the systematic and ongoing improvement of products, programs, services and processes by small increments and major breakthroughs. Continuous improvement is the process of building dynamic, high achieving systems within every organization, and becomes embedded in the way the organization conducts its daily activities. DOL's role in the continuous improvement process is primarily based on providing leadership and technical assistance. In striving to improve performance as measured by the performance and customer satisfaction indicators, states and localities will need resources, information and technical assistance to help them continuously improve organizational effectiveness. The approach to continuous improvement proposed in this consultation paper envisions that DOL will play a strong, proactive role in providing States and localities with information, resources, tools, training and technical assistance to help them enhance their performance. DOL will also apply these tools to continuously improve the effectiveness of ETA National and Regional Offices. DOL's Continuous Improvement Strategy is aimed at improving outcomes for the customers of the workforce investment system by enhancing system-wide performance. The objectives of the strategy are to: - Effectively align system-wide resources to achieve performance excellence. - Recognize and award top performers within the system. - Provide organizations and individuals with learning opportunities to acquire the skills needed to operate in a high performance mode. B. Continuous Improvement in State Workforce Investment Plans A rigorous approach to continuous improvement must be applied at all levels of the workforce investment system in order for that system to achieve the high levels of performance envisioned in the Workforce Investment Act. For States to develop a Statewide workforce investment system that incorporates a rigorous approach to continuous improvement, each State needs to start with a snapshot or baseline of its system capacity--its ``as is'' capacity at the point in time when the State Plan is developed. Ideally, States would establish both an ``as is'' state for each organization's capability to become a high performance organization (organizational effectiveness), as well as the ``as is'' state for each organization's current program results and outcomes. In the spirit of partnership and shared accountability, State officials and DOL officials would have this data before them as the basis for establishing the baseline. From the State's perspective, the baseline or starting point for continuous improvement is simply defined as, ``where you are now.'' (This process has been further defined in the consultation paper on Reaching Agreement on State Adjusted Levels of Performance.) The State's continuous improvement strategy becomes its approach for closing the gap between the current ``as is'' capacity and a time- sensitive ``desired state'' set forth in the State's plan. This approach addresses both the voluntary ``organizational improvement'' strategy and the more traditional compliance-oriented strategies for meeting minimum WIA specified performance measures. This offers states the opportunity to propose a rigorous and comprehensive approach to continuous improvement--one that establishes an effort to develop and improve organizational capacity (systems and processes) thus enabling committed organizations to deliver high performance, customer-focused services, as well as meeting all other requirements of the Act. C. Voluntary Approach to Assessment and Benchmarking DOL's role is to make available to States the resources, tools and services that will help them advance toward high performance through a rigorous continuous improvement strategy. The basic tools and services would include organizational assessment tools, resources to aid in the development of improvement plans, best practices, and benchmarking for continuous improvement services. ``Benchmarking'' is the use of information and data on processes and results that represent best practices and the highest levels of performance. As part of its continuous improvement strategy, DOL would gather and make easily accessible to States and local organization benchmarks of the highest levels of performance both in processes and results within the workforce investment system, and for similar processes and results for organizations outside the system. Benchmarks represent the very essence of high performance business practices--comparing your organization to the very best in class and striving continuously to attain that level of performance. It is a voluntary practice carried out by the best organizations as a fundamental component of their continuous improvement strategy. D. Supporting Continuous Improvement Activities Under WIA, States are to ensure that the principle of continuous improvement is embedded in Statewide workforce investment activities. Again, this would represent the regimen for achieving the systematic and ongoing improvement of workforce investment programs, services, and processes by small increments and major breakthroughs. This continuous improvement regimen will foster enhancements in performance levels desired by each level of the system. The State's Workforce Investment Plan must include a description of the State's strategy for developing and operating this continuous improvement approach. While each State has latitude to use a range of resources, tools and approaches for accomplishing this, the States are encouraged to work with the Employment and Training Administration (ETA) Regional Office to take advantage of resources available from or brokered through ETA's Continuous Improvement Strategy. Generally, DOL is seeking comment on the following strategy to support the local, State, Regional and National organizations in continuous improvement-- - Establish a system of organizational and individual learning to acquire skills needed to support high performance within the workforce investment system. - Utilize the Malcolm Baldrige Criteria for Performance Excellence as a proven and rigorous methodology to transform local, State, and National workforce investment organizations. - Establish an award and recognition system in support of high performing organizations at all levels. - Work closely with early implementing States as partners to begin the system-wide transformation process toward performance excellence. - Provide local and State organizations, Regional Offices and National Office with easily accessible information on benchmarks and best practices, as well as affordable and effective assessment tools. Attachment II: Customer Satisfaction Under Title I of the Workforce Investment Act of 1998 I. Introduction A. Legal Framework In addition to the core measures, the Workforce Investment Act of 1998 [WIA Section 136(b)(2)(B)] states that ``the customer satisfaction indicator of performance shall consist of customer satisfaction of employers and participants with services received from the workforce investment activities authorized under this subtitle.'' The Act [Section 136(b)(3)(A)(i)] also requires that there be State-adjusted levels of performance for customer satisfaction and that ``the levels of performance established * * * shall, at a minimum-- (1) Be expressed in an objective, quantifiable, and measurable form; and (2) Show the progress of the State toward continuously improving in performance.'' WIA draws a clear link between the core indicators of performance and customer satisfaction. The levels of performance attained for the core indicators must ``assist the State in attaining a high level of customer satisfaction'' (WIA Section 136(b)(3)(a)(iv)(I)). WIA further states that ``customer satisfaction may be measured through surveys conducted after the conclusion of participation in workforce investment activities'' (WIA Section 136(b)(2)(B)). Effective high performance organizations listen to their customers and build their organization around meeting their customers' expectations. Determining a customer's expectations and satisfaction is an integral part of a continuous improvement strategy. Under the Workforce Investment Act, customer satisfaction is both a process of identifying and listening to customers, as well as an outcome for measuring program success. WIA emphasizes the importance of a customer-driven workforce system by including customer satisfaction as a required measure, along with the core indicators of performance. Customer satisfaction measures provide feedback to supervisors and staff about how their actions affect customers, giving them critical information to motivate and guide continuous improvement. Customer satisfaction feedback also sends a clear message to staff, management, and customers that customers matter. B. Guiding Principles DOL is using the following guiding principles in designing a system-wide approach for measurement of customer satisfaction: - Customer satisfaction is the foundation of an organization's strategy for continuous improvement. - Customer satisfaction should be measured after completion of the service and should be quantifiable. - Customer satisfaction surveys need to contain a set of required questions to form a customer satisfaction indicator. - Comparability is an important element in negotiating customer satisfaction performance levels and in providing opportunities for benchmarking and sharing best practices. - States and local organizations are encouraged to add customized questions to inform their efforts to align resources or redesign processes to achieve better results. II. Approach A. Overview of the Approach The Act, in requiring a customer satisfaction indicator for employers and participants, presents a general framework for developing a national approach. Customer satisfaction indicators are a specific part of the performance accountability system and are the foundation of an organization's strategy for continuous improvement. They provide a guide to achieving the vision and goals of the Act, and provide a focused and structured process for listening to and learning from customers. To meet the customer satisfaction requirements for Title I, DOL proposes the use of customer satisfaction surveys. There are two purposes for surveying customers. The first is to produce an outcome measure for each State as part of the performance accountability system. This will be accomplished by a small set of required questions that will form a customer satisfaction index. The second purpose is to gain customer feedback to help in improving processes and services. This will be accomplished through a set of recommended questions addressing each service component and any additional questions that the State and local areas choose to ask, depending on their particular needs and service mix. DOL will provide guidelines for collecting customer satisfaction data that will lay out the strategy and standards (e.g., sample size, response rate) for implementing the survey while providing as much flexibility for the states and localities as possible. The survey will contain the required questions that form the indicator. In addition, to cover many of the most commonly delivered services, the guidelines will suggest sets of questions that States may choose to use along with the indicator questions. The advantage of using these questions will be that they provide additional opportunities for benchmarking and learning from the best practices of others. B. Proposed Customer Satisfaction Strategy Consistent with the Workforce Investment Act, measures of customer satisfaction: - Must address participants and employers; - Must be quantifiable; - Must be able to track progress toward improvement; - Must be comparable across states; - May be measured at the conclusion of participation; and - Must promote continuous improvement in performance along with the core measures. The Act calls for assessment of two customer categories: (1) participants, and (2) employers. Consistent with the approach taken for core measures, two options are presented. The first option is to report the participant indicator for each of the four groups: - Adults - Dislocated Workers - Youth 19-21 served with youth funds, and - Youth 14-18. The second option is to aggregate the four groups to provide a single indicator of participant satisfaction. The advantages of reporting each of the four groups separately are to: 1. Allow for a more comprehensive analysis of results. An analysis by group will provide an assessment of the degree to which core indicator performance contributes to customer satisfaction. 2. Allow program managers to evaluate the degree to which they are satisfying different customer segments. The advantage of the second option is that it will simplify customer surveyingand reporting, and will emphasize high expectations for all of the groups. It should be noted that, under both options, the four groups identified above would also include those participating in incumbent worker training. Customer surveying for other services that are not covered under Title I would be at the discretion of other one-stop partner programs. For employers, it is proposed that services to employers be grouped into the following three service categories: (1) informational, (2) labor exchange, and (3) special services such as rapid response. Capturing customer satisfaction within each of these three categories will allow a clearer picture of service to employers and is one way to expand the system's ability to be accountable for services to a significant customer base. While States would not be required to report the three customer indicators for employers at a National level, they may utilize this method as a way to better understand their employer customers. C. Collecting Customer Satisfaction Information There are a number of different methods to collect customer satisfaction information. - The simplest approach is to train staff to listen to the customers they serve and to ask questions that elicit customer needs while they are providing service. - Focus groups and group interviews are another strategy. - A trained manager or staff person can circulate in the resource center where people are waiting and ask questions informally to gain a better understanding of customer needs and concerns. - Suggestion boxes are also a way of gathering information. - Telephone surveys of customers are used to gather specific information. To meet the WIA customer satisfaction requirements for Title I, the method proposed in this paper is customer satisfaction surveys. This is the most effective method that allows state and national aggregation of comparable, quantifiable data. As part of a comprehensive continuous improvement strategy, organizations will use a combination of strategies in addition to the proposed surveys, since each serves a somewhat different purpose and provides different types of information. D. Proposed Measures The customer satisfaction indicator will be derived from surveys that must have a minimum set of common questions asked in a common format to assure comparability. These common questions are used to form an index, which is a single score. An index has the advantage of addressing different dimensions of the customer's experience, and is more reliable than a single question. The creation of an index provides a proven methodology to capture common customer satisfaction information across programs and organizations that can be aggregated to a State and National level. The responses of the embedded questions will be rolled up to the State level and reported annually at a specified time. This approach will continue to be modified as the Department receives feedback and validation through consultation with the workforce investment system. Satisfaction for all customers in all service categories will be measured through a set of 3-5 questions that together form the indicator. We propose that the surveys include these three questions: - ``Overall, how satisfied were you with the services received?'' (Ranging from 1--Very Dissatisfied to 10--Very Satisfied) - ``How likely would you be to refer others to these services?'' (Ranging from 1--Not Very Likely to 10--Very Likely) - ``If you were in a similar situation again, how likely would you be to use these services?'' (Ranging from 1--Not Very Likely to 10--Very Likely) The above questions provide an indicator sensitive enough to record change but less prone to random fluctuations common to indicators that are composed of a single question. [This protects States from being sanctioned when random error depresses the indicator's performance level and prevents states from being rewarded for high performance resulting solely from random error.] The satisfaction score will be reported on a 0-100 scale. To simplify reporting to the Federal level, scores for each service category can be aggregated into two satisfaction indices, one for participants and one for employers. E. Comparability Across States Comparability is important for several reasons. First, customer satisfaction performance levels are negotiated along with the core measures. One of the factors affecting those negotiations are ``how the levels compare with state adjusted levels of performance established for other States * * *.'' Comparability also provides for fairness in determining incentives and sanctions. Additionally, comparability contributes to continuous improvement across the system. Having comparable measures will allow benchmarks to be developed to promote continuous improvement. Comparability will also facilitate the sharing of best practices within and among the States. F. When To Measure Consistent with WIA, it is proposed that customer satisfaction be measured at completion of the service. For continuous improvement purposes, it is particularly important to measure customer satisfaction as close to the point of service for the following reasons: - The immediacy of a person's impression makes a significant difference in terms of what he/she will remember; - The highest response rate is obtained at point of service; - Due to the time delay to track outcome-related data (e.g., the core indicators), this immediate customer feedback provides much needed real time data for staff and program managers. The point in time will vary based on the type of customer and level of service received. Participant Customers For self-help/information and core services, the survey will be conducted at the point of contact, immediately after the service is provided. For intensive and training services, the participant will be surveyed after the completion of services (this does not mean necessarily that they have ``exited'' or been ``terminated'' from a program). Additional surveying may be conducted as part of follow-up to determine other aspects of satisfaction. Such surveys are proposed to be optional, given the additional reporting burden they would create. Employer Customers For informational services, the survey will be conducted at the point of contact, immediately after the service is provided. For labor exchange and special services, the employer will be surveyed after the completion of services. G. Using Customer Satisfaction in a Continuous Improvement Process The customer satisfaction indicators, in addition to being a specific part of the performance accountability system, are also the foundation of an organization's strategy for continuous improvement. The indicators provide a guide to achieving the vision and goals of the Act. Additional questions of local importance to customers, program operators and service providers deepen the understanding of how to reach these goals. By adding customized questions, organizations can use customer satisfaction as part of an integrated continuous improvement approach. They can determine where to focus more resources, or redesign programs or sequences of services in order to achieve better results. This use of customer satisfaction will not be federally mandated in order to maintain local flexibility, and to recognize differing approaches in program designs that vary depending upon the service mix and each area's economic and demographic conditions. H. Definition of Measures Measurement of Participant Customers The degree to which participant customers are satisfied with the core, intensive and training services provided by the workforce investment system. Measurement of Employer Customers The degree to which employer customers are satisfied with the informational, labor exchange, and special services provided by the workforce investment system. I. Pilot Testing DOL will work with a number of pilot sites to better determine the range of customer satisfaction levels (i.e., baseline data), and to explore technical issues of survey timing, methodology, and questionnaire construction. The sites will be selected based on interest and previous experience with customer satisfaction surveys. DOL will use the results of the pilot testing and the feedback from this consultation paper to issue guidance or technical standards for the survey methodology. [FR Doc. 99-20119 Filed 8-4-99; 8:45 am] BILLING CODE 4510-30-P