Excerpts from:

East Stroudsburg University has made great strides in the improvement of assessment practices in the 1990’s. Since the last Middle States self-study, the University has achieved the following:

·The General Education program has been evaluated and revised. The new program, based on nearly a decade of work,became effective in June 1996.Baseline data has been gathered to facilitate future assessment of the program and the General Education Outcomes Assessment Plan has been developed (Chapter 4).

·The Council for Strategic Planning(CSP), the University's main vehicle for planning, coordination, and budgeting has been formed.The CSP has operatedeffectively for the past six years and has held two campus-wide conferences, researched the functions of the University, and made useful recommendations. Over 150 faculty and

staff participated in the CSP’s Action Planning Groups(APGs).

·The Office of Institutional Research has been created to provide relevant data about all aspects of the University’s functioning. The OIR has become a great resource and has disseminated much needed information, including the ESU Fact Book.

·The Center for Teaching and Learning was formed in 1991. The CTL coordinates professional development through seminars, theme meetings, and discussions and also houses resources for innovative educational practice, including outcomes assessment. All activities associated with the CTL are designed to enhance faculty teaching performance and student learning.

·The University-wide Writing Committee has been created and has investigated writing practices, offered faculty writing workshops, and made recommendations concerning the quality and quantity of writing done at ESU.

·The Outcomes Assessment Advisory Committee was formed in 1993 to advise and coordinate outcome assessment efforts.The Committee, composed of faculty, administrators, and students evolved from on-going assessment efforts and became institutionalized three years ago.

·The University participated in a System-wide program review process in which all academic departments and programs are required to provide outcomes assessment results and be examined by outside reviewers.

·Procedures for evaluating non-academic units have been revised and updated by incorporating these programs into the five-year program review process.

·Academic and non-academic student support services have increased to enhance student learning.

·Campus-wide conferences and meetings on outcomes assessment, including internationally known speakers, have been held.

·University representatives have attended national, regional, and state conferences on outcomes assessment and program evaluation.

·The University has participated in the system-wide Continuous Improvement program.

·The University has developed and implementedthe ESU Outcomes

AssessmentPlan (Chapter 2).

University Support

Institutional effectiveness cannot be evaluated nor changes effected without a great deal of support from the University Administration.ESU's administration has provided such support in many ways. The President and the Provost, strong proponents of evaluation and change, have been instrumental in forming the Council for Strategic Planning, the Office of Institutional Research, and the Center for Teaching and Learning.Support has also come from the Office of Human Resources, the Faculty Development Fund, and the Office of Career Services.

The Council for Strategic Planning (CSP) has become the major planning body for the University.CSP has formulated a vision statement for the University's future, and will present to the University for discussion a five-year integrated and prioritized plan based on the evaluations and plans of each segment of the University.Because each unit's future place in the University’s priorities is based upon its evaluation, and because the overall plan is updated annually, each unit has a vested stake in its own evaluative processes and results.

The CSP played a central role in forming the Outcomes Assessment Advisory Committee(OAAC) and has also facilitated the inclusion of outside expert reviewers in the program review process. External review allows programs to gain a broader perspective on their current professional standing during self-evaluations.

The CSP evaluates its own effectiveness through a process of introspection based upon almost constant feedback from the entire University community and through a five- year self-study.

The Office of Institutional Research (OIR) collects data and generates reports related to many aspects of the University's functioning and makes this information available to any unit that needs it for planning or evaluation.The Office’s effectiveness is evaluated through feedback from users of its reports and through comparisons to System-wide and professional standards. The OIR has been a valuable resource in evaluating University and/or unit functions, and is potentially the University’s best information source. However, its usefulness has been limited by staffing constraints.

Created in 1991 and institutionalized in 1993, the Center for Teaching and Learning (CTL) has been active in promoting professional development.During the next two years the CTL and the Outcomes Assessment Advisory Committee will focus on outcomes assessment by providing training for faculty and administrators and holding University-wide meetings to develop outcomes assessment procedures for all divisions of the institution.

The Office of Career Services surveys alumni during the first year after graduation on an annual basis and graduates' employers every five years.This information has great potential for evaluation purposes, but its use has been restricted thus far because of low return rates, oversimplified questionnaires, and minimal data analysis. As a result, many departments are using their own resources to communicate with their graduates.This contact is sometimes successful, sometimes not. Follow-up of graduates is one of the University’s least effective aspects of evaluation and needs to be strengthened.

The Office of Human Resources officially coordinates the University's Continuous Improvement(CI) activities.It has trained all administrative personnel in CI, actively supports several CIgroups, and has recently formed the CI Coordinating Council. Representing all University constituencies, this Council will attempt to broaden interest and activities in support of CI.This function is closely tied to the SSHEContinuous Improvement Initiative.

In addition to these major support functions, the University has supplied small amounts of financial support for various assessment activities. The Provost and the Deans have funded attendance at national, regional, and state assessment conferences to promote ongoing professional development.The Grants Office, through the Faculty Developmentand Research Fund and the University Foundation, has funded some faculty assessment studies.The Provost provides a small annual budget for the OAAC, some of which was used to support a study of academic advising and a study of students' perceptions of General Education.

State System Support

Pennsylvania's State System of Higher Education (SSHE) has provided excellent leadership and support for finding better ways to educate our students and providing a large pool of professional expertise for the state's economy.The SSHE Board of Governors has offered mandates, and the Chancellor's Office has supported the system's universities in their attempts to improve their effectiveness.

The 1990 SSHE plan called for greater attention to student outcomes, especially regarding General Education.ESU responded by redesigning its General Education Program to reflect more innovative educational perspectives. Since 1990, SSHE has sponsored several system-wide conferences on this topic, two of which were followed up by grants to support local outcomes assessment at each of the system's universities. ESU used its grants to conduct research and sponsor speakers about outcomes assessment practices.The Vice Chancellor for Academic Affairs has personally visited each university several times to check on assessment progress and to provide support.

In accord with the Middle States Framework for Outcomes Assessment, the 1996 System plan puts a great deal more emphasis on assessment of learning outcomes.In keeping with this emphasis, SSHE has organized the Outcomes Assessment Resource Group(OARG) composed of representatives from all system universities and designed to provide greater impetus for outcomes assessment and to determine how resources and ideas can be shared among the system's universities. A dean and a faculty member represent ESU on the OARG.In conjunction with OARG, ESU sponsored a day-long outcomes assessment seminar for SSHE department chairs in November 1996. Dr. James Nichols, a nationally known expert in the assessment field, led the program. This seminarprovided the opportunity for department chairs across the state to share ideas and address concerns about outcomes assessment.

SSHE has also contracted a System-wide Student Opinion Survey concerning student satisfaction with all aspects of student life and learning and is planning to conduct a separate follow-up survey of SSHE graduates' job placements and employers' satisfaction.These data will be given to each SSHE university, so it can compare itself with the System as a whole. This information is valuable for the continuous improvement process. These and other system-wide surveys will be completed on a regular basis.

The Continuous Improvement Initiative, created by the Chancellor’s Office, has focused on developing System-wide accountability indicators related to factors such as retention rates, graduation rates, and faculty productivity.These indicators are based on the work of the National Joint Commission on Accountability Reporting.

Further the Continuous Improvement Initiative emphasizes that all human resources offices in SSHE universities should participate in Continuous Improvement programs. These programs are to be coordinated with strategic planning at each university, supported by funds from business and industry and expertise from the Chancellor's Office.The Vice Chancellor for Human Resources has set up a Continuous Improvement Office that coordinates programs at each of the fourteen SSHE universities and arranges several state-wide meetings each year.

Commitment to Continuous Improvement is first among the Chancellor's goals for the System.Reports of progress are sent to the Board of Governors, the Office of the Governor, and the Pennsylvania State Legislature.

University-Wide Assessment and Evaluation Procedures

ESU's process for evaluating academic departments consists of mandatory, systematic reporting by the department to the University Administration, in addition toassessment by departments of all academic programs (majors, minors, and concentrations).All extra-departmental or interdepartmental programs (General Education, Gerontology, Women's Studies, Honors Program) are coordinated by committees responsible for curriculum decisions and assessment procedures.

This process allows for a standard evaluation of departmental management functions yet enables a flexible approach to course and curriculum assessment, so that the most effective ways for assessing the results of disparate programs with different academic goals can be designed. This means every department's personnel, office, and budgetary functions are evaluated the same way, but academic programs are not all assessed in the same way.The staff of individual programs must be responsible for planning and carrying out appropriate assessment.

ESU also has an effective monitoring mechanism: the System-wide mandatory program review process.Every five years, each academic department or program must carry out a thorough self-study of all its operations. This required review has been the strongest impetus for departmental outcomes assessment because the System’s Board of Governors has mandated that each program review must incorporate outcomes assessment results.The self-studies are reviewed by the appropriate Deans and by the Provost, who must provide each department with feedback about the adequacy of its self-study, and also discuss the budgetary implications of the department’s program review.The University budgetary process must take program review results into consideration.

In addition, each department must follow contractual processes for continual evaluation of faculty teaching, scholarship, and service.Nontenured faculty must have student evaluations for all of their courses, peer class observations every semester, annual chair observations, yearly departmental evaluation committee scrutiny, yearly chairperson reports, and annual dean review of all this information.Tenured faculty are evaluated the same way, but on a five-year cycle.This information must be used for any promotion or tenure decisions.

Departments which undergo professional certification or accreditation must include their professional self-studies in their program reviews. Results of the evaluations are reported to the University's Administration.

Administrative units also participate in the five-year review process. Evaluations of current operations, suggestions for long term improvement, rationales supporting these suggestions, and plans for evaluating expected progress are included.These plans are used to justify budgetary support.

All of these assessments and evaluations are considered in the prioritization process which formulates the University's planning goals and budgetary decisions.

Additional Studies

In addition to the formal and periodic evaluations built into the University's information and planning system, some studies are carried out by various groups and are distributed to administrators and faculty.For example, the Outcomes Assessment Advisory Committee surveyed academic department chairs concerning current and planned assessment procedures; the Writing Committee surveyed faculty and department chairs about the amount and quality of writing already in place at the University and also solicited suggestions for topics to be addressed in faculty writing workshops; one of the CSP’s Action Planning Groups conducted a survey of student attitudes toward various aspects of University functioning; and some faculty members and student organizations conducted studies (e.g. a psychology professor used a Faculty Development and Research Grant to run a one-semester freshman experience course and distributed his report; a student honorary society reported results of a survey of students' perceptions of the academic advising system).

Also, several studies related to the success of specific populations of ESU students have been conducted by individual faculty members. One such study compared the academic progress of students who were required to complete a remedial composition class with those who were proficient enough to place into the credited English composition course (Hawkes, 1992). Another study (Francois & Rheinheimer, 1996) compared non-traditional/adult students with traditional-age students. Also completed was a longitudinal study (Miller, 1994) that reviewed the success of underprepared students who entered the University through a summer bridge program. This study compared the progress of these students based on entry reading levels, studied the transfer of strategies to content courses, and piloted an authentic contexturalized reading assessment instrument.

Aside from program reviews and other formal evaluation procedures, much periodic evaluation information is gathered in the context of formal accreditation procedures.For the University's Middle States accreditation every five years, many evaluative studies are completed.

Outcomes Assessment Advisory Committee

Prior to 1990, some departments had already established good assessment processes originating from professional concerns, national standards, or requirements of professional accrediting agencies.Many departments, however, had little knowledge of current innovative outcomes assessment practices.

In 1990, ESU's new Council for Strategic Planning (CSP) created an Action Planning Group (APG) charged with studying this issue and making recommendations for a University-wide outcomes assessment procedure.APG members attended assessment conferences and reviewed the literature on this topic.The APG's 1991 report outlined a full-scale, multi-year plan. Unfortunately, this plan was unworkable, mostly due to cost considerations.In 1992, the APG recommended that the University create a permanent committee charged with developing a workable plan for outcomes assessment and bringing it to fruition.

In 1993, the Outcomes Assessment Advisory Committee (OAAC) was formed. In 1994,it developed a series of guidelines for assessment at the University and endorsed the Middle States definition of outcomes assessment. The Committee participated with an APG in 1993-1994 in developing and recommending procedures for the implementation of the SSHE-mandated program review. These procedures are now University policy.

Outcomes Assessment Advisory Committee Survey

The Outcomes Assessment Advisory Committee surveyed twenty academic departments at the University.Eighteen departments responded. A summary of the results indicated the following:

·100% had specific program objectives that had been reviewed within the past five years;

·50%had capstone courses or senior seminars as part of their programs and 33%had this issue under discussion;

·83% required students to complete an internship or student teaching and 11% had this issue under discussion;

·78% had internships or student teaching programs that required on-site supervisors to evaluate student performance and 6% had the issue under discussion;

·44% indicated that their students engaged in some type of senior performance, while 39% noted that this issue was under discussion;

·94% monitored student progress toward graduation;

·72%had contact with program graduates; 5% expected this within one year; 22% described it as under discussion;

·56% knew if their students felt well-prepared for graduate school; 28% had this issue under discussion.

The data collected supported the existence of a viable foundation for the development of outcomes assessment in the responding departments. The fact that 100% of those participating cited the existence of program objectives that were periodically reviewed was particularly valuable.

In addition, OAAC publicized the need for outcomes assessment and met with the University's General Education Committee to formulate plans for assessing that program.The OAAC also sponsored a student-initiated study of academic advising.Recommendations for improvements in ESU's academic advising process, to be incorporated in the current revision of the Academic Advising Handbook, are under discussion. Further, faculty participation in assessment conferences and workshops has been and will continue to be funded by the administration (e.g. American Association of Higher Education, 1994 & 1996).

Task Force on Institutional Effectiveness

In the Fall of 1995, the OAAC was temporarily reconstituted as the Task Force on Institutional Effectiveness for the purposes of ESU's Middle States Self-Study, and appropriate additions were made to the Task Force's membership. The Task Force on the Assessment ofInstitutional Effectiveness was charged with identifying, examining, andmaking recommendations relative to how the University determines if the ways in which it is evolving and changing are beneficial to the institution and consistent with its stated mission and goals. To that end, the Task Force gathered assessment information concerning all aspects of the operation of the institution and surveyed academic departments and administrative units about their decision-making processes.

Based on the data gathered, the Task Force on Institutional Effectiveness drew the following conclusions concerning assessment strategies at ESU:

·Change at East Stroudsburg University emerges from a diversity of sources ranging from innovations in national professional standards to faculty and student feedback. Further, it is clear from the responses that both academic and administrative units are actively engaged in the systematic assessment of on-going operations.

·Academic and administrative units employ diverse formal and informal assessment strategies including research projects, capstone courses, standardized tests, professor-designed tests, performances, and portfolios( Figure 1).

·The academic and administrative units indicated that the information gained from a variety of assessments is used to make decisions and effect change.

·A majority of the units responding to the Task Force survey acknowledged the students' role in program assessment.

·While academic and administrative units have demonstrated that they use a variety of assessment strategies to gain insight into changes that may be needed, whether the implementation of change includes specific strategies to assess the effectiveness of that change is unclear.

Figure 1

·Responses to the survey indicated that each academic and administrative unit is concerned with the population it serves and institutes change to offer more innovative programs that support the University’s mission. However, whether individual units consciously consider the effects on the University as a whole is unclear.

Other Self-Study Research

Accreditation self-studies, themselves, are excellent vehicles for the evaluation of institutional effectiveness.Data gathered for this purpose fulfill many ofthe requirements of accountability and improvement. Figure 2 lists the studies completed by various Task Forces.

Figure 2

Chapter Two

The ESU Outcomes Assessment Plan

Maureen McLaughlin

Outcomes assessment is a process by which evidence for congruence between an institution's stated mission, goals, and objectives, and the outcomes ofits programs and activities is assembled and reviewed in order to improve teaching and learning (Middle States Commission on Higher Education’s Framework for Outcomes Assessment).

In the Spring of 1996, the Outcomes Assessment Advisory Committee and the Center for Teaching and Learning developed a plan for implementing outcomes assessment throughout the University(Figure 3). While this process is dynamic in nature, the timeline delineates activities for the initial two-year period. Subsequent planning will be based upon progress andparticipant feedback.

As indicated by the SSHE State Assessment Plan, making the transition from traditional assessment practices to outcomes assessment involves a need to reconceptualize teaching and learning. To accommodate this paradigm shift,the ESU outcomes assessment efforts focus on the following:

·the development and implementation of University outcomes that support the University Mission Statement;

Figure3

·the development and implementation of outcomes assessment by each academic and administrative unit;

·the development and implementation of outcomes assessment in courses, including the broadening of classroom assessment practices, with particular emphasis on the integration of performance assessments;

·the use of the results of outcomes assessment for continuous improvement;

·professional development focusing on outcomes assessment and other innovative assessment practices.

Professional development is a critical component of the plan . The Outcomes Assessment Newsletter, the Outcomes Assessment Handbook, University-wide retreat, SSHE Department Chair Seminar, and faculty workshops support this.

Figure 4

During the Fall of 1996, the process of developing outcomes began at the University level. Individual academic and administrative units began developing and implementing outcomes assessment as part of their plan for continuous improvement. Outcomes will then be developed at the course level. Figure 4 shows the two-way interaction of this process. The down arrows illustrate the outcome planning process. The up arrows illustrate the congruence between the stages: the course outcomes support the program outcomes, which, in turn, support the University outcomes.

Figure 5 demonstrates the outcomes assessment process from the academic perspective at the course level. This process includes what the desired course outcomes are (goals); what is being taught (curriculum); how the course is being taught (instruction); how well students are learning (assessment); and how the results are used (improvement). This provides a direct link among curriculum, instruction, assessment, and improvement.

Figure 5

The Teaching and Learning Continuous Improvement Loop

Course Level

Figure 6 delineates the outcomes assessment process from the administrative perspective at the unit level. It incorporates what the outcomes of its work will be (goals); what the unit does (purpose); how the work takes place (performance); evidence of how well it functions (assessment); and how the results are used (improvement).

These models illustrate that the proposed outcomes assessment plan is a decentralized process. The information is developed at the level at which it will be used for meaningful continuous improvement. Progress in developing effective outcomes assessment will be reported through the academic and non-academic program review process.

Figure 6

The Administrative Continuous Improvement Loop

Unit Level

Implementation of the Outcomes Assessment Plan began with ameeting of representatives from the Outcomes Assessment Committee and the Center for Teaching

Figure 7

and Learning during the Spring and Summer of 1996. The major thrust of the plan began in the Fall of 1996 and has progressed through diverse experiences:

A University-wide retreat was held on September 28, 1996, at whichDr. Donald Farmer, a member of the advisory committee on assessment for the Commission on Higher Education of the Middle States Association, addressed 130 members of the faculty, administration, and student body. During his presentation, Dr. Farmer shared essential characteristics of effective assessment (Figure 7).

Dr. Farmer also facilitated the development of outcomes at the University level through a workshop format. These are illustrated in Figure 8.

Figure 8

·A seminar for department chairs on November 1, 1996, was co-sponsored by the SSHE Outcomes Assessment Resource Group and ESU’s Outcomes Assessment Advisory Committee and Center for Teaching and Learning. ESU hosted the event in the Keystone Room. James Nichols, author of A Departmental Guide and Record Book for Student Outcomes Assessment and Institutional Effectiveness, led the seminar in a framework which allowed department chairs from numerous SSHE institutions to interact. This context furthered the SSHE emphases on collaboration and cooperation. Nichols’ topics included: understanding assessment of student learning, formulating outcome statements, identifying means of assessment and criteria for success, use of results for program improvement, and documenting the use of assessment results to improve programming.

·A workshop focusing on the development of outcomes assessment at the department/program level was offered on November 22, 1996, by members of the Outcomes Assessment Advisory Committee and the Center for Teaching and Learning. The goal of this workshop was to assist department and program personnel in the writing of outcomes. As a follow up task, individual departments and programs employed a three month span of time to meet to create three outcomes, describe example assessments, and explain ways in which the results have been used for continuous improvement (Chapter 3). These were completed in February 1997.

·On Friday, April 29, 1997, Jay McTighe, founder of the Maryland Assessment Consortium and nationally renown speaker, will address the University-wide Outcomes Assessment Retreat II. His presentation on outcomes assessment at the course level, emphasizing performance and portfolio assessment, will set the stage for the 1997-1998 emphasis of the ESU Outcomes Assessment Plan.

·The ESU Outcomes Assessment Newsletter is keeping faculty and administration apprised of developments at the national, state, and local levels (Appendix D). The first edition of the newsletter was disseminated on May 14, 1996.The second volume began in August 1996 and has subsequently been published twice a semester.

·The ESU Outcomes Assessment Handbook, available in March 1997, includes the following: information about the theoretical framework of outcomes assessment; assessment terminology;resources to facilitate the development of outcomes assessment; examples of outcomes from various departments and programs; examples of performance and portfolio classroom assessment; and annotated resources.

·Other outcomes assessment resources will continue to be available including attendance at professional conferences and access to professional publications.

It is important to note that the ESU Outcomes Assessment Plan is collaborative in nature. The seminars have been designed and will be coordinated by the Outcomes Assessment Advisory Committee and the Center for Teaching and Learning. The newsletter is published by the Outcomes Assessment Advisory Committee and the handbook is being developed by faculty, students, and administrators. Additional support is being offered by those units that are already using outcomes assessment and faculty members who employ performance assessment and portfolio assessment in their courses. The University Administration takes an active role in these efforts.

Alignment with the ESU Mission Statement

The purpose of the outcomes assessment movement on campus is to clearly demonstrate how the University’s mission is being accomplished. The ESU Outcomes Assessment Plan illustrates how various levels of outcomes assessment document the achievement of the University ’s mission: “to provide high quality programs in both traditional and emerging fields of study which recognize and promote human and intellectual diversity, and to prepare graduates to enter a complex, changing global society with competence and confidence” (ESU Undergraduate Catalog, 1996-1997).

Alignment with State System of Higher Education

In accord with the SSHE State System Plan (May 1996) and the 1997 Board Policy onOutcomes Assessment (Appendix C), this model is learning-centered. It supports the shift from the traditional instructional delivery paradigm to “ a learning paradigm in which faculty determine expected learning outcomes and faculty and students alike assume greater responsibility for learning.” Further, the learning-centered university “refocuses leadership, management, and support services in terms of their impact on learning.”

Alignment with the Middle States Framework for Outcomes Assessment

The definition of outcomes assessment which serves as the foundation of the ESUOutcomes Assessment Plan is taken from the Framework for Outcomes Assessment. Further, the University plan is aligned with the Framework’s theoretical underpinnings and accommodates the “Paradigm for Institutional Effectiveness.”

The ESU Outcomes Assessment Plan is a strength. It is dynamic, decentralized, and offers meaningful information for continuous quality improvement of courses, academic and administrative units, and, therefore, the University .

Prioritized Recommendations

Based on the qualitative study and current assessment practices, the Task Force on Institutional Effectiveness makes the following recommendations concerning outcomes assessment at East Stroudsburg University:

1. Faculty, administration, students, and staff should continue efforts to bring the ESU Outcomes Assessment Plan to fruition. The plan is a University effortthat requires a commitment from all members of the ESU community. This plan entails more than writing outcomes; it involves embracing an innovative assessment perspective: shifting from a teacher-centered paradigm to student-centered pedagogy.

2. The University should increase the visibility and improve the coordination of outcomes assessment in academic and non-academic programs.

3. To accommodate innovations in outcomes assessment on campus, the University should create financial resources, such as a competitive grants program. This recommendation from the original task force report is in the process of being addressed through the Continuous Improvement Grant Program.

4. To provide better analysis and wider dissemination of information about our graduates, improved procedures for surveying alumni and their employers should be implemented. Departments and offices should be encouraged to use the centralized database of alumni addresses maintained by the Office of Alumni Relations.

5. The Center for Teaching and Learning should continue to serve as an outcomes assessment resource center that holds current professional publications, conference information, program review formats, grants information, audiotapes, survey instruments, and videotapes.

Conclusion:

ESU has developed and begun to implement a viable Outcomes Assessment Plan. The plan has a sound theoretical framework and is supported by survey results and progress the University has made in assessment practices during the past five years. Recommendations made by the Task Force for Institutional Effectiveness in this report are facilitating bringing the plan to fruition. This is a University effort. It requires a commitment from all members of the ESU community.

Outcomes Assessment Handbook Volume I