Assessment Plan – Year 0 Report

College:  Education, Health, & Human Development
Department: Education
Submitted by: Sarah Pennington
 

Year 0 Assessment Plan Report is due September 15th .

 

Indicate all majors, minors, certificates and/or options that are included in this new assessment Plan

Majors/Minors/Certificate Options
Doctor of Education - Education (Ed.D.) Adult & Higher Ed: Higher Education Academics
  Adults & Higher Ed: Higher Education Administration
   

Part 1: Program Learning Outcomes (PLOs)

PLOs should be written as specific, measurable statements describing what students will be able to do upon completion of the program. The assessment of PLOs provide feedback on the expected knowledge, skills, and attitudes that students develop as they progress through their program.
List the program learning outcomes:

PLO# PLO Description
1. Demonstrate disciplinary skills and knowledge, applying these to analyze problems.
2. Develop meaningful solutions to complex problems of practice from a position of equity and social justice.
3. Integrate personal, practical, theoretical, and empirical knowledge with systemic and systematic inquiry.
4. Communicate effectively using multiple modes.
5. Conduct scholarly and professional activities in an ethical manner.

Part 2: Development of Assessment Plan

Each plan will require the following information:
Threshold Values: Along with PLOs, plans should include threshold values; minimums against which to assess student achievement for learning outcomes. Threshold values are defined as an established criteria for which outcome achievement is defined as met or not met.
Methods of Assessment & Data Source:  Assessment plans require evidence to demonstrate student learning at the program level. This evidence can be in the form of a direct or indirect measure of student learning. Both direct and indirect assessment data must be associated with the program’s learning outcomes. An assessment rubric will also need to be included that demonstrates how evaluation of the data was used to assess student achievement.
Timeframe for Collecting and Analyzing Data:  Develop a multi-year assessment schedule that will show when all program learning outcomes will be assessed. As graduate assessment reports are biennial, faculty review of assessment results may only occur every other year, however, annual faculty meeting to review these data and discuss student progress may be beneficial.

2a. Curriculum Map

ASSESSMENT PLANNING CHART
Program Learning Outcomes Course Alignments:
Include rubric, number and course title
Identification of Assessment Artifact
1, 5 EDU 600 – Doctoral Seminar Literature Review
1, 4, 5 EDU 690 – Doctoral Thesis Doctoral Dissertation
1, 2 EDLD 605 – Higher Education History & Philosophy Paper 3 (American Indian HE); Final Paper
3 EDLD 616 – Organization & Administration of Higher Education Final Position Paper
3, 4 EDLD 628 – College Students Research proposal & Presentation
  EDU 637 – Institutional Research & Assessment  
3 EDU 614 – Program Evaluation Executive Summary of Final Evaluation Report
3, 5 EDU 610 – Qualitative Research Research Proposal/Project/IRB app
3, 5 EDU 607 – Quantitative Educational Research Research Proposal/Project/IRB app
3, 4 Program Benchmark: Comprehensive Exams Written comprehensive exams with oral defense
ASSESSMENT SCHEDULE
  Year to be assessed
PLO Course 2022-2023 2023-2024 2024-2025 2025-2026 2026-2027
1 EDU 600 – Doctoral Seminar   X     X
1 EDU 690 – Doctoral Thesis   X   X  
1 EDLD 605 – Higher Education History & Philosophy X        
2 EDLD 605 – Higher Education History & Philosophy X        
3 EDLD 616 – Organization & Administration of Higher Education X        
3 EDLD 628 – College Students   X      
3 EDU 614 – Program Evaluation     X    
3 EDU 610 – Qualitative Research X     X  
3 EDU 607 – Quantitative Educational Research     X    
3 Program Benchmark: Comprehensive Exams     X    
4 EDU 690 – Doctoral Thesis   X      
4 EDLD 628 – College Students X     X  
4 Program Benchmark: Comprehensive Exams     X    
5 EDU 690 – Doctoral Thesis   X      
5 EDU 610 – Qualitative Research X     X  
5 EDU 607 – Quantitative Educational Research     X    
             

Part 3: Program Assessment

The assessment plan will need to include: 1. how assessment will be conducted; 2. who receives the analyzed assessment data, and 3. how it will be used by program faculty for program improvement(s).

1) How will assessment artifacts be identified?

The adult and higher education program faculty in collaboration with the Director of Accreditation and Department Head. Identification of artifacts will consider how effectively each artifact provides evidence of the appropriate PLO.

The EdD Higher Education Academics specialization is intended for individuals who will teach and/or provide academic leadership and support in the area of college teaching and learning within a diverse range of post-secondary settings. Individuals pursuing this option may hold a Master's degree within higher education or another discipline in which they will teach or provide academic leadership. The EdD in Higher Education Administration specialization is intended for individuals who will contribute to the administrative leadership and support within a range of diverse post-secondary settings. The primary objective of the degree is to produce informed scholarly practitioners for mid- to upper-level management or administrative positions. These individuals typically manage or direct either academic or student affairs operational functions. 

EDLD 616: The final paper for EDLD 616 will be assessed for evidence of the program learning outcomes. This paper synthesizes students’ work over the semester on a problem faced in administration, resulting in implications for practice in higher education or student affairs. The paper is between 6-8 pages in length and prompts students to analyze and evaluate research evidence to propose a solution to their problem of focus.

EDU 628: Ethical Considerations, Validity, and Reflection: There are four areas to cover as part of the ethical considerations. One, describe any ethical considerations such as power differences, sensitive populations, anonymity, and how you will address them in the study. Two, explain how you will adhere to ethical standards for practice and any potential concerns. Three, discuss any limitations to the evaluation based on the methodology. Four, after the evaluation you will provide a reflexivity of your experiences such as pressures, bias, or problems associated with the evaluation and how you managed or mitigated them. Problems may include but are not limited to financial or resource constraint, political pressures, responsiveness of participants or leadership, other.

Comprehensive Exams/Dissertation


2) How will they be collected (and by whom)?

Data will be collected by the Graduate Programs Coordinator, course instructors, and committee chairs as appropriate. Data may include course and project-specific grades as well as results of comprehensive exams and dissertation defenses.   

b. How sample will be collected?
All final papers from the course were collected, and those completed by students in the M.Ed. program will be assessed for the learning outcomes. Samples will be collected through the D2L Course Home Page typically in the Assignment Drop Box: This is where students submit formal assignments. In the Drop Box Forum, students see the titles for each formal assignment. Students click on the tile of the corresponding assignment you want to submit and add the document as a file and submit.  A census sample will be collected for the comprehensive exams for both Fall and Spring semesters.

3. Data Analysis
a. Who participated in the process, the nature of the rubric utilized (or other norming methods), and the threshold outcome desired.  

For EDLD 616, the rubric developed to assess the final papers was informed by both the specific course learning objectives and the broader program learning outcomes as stated above. Three outcomes, reflective of the two program learning outcomes being assessed, were assessed for the paper along a 10-point scale:

Criterion Exemplary
(10 points)
Proficient
(8 or 9 points)
Developing
(7 points)
Unacceptable
(6 or fewer)
Discussion of problem Work shows evidence of synthesis, summary, and critique of empirical findings and professional experience to articulate understanding of problem; demonstrates use of at least 10 empirical articles in support Work shows evidence of selection and explanation of empirical findings, and some professional experience, to articulate understanding of problem; demonstrates use of at least 7 empirical articles in support Work shows some evidence of reliance on empirical findings to articulate understanding of problem, but leans heavily on experience; demonstrates use of at least 5 empirical articles in support Work shows little evidence of reliance on empirical findings to articulate understanding of problem and tends to solely situate conclusions in personal experience; does not include at least 5 empirical sources
Appropriateness of proposed solution Work demonstrates ability to infer potential solutions from understanding of problem; work fully evaluates proposed solution and one alternative (particularly in relation to empirical support) Work demonstrates ability to identify potential solutions from understanding of problem, but may fail to fully justify connection; work comprehensively but not fully evaluates proposed solution and/or one alternative Work outlines major elements of proposed solution but fails to fully connect these elements to problem; evaluation of proposed solution not fully developed, alternative omitted Work fails to connect solution to problem and fails to fully explain solution and to provide an alternative
Feasibility of implementation plan Work demonstrates evidence of application of frames and conceptual knowledge of the higher education system to problem in a manner that depicts a thorough, feasible, multi-faceted approach to influence change toward the proposed solution to the identified problem; work also pays heed to potential weaknesses and limitations of implementation plan Work demonstrates evidence of application of frames and conceptual understanding of the system that supports implementation plan feasibility, but fails to assess potential program weaknesses or limitations Work demonstrates explanation of feasibility and depth of proposed implementation plan, but fails to explicitly connect to identified problem and/or conceptual understanding of higher education; also fails to attend to potential weaknesses or limitations Work fails to fully demonstrate feasibility and thoroughness of implementation plan, particularly in relation to frames and conceptual understanding of the system; fails to attend to weaknesses and/or limitations

 

EDLD 614: 

  Developing Emerging Sophisticated
Poster Elements The element is overly generalized, lacks precision, with little evidence of support and synthesis The element is more general, yet with reasonable support and adequate synthesis The element is original, creative, specific, substantiated, defendable, and complex
Title      
Positionality      
Evaluation questions & evaluand context      
Critique of related literature      
Theory driven      
Methodological cohesion and soundness      
Analysis of data      
Conclusions and recommendations to important stakeholders      
Clear, well organized, and visually aesthetic representation to all important stakeholders      

 

EDU 628:

Learning Outcomes 1 and 2
The AHE program faculty formalized their comprehensive exam, these efforts were intended to (1) make the process and expectations clear to students and faculty, (2) ensure consistency, and (3) systemize the execution of the procedures and assessment. The processes for the procedures are in Appendix A. We expect every student to pass the important milestones associated with degree completion. The committee chair will assess the students’ readiness for their exams. This proactive strategy uses a developmental approach to ensure a successful outcome for the student. If the committee chair thinks that the student may need further development in a content or research area, the chair will provide the students with additional resources and recommendations for preparation for the exam. Therefore, our program strives for a 100% success rate for comprehensive exams based on the use of these proactive and developmental procedures.

AHE Rubric for Assessing Learning Outcomes
Introductory Developing Mastery
Remembering: Student is able to recall information. Key words for assessment include: define, list, recall, and reproduce. Understanding: Student is able to explain ideas and concepts. Key words for assessment include: classify, described, discuss, explain, translate, and paraphrase. Applying: Students is able to use information in a new way. Key words for assessment include: choose, demonstrate, illustrate, interpret, solve, and write. Analyzing: Students is able to distinguish between different concepts/parts to recognize relationships and patterns. Key words for assessment include: compare, contrast, examine, experiment, question, test, criticize, and differentiate. Evaluating: Student is able to justify a position or decision. Key words for assessment include: argue, defend, judge, support, value, and appraise. Creating: Student is able to create a new product or point of view. Key words for assessment include: assemble, construct, create, design, develop, and formulate.
1 2 3 4 5 6
This rubric was created using Bloom’s Taxonomy

 

The program does not currently use a common rubric for assessing students’ work. The faculty are working on a common rubric that can be used for all five learning outcomes above. Here is a draft of the rubric that should be implemented in future reviews. It is expected that MEd students would meet the “developing” or “mastering” criteria

Dissertation/Comprehensive Exams
Pass rates will be used to assess learning outcomes one and two listed above. All XX EdD students who took their comprehensive exams passed, resulting in a XX% pass rate. 

Dissertation
Pass rates will be used to assess learning outcomes one and two listed above. All XX EdD students who took their comprehensive exams passed, resulting in a XX% pass rate. 


4. Results
a. What was revealed from analyzing learning outcome data regarding student learning?  Describe any result, pattern, or trends that you identify as meaningful. 

EDLD 616
On average, M.Ed. students scored xxx on discussion of the problem, reflecting mastery of content; and xx each on appropriateness and feasibility of proposed solution, reflecting proficiency (but not mastery) of developing potential solutions—around x-x on the proposed six-point scale below. In general, our master’s students appear to be reaching the desired program learning outcomes as a result of participation in EDLD 616 which is then reflected in the integration of their practical knowledge with course material.
EDU 628:
Students will be assessed on the overall contributions they make to the class as a whole including class meetings and online engagement as well as on the attributes of their written/oral submissions.


Comprehensive Exam/Dissertation

3) Who will be assessing the artifacts?

Program faculty, in collaboration with the Graduate Programs Coordinator, Director of Accreditation, and Department Head, will be assessing data on an annual basis to identify areas for program improvement. Additionally, program learning outcome assessment scores for the specific artifact assignment
will not influence the student’s earned grade in the course.

Part 4: Program Assessment Plan

All plans must include assessment rubrics (the methodology of how student artifacts are to be assessed, and a threshold for student success attainment. (The chart below is an example of the information requested…you can configure your rubrics in different ways)

PLO #1 Demonstrate disciplinary skills and knowledge, applying these to analyze problems. Threshold Values
Indicators Level 4 Level 3 Level 2 Level 1 80% of students will meet or exceed Level 3
competency
Identification of problems Provides multiple perspectives on a problem relevant to their field of study Describes the complexity of a problem relevant to their field of study Identifies a problem relevant to their field of study Partially identifies a problem OR identifies a problem that is not clearly relevant to their field of study  
Contextualized analysis of problem(s) Applies disciplinary skills and knowledge to thoroughly analyze a problem relevant to their field of study resulting in a nuanced discussion of perspectives Applies disciplinary skills and knowledge to thoroughly analyze a problem relevant to their field of study Logical analysis of a problem relevant to their field of study is guided by elements of disciplinary skills & knowledge Analysis of a problem relevant to their field of study is incomplete and may include assumptions and statements not supported by disciplinary knowledge.  
PLO #2 Develop meaningful evidence‐based solutions to complex problems of practice from a position of equity and social justice.  
Indicators Level 4 Level 3 Level 2 Level 1 80% of students will meet or exceed Level 3 competency
Meaningful evidence-based solutions Designs a study utilizing appropriate, evidence-based methods firmly grounded in theory and previous literature (or gaps therein) to study a complex problem of practice relevant to their field Designs a study utilizing appropriate, evidence-based methods grounded in theory and/or previous literature (or gaps therein) to study a complex problem of practice relevant to their field Designs a study utilizing minimally appropriate, evidence-based methods which may or may not be grounded in theory and/or previous literature (or gaps therein) to study a problem of practice relevant to their field Designs a study that does not utilize appropriate, evidence-based methods grounded in theory and/or previous literature (or gaps therein) to study a problem of practice relevant to their field  
Social Justice & Equity Integrate understanding of social justice and equity into investigation of solutions related to a complex problem relevant to their field, resulting in solutions that support the vitality and self-determination of diverse stakeholders. Integrate understanding of social justice and equity into investigation of solutions related to a complex problem relevant to their field Apply minimal understanding of social justice and equity into investigation of solutions related to a complex problem relevant to their field Does not apply understanding of social justice and equity into investigation of solutions related to a complex problem relevant to their field  
PLO #3 Integrate personal, practical, theoretical, and empirical knowledge with systemic and systematic inquiry.  
Indicators Level 4 Level 3 Level 2 Level 1 80% of students will meet or exceed Level 3 competency
Synthesis Synthesizes personal knowledge, as well as practical, theoretical, and empirical knowledge from the literature to inform systemic and systematic inquiry. Synthesizes personal knowledge, as well as two of the following: practical, theoretical, or empirical knowledge from the literature to inform systemic and systematic inquiry. Synthesizes personal knowledge, as well as one of the following: practical, theoretical, or empirical knowledge from the literature to inform systemic and systematic inquiry. Does not synthesize personal knowledge, as well as practical, theoretical, and empirical knowledge from the literature to inform systemic and systematic inquiry.  
Critical Analysis Critically analyzes sources identified for synthesis to identify biases and gaps and utilizes these to inform systemic and systematic inquiry. Critically analyzes sources identified for synthesis to identify biases or gaps and utilizes these to inform systemic and systematic inquiry. Conducts surface analysis of sources identified for synthesis to identify biases or gaps and utilizes these to inform systemic and systematic inquiry. Conducts surface analysis of sources identified for synthesis that does not identify biases or gaps and/or utilizes these to minimally inform systemic and systematic inquiry.  
PLO #4 Communicate effectively using multiple modes.  
Indicators Level 4 Level 3 Level 2 Level 1 80% of students will meet or exceed Level 3 competency
Effective Communication Consistently and successfully utilizes strategies for effective communication, including awareness of the intended audience, purpose of the communication, and clear organization and support to enhance the message. Frequently and successfully utilizes strategies for effective communication, including awareness of the intended audience, purpose of the communication, and clear organization and support to enhance the message. Inconsistently utilizes strategies for effective communication, including awareness of the intended audience, purpose of the communication, and clear organization and support. Rarely utilizes strategies for effective communication, including awareness of the intended audience, purpose of the communication, and clear organization and support.  
Communication in Multiple Modes Consistently and successfully demonstrates the ability to communicate effectively in multiple modes, including a variety of written/visual products, oral presentations, and interpersonal communications. Demonstrates the ability to communicate effectively in multiple modes, including a variety of written/visual products, oral presentations, and interpersonal communications. Demonstrates basic ability to communicate effectively in multiple modes, including a variety of written/visual products, oral presentations, and interpersonal communications. Does not demonstrate the ability to communicate effectively in multiple modes, including a variety of written/visual products, oral presentations, and interpersonal communications.  
PLO #5 Conduct scholarly and professional activities in an ethical manner.  
Indicators Level 4 Level 3 Level 2 Level 1 80% of students will meet or exceed Level 3 competency
Ethical Scholarship Adhere to guidelines for ethical scholarship, including those for the protection of human subjects, as well as principles of Diversity, Equity, and Inclusion within all scholarly activities. Adhere to guidelines for ethical scholarship, including those for the protection of human subjects within all scholarly activities. N/A N/A  
Professionalism Consistently and successfully adhere to guidelines for professional conduct, including respectful interactions with others, collegiality, and collaboration. Frequently and successfully adhere to guidelines for professional conduct, including respectful interactions with others, collegiality, and collaboration. Inconsistently adhere to guidelines for professional conduct, including respectful interactions with others, collegiality, and collaboration. Infrequently adhere to guidelines for professional conduct, including respectful interactions with others, collegiality, and collaboration.  

 

Part 5: Program Assessment Plan:

1) How will annual assessment be communicated to faculty within the department? How will faculty participating in the collecting of assessment data (student work/artifacts) be notified?

Faculty, staff, and graduate students in the program meet bi-weekly to discuss programmatic matters. The assessment reports will be communicated in a Fall department meeting each year. This will be communicated ahead of time through the weekly Monday Minutes communication that is sent out to the department community.

2) When will the data be collected and reviewed, and by whom?

Data will be collected each semester and reviewed on an annual basis by program faculty and the Assessment & Alignment Committee within the department. This committee consists of faculty and staff within the department.

3) Who will be responsible for the writing of the report?

The report will be written by program leadership with input and feedback from the faculty.

4) How, when, and by whom, will the report be shared?

The report will be shared by program leadership with the faculty at a fall department meeting. It will be sent to faculty ahead of the meeting in order to provide time for them to preview and prepare to provide feedback at the faculty meeting.

5) How will past assessments be used to inform changes and improvements? (How will Closing the Loop be documented)?

As part of each cycle’s assessment, we will set goals for continuous program improvement. These goals will be revisited throughout the cycle to determine what progress is being made toward the goals and what additional adjustments need to be made to continue progress.

6) Other Comments:

 

Submit report to programassessment@montana.edu