CHAPTER III

METHODOLOGY

Introduction

This chapter describes the methods and procedures used to conduct this study. It includes the purpose of the study, the research questions, the research design, a description of the population and sample, the development and descriptions of the instrument, field-testing, a description of the data collection procedures, explanation of the statistical analysis, and the limitations of the study.

Purpose Statement

The purpose of this study was to investigate and compare supervisors' assessments of traditionally certified site administrators and administrators certified via examination on leadership performance as measured by the ISLLC Standards for School Leaders. The six Standards for School Leaders address shared vision, school culture, management, collaboration, professional ethics, and community advocating.

Research Questions

Answers to the following questions were sought to address the purpose of the study:

What percentage of traditionally certified site administrators demonstrates each of the six standards for school leaders at the "highly effective," "effective," "ineffective," and "highly ineffective" levels, as assessed by their supervisors?

What percentage of site administrators certified via examination demonstrates each of the six standards for school leaders at the "highly effective," "effective," "ineffective," and "highly ineffective" levels, as assessed by their supervisors?

Is there a significant difference in the assessment of performance by the supervisors of traditionally certified site administrators and administrators certified via examination as measured by the ISLLC Standards for School Leaders?

Research Methodology

This study was a descriptive research study as described by Isaac and Michael (1995) in their Handbook in Research and Evaluation. The descriptive design is an appropriate type of design for this study as its purpose is "to observe systematically the facts and characteristics of a given population or area of interest, factually and accurately" (p. 50). Descriptive research is useful for investigating a variety of educational problems. Descriptive research data are usually collected through means of a survey, interviews, or observation (Gay, 1981). This study utilized a survey to gather the information needed to compare traditionally certified school site administrators and administrators certified via examination. Creswell (2005) describes survey research as "quantitative research in which investigators administer a survey to a sample or to the entire population of people in order to describe the attitudes, opinions, behaviors, or characteristics of the population" (p. 354). Further, survey research is an economical means of gathering data quickly. The survey is cross-sectional since the data were collected at one point of time and the instrument was self-administered.

According to Babbie (1990), survey designs are useful because certain types can be used not only to describe the characteristics of a population derived from data obtained from a sample, but they can be used to determine relationships between variables at the time of the study. In the present study, the differences noted between the supervisors' assessment of performance on the ISLLC Standards for School Leaders of traditionally certified site administrators and administrators certified via examination can be examined to determine if they differ in a statistically significant way.

Based on the advantages of this type of research strategy, a self-report survey of site administrators was the method chosen for identifying the study population and their demographic and personal characteristics (Appendix D), followed by a survey of their supervisors to determine their assessed performance on each of the six ISLLC Standards for School Leaders (Appendix E).

Population and Sample

The initial population for this study was school site administrators employed in one of the 23 school districts located in Riverside County, California. Within Riverside County, 465 schools are operated including elementary, middle/junior high, high, alternative ed/adult ed, and charter schools (Table 8) (Riverside County Office of Education [RCOE], 2008). According to the CDE, in 2006-2007, 1,514 administrators provide service in Riverside County. However, there is no distinction made between those administrators who work at a school site and those who serve as district-level administrators or work for the county office of education.

Table 8

Public School Sites in Riverside County 2007-2008

School sites	Total
Charter schools	11
Elementary sites	265
Middle/Junior high sites	74
Continuation/adult ed	50
High school sites	65
Total K-12 sites	465

The study sample was selected to make comparisons about site administrators certified traditionally and those certified via examination in Riverside County. Data from site administrators traditionally certified and those certified via examination were collected using a cross-sectional survey to measure what percentage demonstrated each of the six standards for school leaders at the "highly effective," "effective," "ineffective," and "highly ineffective" levels as assessed by their supervisors.

A list of superintendents was obtained from the RCOE (2008) website. On May 6, 2008, superintendents were contacted for each of the 23 school districts in Riverside County in order to secure approval for participation and obtain a list of site administrators (Appendix F). Superintendents were also requested to encourage site administrators to participate in the research study. Initially, 17 of 23 districts agreed to participate in the study. After follow up, 12 school districts including Alvord Unified, Coachella Valley Unified, Desert Sands Unified, Hemet Unified, Menifee Union, Moreno Valley Unified, Nuview Union, Palo Verde Unified, Perris Elementary, Perris Union High, San Jacinto Unified, and Val Verde Unified provided the requested information resulting in 404 identified site administrators.

The sample was identified through the use of a demographic survey (Appendix D) to the 404 site administrators within Riverside County who were previously identified by their district of employment. The survey was created to identify all site administrators who secured employment in 2003 or later, their certification route (traditional graduate program or via examination), and demographic and personal characteristics, such as gender, ethnicity, years in education, years in administration, and current position and level served.

Only site administrators who secured their first administrative position in 2003 or later were included in the study for the following reason; the SLLA was first administered in the state of California on January 11, 2003. During this administration, 61 examinees took the test and therefore created the first potential pool of site administrators certified via examination (CCTC, 2003b). In the year 2003-2004, the first full year the test was offered, 679 examinees in California took the test. In 2004-2005, 962 examinees took the exam. Passing rates were 78% and 83%, respectively (CCTC, 2006). This created a potential pool of 1,328 potential administrator candidates who became eligible for certification via examination (CCTC, 2006). As of 2006, the CCTC (2008) reported that 1,272 individuals applied for a Certificate of Eligibility or a Preliminary Administrative Services Credential via examination. However, it is unknown how many examinees certified via examination have secured a site administrative position. Further, school districts typically do not maintain records that distinguish between traditionally certified site administrators and those certified via examination, making tracking of these individuals difficult.

Of the 404 surveys initially sent, 161 site administrators responded, resulting in a 40% response rate. Of the 161 respondents who participated in the initial survey, 72 identified themselves as having earned their first administrative credential in 2003 or later. Of the 72 responses, 61 were from self-reported traditionally prepared site administrators and 11 were from site administrators who were certified via examination. However, 10 traditionally certified site administrators did not provide the name and/or proper contact information for their direct supervisor and 7 reported greater than 6 years of school administration experience. Greater than 6 years of administrative experience is inconsistent with administrators' earning their first administrative credential in 2003 or later and therefore these 17 site administrators were excluded. The final number of useable survey responses was reduced to 55, 11 certified via examination and 44 traditionally certified.

A random sample of 26 traditionally trained school site administrators was chosen from the 44 valid survey responses. Additionally, all 11 site administrators certified via examination were chosen for participation due to the small number observed during the initial survey of site administrators found within 12 districts in Riverside County. Once the site administrators were chosen for participation in the study, each of their direct supervisors was identified and invited to participate by providing their assessment of the site administrators' performance on each of the six ISLLC Standards for School Leaders.

Instrumentation

The survey instrument was designed specifically for this study based on the construct of the ISLLC Standards for School Leaders. The survey was designed to gather data to determine what percentage of site administrators certified traditionally and those certified via examination demonstrate each of the six standards for school leaders at the "highly effective," "effective," "ineffective," and "highly ineffective" levels, as assessed by their supervisors and to make comparisons among both groups. The survey was conducted via SurveyMonkey, a secure online survey provider. The instrument was divided into six sections that correspond to one of the six standards. Within each of the standards, five performance indicators were chosen for inclusion in the survey. In addition to defining the six standards, grouping in this manner also prevented the respondents from having to shift their thoughts from standard to standard, and organized the survey into shorter and more manageable segments. Effectiveness was measured using a 4-point Likert scale, where 4 represented "highly effective," 3 represented "effective," 2 represented "ineffective" and 1 represented "highly ineffective." Likert-type survey items are frequently used to investigate the strength of respondents' feelings on a categorical scale (Dittrich, Francis, Hatzinger, & Katzenbeisser, 2007). The survey is included in Appendix E.

The six standards for school leaders to be examined using this particular instrument and the corresponding performance indicators were:

Standard 1— Shared Vision as described by performance indicators: (1.1) the vision is developed with and among stakeholders; (1.2) assessment data related to student learning are used to develop the school vision and goals; (1.3) the core beliefs of the school vision are modeled for all stakeholders; (1.4) barriers to achieving the vision are identified, clarified, and addressed; and (1.5) the vision, mission, and implementation plans are regularly monitored, evaluated, and revised.

Standard 2—School Culture as described by performance indicators: (2.1) all individuals are treated with fairness, dignity, and respect; (2.2) professional development promotes a focus on student learning consistent with the school vision and goals; (2.3) professional development promotes a focus on student learning consistent with the school vision and goals; (2.4) student and staff accomplishments are recognized and celebrated; and (2.5) a variety of sources of information is used to make decisions.

Standard 3—Management as described by performance indicators: (3.1) knowledge of learning, teaching, and student development is used to inform management decisions; (3.2) problems are confronted and resolved in a timely manner; (3.3) stakeholders are involved in decisions affecting schools; (3.4) responsibility is shared to maximize ownership and accountability; and (3.5) effective communication skills are used.

Standard 4—Collaboration as described by performance indicators: (4.1) high visibility, active involvement, and communication with the larger community is a priority; (4.2) available community resources are secured to help the school solve problems and achieve goals; (4.3) diversity is recognized and valued; (4.4) partnerships are established with area businesses, institutions of higher education, and community groups to strengthen programs and support school goals; and (4.5) opportunities for staff to develop collaborative skills are provided.

Standard 5—Professional Ethics as described by performance indicators: (5.1) demonstrates a personal and professional code ethics; (5.2) treats people fairly, equitably, and with dignity and respect; (5.3) Protects the rights and confidentiality of students and staff; (5.4) recognizes and respects the legitimate authority of others; and (5.5) demonstrates values, beliefs, and attitudes that inspire others to higher levels of performance.

Standard 6—Community Advocating as described by performance indicators: (6.1) the environment in which schools operate is influenced on behalf of students and their families; (6.2) communication occurs among the school community concerning trends, issues, and potential changes in the environment in which schools operate; (6.3) there is ongoing dialogue with representatives of diverse community groups; (6.4) the school community works within the framework of policies, laws, and regulations enacted by local, state, and federal authorities; and (6.5) lines of communication are developed with decision makers outside the school community.

A field-test of the instrument was given to a sample of supervisors of traditionally certified site administrators and a sample of supervisors of site administrators certified via examination to determine if the individuals in the sample are capable of completing the survey and answering the questions (Creswell, 2005). Participants were selected from districts located outside of Riverside County in order to ensure that they would not be study participants. After completion of the instrument, participants were given the opportunity for individualized input based on the following questions:

Is the questionnaire too long?

Are the directions and wording clear and unambiguous?

Is the format conducive to ease of response?

Do some of the items need to be rephrased or dropped?

Suggestions and modifications were made with regard to the items on the survey.

Validity was also established. Validity refers to the degree to which an instrument measures what it claims to measure, allowing meaningful and justifiable inferences (Creswell, 2005). To establish validity, a panel of experts reviewed the survey. The panel included an expert in leadership preparation from a local university, a traditionally trained school site administrator, and an administrator certified via examination. Suggestions and modifications were made based on their recommendations with regard to specific items on the survey.

Data Collection Procedures

Upon approval by the University of La Verne's Institutional Review Board (Appendix M), all superintendents within Riverside County were contacted personally or via e-mail in April 2008 with a request for a list of all site administrators and their e-mail addresses (Appendix F). An initial follow up e-mail was sent one week following the initial request and a second a week later. Twelve of 23 school districts provided e-mail addresses of their site administrators including Alvord Unified, Coachella Valley Unified, Desert Sands Unified, Hemet Unified, Menifee Union, Moreno Valley Unified, Nuview Union, Palo Verde Unified, Perris Elementary, Perris Union High, San Jacinto Unified, and Val Verde Unified. Once site administrators were identified, each participant received an e-mail that identified the researcher, explained the purpose of the study and informed consent, and provided a survey link to SurveyMonkey, an online, secure survey provider on May 13, 2008. The Site Administrator Informed Consent is found in Appendix G. Participants were informed that all responses related to or containing identifiable characteristics would be used for statistical purposes only and would not be disclosed or used in identifiable form for any other purposes. No individual data that linked the respondent's name, school, district, or address would be included in the statistical report.

Additional statements describing the need for the administrators' direct supervisor to answer the survey with regard to their performance on the ISLLC Standards for School Leaders were also indicated. A follow-up e-mail was sent seven days later to elicit additional responses. A week later, a third reminder e-mail was sent to increase the participation rate. Of the 404 surveys initially sent, 161 site administrators responded resulting in a forty percent (40%) response rate.

After identification of study participants and collection of demographic and personal characteristics, a second survey was sent to the supervisors of site administrators traditionally certified and those certified via examination who were credentialed in 2003 or later in order to assess the site administrator's performance on the ISLLC Standards for School Leaders. To assure participants of anonymity, no information in the survey instrument was specific to individual participants or their schools, nor was the survey designed in any way to track them.

Each supervisor received an e-mail that identified the researcher, explained the purpose of the study and informed consent, and provided a survey link to SurveyMonkey, an online, secure survey provider on June 2, 2008. The Supervisors' Informed Consent is found in Appendix G. Follow up e-mails were sent on June 6, 2008, and personal phone calls were made the week of June 16, 2008. SurveyMonkey results are reported in Appendix I.

Twenty-four responses were collected from traditionally trained site administrator supervisors resulting in a 92% response rate. Twenty-three (n = 23) responses were determined to be valid, as one respondent did not provide the proper tracking information. A 100% response rate was obtained from supervisors of site administrators certified via examination as all 11 (n=11) completed the second survey and gave valid responses

Data Analysis

Research questions 1 and 2 focused on what percentage of traditionally certified site administrators and administrators certified via examination demonstrate each of the six ISLLC Standards for School Leaders at the "highly effective," "effective," “ineffective,” and “highly ineffective” levels, as assessed by their supervisors. Likert scale responses serve as the basis for quantitative analysis for each of the two research questions.

The instrument used for this study was divided into six sections that correspond to one of the six standards. Within each of the standards, five performance indicators were listed for a total of 30 performance indicators. Site administrators were rated on their effectiveness on each performance indicator as measured by a four-point Likert scale, where 4 represented "highly effective," 3 represented "effective," 2 represented "ineffective," and 1 represented "highly ineffective." Frequency distributions were completed for each performance indicator by summing the responses of each item. Raw data collected for both traditionally trained site administrators and those certified via examination can be found in Appendixes H and I. The percentage of traditionally trained site administrators and site administrators certified via examination who demonstrate each of the standards for school leaders at the "highly effective," "effective," "ineffective," and "highly ineffective" levels was then calculated and tables generated. Responses from each of the two groups were analyzed separately using the same procedures.

Research question 3 focused on the determination of a statistically significant difference between the assessment of performance by supervisors of traditionally certified site administrators and site administrators certified via examination as measured by the six ISLLC Standards for School Leaders: (1) shared vision, (2) school culture, (3) management, (4) collaboration, (5) professional ethics, and (6) community advocating.

An independent t-test was used to determine whether a significant difference existed on each of the performance indicators representing each standard. There are two assumptions that must be met when conducting an independent t-test: (a) the samples have been selected from one or more homogeneous populations in which the population parameter is distributed normally (Goehring, 1981), and (b) variation of scores in the two groups must not be reliably different (Isaac & Michael, 1995). To ensure that the populations from which the samples are drawn actually are similar or homogeneous, an equal variance test was performed prior to conducting the t-test. The equal variance test determined the two groups to be equal. In order to determine if the data were normally distributed, tests for skewness and kurtosis were conducted to determine normality, which revealed the data were not normally distributed on 12 of 30 performance indicators. Because of the cases of nonnormal distribution, the Mann-Whitney U test was conducted to confirm the following independent t-test results. The Mann-Whitney U is considered a nonparametric equivalent of the independent t-test where the normal distribution is unclear (Goehring, 1981). Results from the Mann-Whitney U were identical to the independent t-test; therefore, the results were unaffected. The results from the Mann-Whitney U are found in Appendix L. All results are reported as independent t-tests and all decisions on the results were made using alpha level .05.

Delimitations

The following delimitations were accounted for in this study:

The study was restricted to site administrators in Riverside County.

The study was restricted to site administrators that currently serve as principals, assistant principals, or in other positions requiring an Administrative Services Credential whose primary role is to work at a school site.

The study population of administrators certified via examination was restricted to the participants self-identified as having earned their administrative certification by earning a passing score on the SLLA.

Limitations

In all research designs there are limitations. The limitations of this study are both the small sample size with respect to the overall population, which may limit the ability to generalize to other populations. This is in part due to the lack of statewide a tracking system which limited the researcher’s ability to identify the study population. Further, the specific standards and performance indicators within the survey limit this study. There may be other performance indicators that contribute to the effectiveness of site administrators. Additionally, this study was limited by the truthfulness and accuracy of the supervisors' assessment of site administrators' performance on the ISLLC Standards for School Leaders.