How the FTCE and FELE Were Developed

Committees are established for each subject area to develop or review test materials for each stage of the development process. The committees are typically composed of Florida public school teachers, district coordinators, and college and university faculty. Committee members are selected to represent various institutions and educational philosophies and to reflect the diversity of the Florida population.

Competencies and Skills Development

For each subject area undergoing development, a committee of Florida educators is established to review existing competencies and skills and to revise, delete, or add competencies and skills to reflect current job-related practice, accepted teaching theory, Florida Statutes, and State Board of Education-approved educational standards. The competencies and skills are targeted to describe the core knowledge of subject matter knowledge and instructional strategies teacher candidates should have to meet effective teacher proficiency expectations for beginning teachers. The committee also reviews and adjusts the proportions of the test used to measure each competency as needed, resulting in the test blueprint.

Competencies and Skills Validation

Another committee of Florida educators validates the revised competencies and skills and test blueprint according to specified review criteria. A statewide content validation survey is also conducted to gather data regarding the degree to which each skill statement is relevant to the job of an entry-level educator in the field and to gather input on the appropriateness of the test blueprint weighting.

Competencies and Skills Finalization

A third committee of Florida educators finalizes the revised competencies and skills and test blueprint according to specified review criteria. Content validation survey data are presented to the committee and used during the finalization process. The competencies and skills are then presented to the State Board of Education for approval and implementation.

Item specifications guide the selection and development of test items. Item specifications are updated by a committee of Florida educators after the finalization of the competencies and skills. The item specifications identify the competency and skill for which the specification is developed, possible item formats and content to be tested, and an exemplary test item. A committee of Florida educators is also convened to finalize the item specifications according to specified review criteria.

New test items are written according to the item specifications. These test items are either field tested or pilot tested. Field testing involves adding newly developed items to current test forms administered operationally. Pilot testing involves assembling the newly developed items into separate forms and inviting certification candidates to volunteer to take the pilot tests. Following pilot and field testing, a committee of Florida educators reviews and validates each item according to specified review criteria. Pilot and/or field test data are provided to guide the committee in the task.

Test forms are assembled in accordance with the test blueprint and other specified measurement guidelines and criteria. A committee of Florida educators reviews and validates the forms according to specified review criteria. Committee members take each test, and their results (as well as examinee data) are provided to guide them in the validation task.

Before and after the first administration of each test form, a subject matter expert reviews each test item. This process serves as a further check to ensure the validity of the examination forms and the accuracy of the answer key used to score the test, before scores are reported.

Updated test information guides (TIGs) are developed for each subject area undergoing development. The guides include the competencies and skills, test blueprint, sample test items linked to specific competencies, and an annotated bibliography of Florida textbooks or other preparation resources.

For the FTCE and FELE examinations, a modified Angoff method (Angoff, 1971) is used to set passing scores. In this method, subject matter experts review each item and mark the proportion of the target population (i.e., beginning effective teacher candidate ) that would provide a correct response. After the experts complete their ratings, the data are analyzed to produce a preliminary passing score.

The standard setting panels that participate in standard setting activities consist of district personnel, public school teachers licensed and practicing in the field, and college faculty from Florida colleges and universities who are currently preparing (or have prepared) prospective teachers in the field. Selected panel members include educators from public schools and colleges who represent districts and colleges across the state and generally reflect the diversity of the state of Florida.

Performance-level descriptors outline the skills and knowledge that examinees in a performance category should possess. Performance-level descriptors form the basis for making judgments and providing ratings in standard setting. For each test field, a single description of the beginning effective teacher candidate  is provided because only one passing score is determined.

Training allows panelists to receive information about the testing program, such as test development procedures and scoring, and the task that panelists are required to complete. The panel members are given an orientation that explains the passing score recommendation process, the materials they will use, the concept of the beginning effective teacher candidate , and the judgments about test items that they will be asked to make. Panelists also complete an item rating training exercise, using items from the field's test information guide, to prepare them for the actual rating activity. The Department and the State Board of Education's role in establishing the final passing score is also explained.

Training also involves a simulated test taking activity by the panelists, which allows them to become familiar with the assessment content as well as the testing environment that candidates would experience. Panelists are asked to review the competencies and skills for their field, and then answer the test questions. Each panelist is provided with a copy of the test form for their field, and is asked to read and answer the questions on the test without referring to the answer key.

An iterative procedure is used in which standard setting ratings are gathered in three rounds. In the first two rounds, the panelists provide performance-level judgments for each item on the test form for their field using a modified Angoff procedure. Referring to the test form they had reviewed and taken earlier, panelists are asked to provide their individual, independent judgments regarding the expected performance of the beginning effective teacher candidate  on each of the test items. Panelists are asked to envision a group of Florida educators or school administrators who are just at the level of subject matter knowledge required to be beginning effective teacher candidates in Florida public schools. In the third round, panelists are asked to consider the feedback provided; the purpose of the FTCE/FELE testing program, test blueprint, and the test items; and the requirements for the beginning effective teacher candidate. Panelists will also provide their individual recommendations for a test-level minimum passing score.

Giving feedback to panelists helps to improve consistency of ratings and reduces variability among panelists as the panelists develop a clearer and common understanding of the expected performance. At the beginning of the second round, panelists review the results from the first round of ratings as well as candidate performance on the items. Ratings from the first round are analyzed to produce a panelist-specific item rating summary. Similar calculations are made after the round two ratings. In the final round, panelists review the results from the second round of ratings and a statistical summary showing the percentage of examinees taking the test in the initial test administrations who would pass the test at each possible passing score value. Panelists are provided an opportunity to ask for further clarification on how to read and interpret the analyses.

After the panelists complete their ratings, the data are analyzed to produce a passing score. The test-level ratings provided in round three are used to compute a median score of the standard setting panel, which is then presented to the State Board of Education for approval and implementation.