Virtual Comparison Groups

NWEA researchers have developed a research control group methodology called Virtual Comparison Groups, or VCGs. Using VCGs, researchers can compare a student’s academic growth (based on the difference between two MAP scores) to students who perform similarly and share similar demographic characteristics. Unlike comparisons using norms, VCG reports provide apples-to-apples comparisons.

What is a Virtual Comparison Group?

A virtual comparison group is a group of students that matches key characteristics of the group of students being compared. Researchers refer to it as a matched group quasi-experimental design. The comparison group is “virtual” because it is comprised of data from NWEA’s longitudinal student achievement database that includes growth data from millions of students. In addition to student growth data, the database holds relevant district, school, and student characteristics, such as socio-economic status (SES) data, urban/rural designations, grade level, and beginning performance level.

To construct a VCG, NWEA identifies the key characteristics of each student in the study group. The study group may be a class, school, or other logical organization of students. Next, for each student, NWEA filters the database to find students who perform similarly and share these key characteristics. A random sample of 51 comparison students is selected from that result and these become the Virtual Comparison Groups for each student in the initial study group. VCGs can provide valuable information for making program decisions or evaluating the effect of an educational intervention or policy. The VCG results control for variables that may otherwise mask the impact of the intervention.

How are VCGs created?

NWEA uses three types of filters to create a VCG for each student being studied.

1. General Assessment Filters – Only students with valid test scores for the same year and subject area are considered.
2. School Filters – The percentage of students qualified for free and reduced lunch is within 5 percentage points of the school being studied and schools have the same urban/rural classification (based on the National Center for Educational Statistics Common Core of Data Survey).
3. Student Filters – Students are in the same grade level, within 1 RIT point of the beginning score, and the beginning and ending tests were taken within 7 days of the student’s test date. This assures that the student and members of the VCG group had about the same amount of instructional time.

After applying these three types of filters, a group of 51 students is randomly selected from the database. The median score of this group is the metric that is used to compare to the study group student score.

If you are interested in learning more about how to use VCG data for your research study design, please contact us.