The ecta center The DaSy Center

SSIP Child Outcomes Broad Data Analysis Template

Developed in collaboration with the Regional Resource Center Program (RRCP) as part of the Part C and 619 State Accountability Systems Priority Area

As part of the State Systemic Improvement Plan (SSIP), a broad data analysis of child outcomes data may be a useful first step in understanding your state’s performance. Before diving into the broad data analysis it will be important to look at the quality of the child outcomes data in your state.¹

Data quality issues will be an important consideration as you move into the interpretation of your data analysis and will likely determine what questions you can reliability ask of your data. For more information about examining the quality of your state data go to the quality assurance section of the ECTA website: http://ectacenter.org/eco/pages/quality_assurance.asp.

The purpose of broad data analysis is to look at how children in the state are performing relative to national data, across years, within the state and by comparisons across programs within the state. This template has been developed to assist in conducting an initial analysis with data you currently use for reporting in the APR. An example of one state’s data is used below to illustrate how the template can be used.

The first step of the broad child outcomes analysis is to look at the comparison between the state and national data. See Figure 1 for a comparison between state and national data in the percent of children that substantially increased their rate of growth (Summary Statement 1) and Figure 2 for a comparison in the percent of children that exited within age expectations (Summary Statement 2). Looking at Figures 1 and 2 you can see that the largest difference between the state and national percentages are in children’s knowledge and skills. The state percentage is 7 points above the national data for Summary Statement 1- substantially increased rate of growth and the state percentage is 9 points below the national data for Summary Statement 2 - exiting at age expectations.

Conclusion: Compared to the national data, children in this state make significant growth but are less likely to exit functioning at age expectations. The reason that fewer children in the state exit at age expectations cannot be determined from this limited comparison to national data. Further analysis is needed to identify the root cause.

To create graphs comparing your state data to the national data click on the following link: http://www.ectacenter.org/~xls/eco/NationalchilddataGraphtemplate_2011-12.xlsx

Graphs comparing your state data to the national data are also available upon request by emailing abby.winer@sri.com.





Step 2: Analysis of Trends in State Performance

The next step in the broad data analysis is to look at trends in Summary Statement 1 and 2 across time within the state. In this state example, for Summary Statement 1, the trends look relatively stable across time. For Summary Statement 2, the trends for Outcome 1 and 3 look stable but the trend for Outcome 2 Knowledge and Skills is declining from about 60% in 2008-09 to 40% in 2011-12.

Conclusion: This analysis does not provide enough evidence for us to make inferences or interpretations about the downward trend in the percent of children exiting at age expectations in knowledge and skills but it does provide a direction in which to do more in-depth/root cause analysis.

To create these graphs for your state, download an excel longitudinal graphing template here: http://ectacenter.org/~xls/eco/longitudinalgraphingtemplate.xlsx

The final step in the broad data analysis compares your local programs to each other and to the state. The purpose of this analysis is to identify high and low performing programs. In general, program improvement activities focus on low performing programs. However, when you see programs that are performing much higher than other programs in the state you will want to confirm that the programs do not have data quality issues (e.g. inflated exit scores/ratings). An important consideration in any analysis comparing local programs is the number of children included in the summary statement for the program. If there are fewer than 35 children included in the summary statement you would expect large variation from year to year in the summary statement percentages (10 – 15 percentage points). This means that a program’s status relative to other programs is not very stable from year to year and not a reliable way to understand the program’s performance relative to other programs (e.g. they could be low one year and average the next without any changes to their programs). One option for graphing local data to make the interpretation more reliable is to include only programs that have 35 or more children. In addition, you will want to make sure that you include the number of children in the summary statement in the program label (see Figure 5). This will allow you to consider the stability of the summary statement as you interpret the data across programs.

In this example, we are only showing data for Outcome 2 - Knowledge and Skills, Summary Statement 2 – exits at age expectations. In your state, you may need to do the analysis by local programs for all outcomes and summary statements or you may be able to limit your analysis based on the earlier results. Some examples of cases where a state would reduce the set of summary statements and outcomes analyzed include:

• 
  The previous analysis leads to a more specific question.
  
• 
  Stakeholders chose to focus on a reduced set of summary statements and outcomes.
  
• 
  You are focusing on a particular activity or initiative in the state that is related more to one summary statement and outcome than others. For example, you are focusing on an early math initiative so you only look at Outcome 2.
  

Conclusions: The current analysis does not provide enough information for us to make inferences about why the cluster of 5 low performing programs are not performing at the same level as other programs. To make these inferences we would need to look at the characteristics of the children and of the services in the cluster of 5 low performing programs and how these compare to other programs in the state.

For more information on how to create a chart like this, view this 5 minute video: http://www.screenr.com/xJIH

After you review and present the data from your broad data analysis it will be important to add some interpretation and decide on the next steps for more indepth analysis and determination of root causes related to the performance. Involving stakeholders is recommended to broaden the perspective as the analyses continue . The table below includes some questions that you could ask to help guide interpretation and next steps.

• 
  Does our state’s child outcomes data look like that of other states with similar eligibility criteria and/or service delivery systems?
  
• 
  For a given outcome of interest, how does the breakdown by progress categories a through e inform our interpretation and possible next steps?
  

For more information on possible questions for more in depth quantitative analysis see the Analyzing Child Outcomes Guidance Table: http://ectacenter.org/~pdfs/eco/AnalyzingChildOutcomesData-GuidanceTable.pdf