Novel program research: standards and background

PEER staff
7/3/2018

Overview

This is a primer on research standards.

It has three parts.

  • Intuitive (but insufficient) evaluation
  • Standards of research quality
  • Implementation of quality research

Part one

Intuitive (but insufficient) evaluation:

  • Anecdote
  • Simple numerical comparison

Part two

Standards of research quality:

  • The statutory definition of 'evidence'
  • The Maryland Scientific Methods scale

Part three

Implementation of quality research:

  • Pre-registration
  • Reproducibility

"Evidence"

“Evidence-based practice” is kind of like “the right outcome” of a trial.

Nobody's against it – but people have very different opinions of what it is!

But they're alike in another way: Not all opinions are created equal.

Anecdotes vs. evidence

“Our program definitely works. Just look at Timmy! He went through it, and turned his whole life around!”

This is not, by itself, evidence of a program's effectiveness.

Why?

Here are Timmy and a couple of his friends.

plot of chunk unnamed-chunk-1

But here's the rest of Timmy's program.

plot of chunk unnamed-chunk-2

The point

With enough participants and normal conditions, you're effectively guaranteed to have successes even in a bad program.

In other words:

Anecdotes aren't good evidence!

What if we get more numbers?

Imagine two programs doing the same thing.

  • One of them achieves 40 units of effect on average.
  • The other achieves 80 units of effect on average.

We normally believe that the 80-unit program is better.

Here's what we probably imagine...

plot of chunk unnamed-chunk-3

But here's what could be happening!