David Van Story

some insights on student motivation from building edtech apps: > having shorter lessons with a high number of quick tasks is more motivating than longer lessons with less tasks > if a student has a problem with the app that you fix, they love being told about the fix and being thanked for raising it. > a student is far more motivated by a daily goal than a weekly goal. it’s important to have longer term goals, like a weekly. but daily goals drive much higher engagement. > I was surprised to learn how motivated students are by “pets”. > having an XP system drove daily engagement way up. students “see their progression” with rings that fill up daily and they really like it. > students like when we link their daily scores to how they will perform on the exam (e.g. predicted AP scores) > they love quick and personal responses to bug reports they submit > for most students having a parent, teacher, or guide in the loop is a key part of a student staying consistently engaged/motivated over long periods of time > if AI grading feels unfair or makes students redo work they feel they have already mastered it is very demotivating for students

a not-so-obvious playbook on how to field test curriculum: Curriculum should be tested in staggered cohorts. It’s the most efficient and reliable way. 1. Cohort A tests a chunk of the curriculum first (20 lessons). Their lesson performance is tracked. Fixes are made to all the parts of the curriculum that students perform poorly on. 2. The fixes are propagated forwards and backwards in the curriculum. Cohort A continues on. 3. Cohort B tests the same chunk of curriculum, that was just changed. The goal is to test if the fixes worked. Performance is tracked. The goal is that student performance increases compared to Cohort A. Again low performing lessons are tracked and fixes are made. 4. The fixes are propagated forwards and backwards. Cohort B continues on. 5. Repeat for as many cohorts until performance on lessons meets the desired performance standard.

notes on why I believe digital curriculum is the future: > curriculum is created instantly. no time is wasted on delivering materials, printing, distribution of content, or teachers having to become familiar with the material. > the content is delivery exactly as designed, every single time. there is no variability on how the instruction is delivered. variation of how the content is taught to students is the biggest source of error when testing curriculum. > you can discover what works and what doesn’t on large scales because every student is receiving identical instruction. > collecting field data is effortless because every students click, wrong answer, right answer, and dwell is tracked in a single database. >(getting traditional pen/paper field data is a mix of audio and video recording teachers and students in classrooms, and notes or student work on paper - it is easy to see how this data would be enormously hard and slow to process) > when new insights are revealed based on student’s data, updating the curriculum, or A/B testing different theories can happen instantly and on large scales. > it is easy to keep a record of what has changed over time, making future iterations easier and easier. non-digital curriculum is very hard to track changes over time. > the biggest limitation with digital curriculum is that it is more difficult to control and observe environmental factors (noise, distractions, technical issues) which can be the source of disruptions and performance issues. > teachers, guides, parents are still vital for pacing, motivation, accountability, and setting performance expectations.

You've heard of autonomous driving. We're building autonomous curriculum.

The story behind my favorite painting: When I first moved to San Francisco many years ago I had very little money. So I rented a tiny cheap room. Tiny. The type of room that would end up on some internet “list” of rooms that are probably not legally rentable. But I was young, broke, and getting stuck into a new city and so moved in. Anyway I told my mother about the room, and mentioned that the biggest nuisance was that it had no windows (see “list”). She lives in the countryside in Ireland and was a little horrified by the whole situation. A few weeks later a package arrived for me. I opened it up and it was a painting. I was confused, this was not something I was used to receiving. It turns out my mother had spoken about my room situation to her neighbour in Ireland, who is a painter. My mother had commissioned her to paint the view from our home’s living room window - so that my windowless room would have a nice view.

if you want to design en excellent curriculum, it isn’t going to be excellent on the first try. At the Direct Instruction conference this year, this world expert got on stage and shared what parts of their English curriculum needed to be changed. The part that blew my mind was that this curriculum was like 10 yrs old. This world expert was still finding meaningful ways to improve it. And it turned out, they give this talk every year. Every year they share what they will change in the curriculum to make it better. It was so unexpected for me.. I’d been building with the mindset of ’good-curriculum-is-right-on-first-try’, and this world expert was still finding ways to improve their curriculum after years in the field. the real way excellent curriculum becomes excellent is by field testing it with students and seeing what doesn't work. making changes is inevitable, and necessary.

some things I've learned about how to build edtech apps: > the goal is to find the lean-but-complete way to teach the subject > work backwards from the test, whatever that may be. you need to know where you want the student to go in order to build the right path there > e.g if you are preparing a student for a standardized test (AP) look at past exam papers to understand the skills being tested. write down the big skills and question types that students need to be firm on to succeed on the test > break those big skills and questions down into very small sub skills. this is called atomization. the goal is to ultimately just teach one small skill, rule, fact at a time (atoms) and then teach these atoms in the right sequence to master the big skills > the goal is to find the sweetspot of not breaking things down too much or too little > break it down too much makes the curriculum slower and boring because its too easy, and too little makes it slower and less motivating because it’s too hard > figure out the right order to teach these skills and sub skills. there will be an order in which to teach the skills and sub skills so that students can efficiently learn things > the order matters more than you might think. if you sequence things right students will master later material faster than if you sequence them wrong > teach the skills and sub skills in lessons that last 3-12 minutes > set up the lessons so that students have to perform a task 2-5 times per minute. keep text short. long passages of text with little calls for student response is death > use video and images along with text > multiple choice is not a very effective question type. it is good, but should be used not exclusively becuase students done have to generate anything to respond they can always just click and guess > field test with students > track student success rates. you are aiming for ~85% first time correct on new material. 95% correct on review material > fix the lessons and questions that have high failure rates. fix the sequences that you discover should be in a different order. fix the lessons that teach too many things at once. combine the lessons that should be combined > the goal is to find the lean-but-complete way to teach the subject this is the method we use at crushap.com to teach Physics

this wasn't obvious to me at first, but is increasingly clear: when designing a learning app, every single lesson should feel easy-to-medium difficulty. if a lesson feels high difficulty it means that you're teaching too much at once.

great insights from @ganz on what effects his son's motivation when learning on apps at home to parents, what have you seen motivates/demotivates your kid when learning on apps?

Well deserved @MrZachG! I attended the @NIFDI conference this year and it was excellent - for anyone building in edtech strongly consider going.

This is a really important point for folks building edtech and learning applications. > aim for 3-5 student responses per minute in a classroom. with digital learning platforms, where everything is individual, it can be even higher.