This App Goes to Eleven

What building an app taught me about Socratic Method.

Photo by Its me Pravin on Unsplash

I’ve read two excellent articles this week about the release of ChatGPT’s Study Mode, one here by Stephen Fitzpatrick and one here by Louise Vigeant, Ph.D. Both approach it with caution and healthy scepticism and I think it is fair to say, they both have misgivings, although for different reasons. Do read them.

Apologies, by the way, if you have also written an excellent article about Study Mode this week and I haven’t got to it yet. Do send me a nudge in the comments.

Both articles discuss Socratic dialogue, a term that I sense has become much more common since the advent of generative AI - indeed, it’s right there on the front page of OpenAI’s release announcement.

I was introduced to the concept at university when, as an undergraduate, we were expected to complete some assignments in that form. They were exciting to write - but really difficult!

I’ve been interested in the idea ever since and have tried a number of ways to use it in the classroom over the years. On occasions, the results have been spectacular; one student recently used the format in her prize-winning entry to the John Locke essay competition and her writing was beautifully elegant and clever.

Much more often, though? Something of a damp squib.

For a start, it requires a highly skilled teacher who not only has a deep knowledge of their subject but is also able to frame suitable questions, on the fly, designed to guide the students, who also need to be pretty knowledgeable, towards a conclusion that is both useful and purposeful. Add to that the need to maintain discipline in a class of 30, to challenge the most able or engaged, to include as many students as possible, while showing kindness to those students who are terrified by the prospect of voicing their hesitant opinions in public and you start to get a sense why it’s tempting to just … make a Powerpoint.

So, it’s exciting to think that GenAI might open up new ways of making it possible.

I do think there is a bit of ‘definition slip’ to put up with; the term often seems to be used, somewhat loosely, to mean just open-ended questioning. It also seems to have become a rather glib shorthand for academic validity and rigour. ‘This tool must be good; it uses the Socratic method’. Which I imagine is spoken in the same uncomprehending tone as Nigel Tufnel in ‘Spinal Tap’ explains his amp goes up to eleven. Study Mode is ‘one louder, isn’t it’.

That aside, it is a useful term within the sphere of prompt engineering. It’s something the LLMs seem to understand.

I mentioned a couple of weeks ago that I have been experimenting with building apps on the Playlab platform. In particular, I’ve been working on an app that will guide GCSE Drama students through the questions they might face on the AQA written exam. It’s quite a tricky paper because there is scope for a huge range of interpretation and students need to express their nuanced conceptual understanding through highly detailed practical descriptions. It takes lots of practice and lots of feedback to get good at it. Hence the app.

My first attempts were rather disappointing. Not because the app didn’t help the students but because it helped too much. It jumped far too easily to an answer the student could merely accept. They might as well have just asked ChatGPT to do it for them.

Here I need to give thanks to Eric Lars Martinsen who wrote this post about the way he had created a ‘Quoteweaver’ app. Because he generously shared a link to the app, I was able to look ‘under the bonnet’ and see how he had constructed his prompts to prevent the bot from obeying its training and just giving an answer. I learnt a huge amount from that process but the thing I found made the most dramatic difference was the simple instruction at the start: ‘Use the Socratic method - ask questions that guide students to discover insights rather than providing answers directly.’

That wasn’t enough in itself - I also uploaded reference documents with suggested essay structures, model answers, mark schemes, examiners’ reports - but it was the prompt that seemed to move everything in the right direction.

I’ve been trialling it with a number of students and the signs so far have been really promising. They and I have noticed how much more they write with the app than they did before and, in particular, how much more detailed their descriptions are becoming. I’ve also programmed the app to ask the students to reflect on their performance at the end and it’s striking how much more deliberately they structure their answers now.

If you’re interested, you can access the app here. I’d love to hear any thoughts you might have. You could even try to ‘red team’ it and let me know how easy it is to break!

I don’t claim it’s perfect but it’s good enough to convince me that writing apps with quite a narrow focus is going to be a good route forward. It’s one of the methods I plan to use to slow down my students, to make their learning more effortful but also more purposeful and also to place more explicit emphasis on their metacognitive skills.

Writing the app has been great fun but it has also served as a reminder that, when you have precise learning outcomes in mind, it takes a lot of time to mould an LLM to those ends.

Turns out, Socratic dialogue is not only really difficult to write as an undergraduate. It’s still difficult!

And I’m not sure that just selecting the ‘Study Mode’ option from a drop down menu is going to achieve what I want.