Activity: Teachable Machine
Time: about 30 minutes
Setup: on your own, or in a small group
Requirements: a web browser and a webcam (for image or pose models) or a microphone (for sound models)
URL: teachablemachine.withgoogle.com
This activity is about the most basic idea in machine learning: that a model learns from examples, and only from examples. Teachable Machine is a free Google tool that lets you train a small image, sound, or pose classifier in your browser, with no code. You decide on the categories, show it examples of each, and it trains a model to tell them apart.
The goal is not to build a useful model. It is to develop an intuition for what training data is, how your choices of categories and examples shape a model, and what training involves in practice. What you do here mirrors, on a tiny scale, what happens inside much larger models. You’ll also see, in miniature, the training cycle that Model Training describes next.
Before you start, you may want to read Basic Concepts, especially From Rules to Examples.
Teachable Machine is a third-party site and may change over time. Your images and sounds are processed in your browser and are not uploaded unless you choose to export your model to the cloud.
Transfer Learning
When you click Train Model, Teachable Machine doesn’t build a model from nothing. It starts from a model that has already been trained, by Google, on millions of images (or sounds, or poses), and has learned general visual features: edges, textures, shapes (Russakovsky et al. 2015; Howard et al. 2017; Zeiler and Fergus 2013). It then trains only the last part of that model, on your examples, to separate your classes. This is called transfer learning, and it’s why a few dozen examples are enough, where training from scratch would need thousands (Yosinski et al. 2014; Goodfellow et al. 2016, chap. 15). (The NER models in Unit 3 are pre-trained in the same sense.)
Training follows the cycle you’ll read about in Model Training, in short:
- It splits your examples. For each class, 15% of the samples (rounded up, according to its source code) are held back to test the model on examples it hasn’t seen, and the rest are used for training.
- It trains in epochs. Each epoch is one pass through all your training samples, adjusting the model’s weights a little each time.
- It shows you the result. The
Previewpanel runs the trained model live on your webcam or microphone, with a bar for each class showing how confident it is.
You don’t need to change any settings to do the activity. But if you open Advanced, under Train Model, you’ll find the settings that control training: the number of Epochs (50 by default), the Batch Size (how many samples the model looks at before each adjustment of its weights), and the Learning Rate (how big each adjustment is). Model Training explains the cycle of adjustments these settings control.
Under the hood, at the bottom of the same section, shows how well the model did on the samples it held back. For the model in the videos, it held back 5 of the 28 green-square samples and 4 of the 21 orange-circle samples, and got all 9 right: 100% accuracy on the test samples, reached in about one epoch.
That sounds perfect, but look at what those 9 samples were: taken in the same minute, on the same desk, in the same light as the training samples. A perfect score on them tells you the model learned to separate those pictures. It doesn’t tell you what the model will do in another room, which is what Part 2 is about. In the terms of Model Training, these test samples are really a development set: Teachable Machine scores them after every epoch, while it trains. Part 2 is the real test.
This is the Under the Hood panel for the model in the videos:
From the top: accuracy for each class on the test samples; a confusion matrix, showing which class the model predicted for each test sample; and accuracy and loss for each epoch, on the training samples (blue) and the test samples (orange). Click the image to enlarge it.
Part 1: Training a Model
The videos below show the whole process, with two classes: a green paper square and an orange paper circle. They have no sound. Watch them first, or refer back to them as you go.
Step 1: from the home page, Get Started, then Image Project → Standard image model.
Steps 2–3: rename the classes, then hold Hold to Record to capture webcam samples for each one.
Step 4: Train Model. (The progress bar is sped up.)
Step 5: the Preview panel classifies the webcam image live. Notice what happens when both shapes are in view.
- Go to teachablemachine.withgoogle.com, click Get Started, and create a new project. If you’re not sure which to pick, choose Image Project → Standard image model, it is the easiest to work with.
- Define two or three categories (Teachable Machine calls them classes). Rename them from
Class 1andClass 2by clicking the pencil icon. Keep them simple and clearly distinguishable: hat/no hat, thumbs up/thumbs down, a mug/a book, or two shapes cut from coloured paper, as in the videos. - Collect samples for each class, using Webcam (hold the button to record a burst of images) or Upload. Aim for a few dozen samples per class.
- Click Train Model, and keep the tab open while it trains. It takes from a few seconds to a minute.
- Test the model in the
Previewpanel: show it each of your classes, and watch the confidence bars. - Keep experimenting: add more samples to one class and retrain. Does the model’s behaviour change for the other classes too?
As you work, notice the assumptions you are building in. What makes something count as a hat? What did you include in your examples without meaning to: the background, the lighting, your face, the angle of the camera? The model has no idea which of these you meant it to learn.
In a group: work in groups of 3–4. When you’re done, export your model: click Export Model, and on the Tensorflow.js tab choose Upload (shareable link), then click Upload my model. Copy the sharable link that appears, and share it with another group, along with the data type, your categories, and a one-line description of what the model does. The link shares the trained model, not the images you trained it on.
Part 2: Testing a Model
Now try to break a model: yours, or another group’s.
On your own: test your model under conditions that differ from your training data:
- change the lighting or the background
- move closer to or farther from the camera, or tilt it
- ask someone else to try it, or use objects that are similar but not identical to your examples
- show it something that belongs to none of your classes
With someone else’s model: in a group, open another group’s model link. On your own, try the model from the videos, trained on a green paper square and an orange paper circle: draw, cut out, or find some shapes to show it. Whichever model you use, read what it is for, and predict how it will behave before you test it.
Either way, here are some things to look for:
- Does the model work as described?
- Which inputs does it misclassify? What does that reveal about the training data?
- What happens with ambiguous or unusual examples?
Focus on why the model fails when it does, not just that it fails. The failures are the most instructive part of this exercise.
- It works well on examples like its training data, and much less well on anything else. A model trained in one room, in one light, often fails in another.
- It learns shortcuts. If everything in one class shared a background, a colour, or a person, the model may have learned that instead of the thing you meant. You’ll often find this out only by changing that one thing (Geirhos et al. 2020; Ribeiro et al. 2016).
- It’s always confident about something. Show it something that belongs to none of your classes, and it still has to choose one, often with high confidence. A classifier can only answer the question it was trained on: it has no way of saying “none of these” (Nguyen et al. 2015; Hendrycks and Gimpel 2017).
- More examples of one class can change the others. The model learns what separates the classes, so changing one changes the boundary for all of them.
An Experiment: Shape or Colour
The model in the videos was trained on green squares and orange circles. Every example it saw differed in two ways at once, shape and colour, so either one would separate the classes perfectly. Which did it learn?
Predict, before you look: what will it say about a green circle? And an orange square?
The model learned shapes. The green circle came out as Orange Circle (96%), and the orange square as Green Square (99–100%).
That’s what we wanted, but nothing in the training data could have told us so in advance: both explanations fit every example the model had seen. Only an input that pulls the two apart, a green circle, could reveal which one it had learned. With different samples (other lighting, other paper, other shapes) it could have gone the other way: image models trained on large photo collections often rely more on texture than on shape, the opposite of people (Geirhos et al. 2018). To find out what a model has learned, you have to test it on the cases your training data didn’t cover.
Explaining Model Behaviour
Everything the model knows comes from your examples. It found whatever statistical pattern best separated your classes in the samples you gave it, and that pattern may or may not be the one you had in mind. It can’t tell the difference between the object and everything else in the picture, unless your examples made the difference clear: the same object in different places, lights, and angles, and different objects in the same place.
This is the same for every model, including the largest of them. The data is where the model’s abilities come from, and also where its blind spots come from. Model Training looks at how this works, and why testing a model on data unlike its training data is the most reliable way to find out what it actually learned.
Key Takeaways
- Training a model means choosing categories and examples, and every choice builds in assumptions.
- A model learns whatever separates your examples, which may not be what you intended: the background, the lighting, your face.
- A classifier always picks one of its classes, even for inputs that belong to none of them, and its confidence doesn’t tell you when it’s wrong.
- The real test of a model is data that differs from its training data. How and why it fails there tells you what it actually learned.
- Large models are trained on the same basic principle, at vastly greater scale, with the same dependence on their data.
If you want to read more, the following are a good place to start:
- Geirhos et al. (2020): a short essay on how models learn shortcuts instead of the thing you meant, with examples from image recognition, medicine, and language. The figures make the point without the technical detail.
- Nguyen et al. (2015): a research paper, but its images are worth a look on their own: white noise and abstract patterns that a model classified, with 99.99% confidence, as a lion and other familiar things.