Skip to content
Hansi Lee
← Back

Embedded ML · Project founder · Aug 2024 – Jun 2025

An ML Sudoku Solver Built on a Microcontroller Camera

The goal: point a camera at a Sudoku and get it solved. That means running digit recognition on a microcontroller-class device, then handing the grid to a solver. I founded the project in Chang Gung University's Intelligent Control Lab: I built the rig, trained and deployed the models, and wrote an original solving algorithm that finishes in under 0.2 seconds.

Read the engineering notebook (PDF) →
Single-digit F1
~94%
Hardware
AI Thinker ESP32-CAM
ML tooling
Edge Impulse → Python
Solver speed
< 0.2 s per puzzle

Try the solver

This is the solver half of the project, running live: backtracking that always fills the most constrained cell first. Pick Hard to watch it guess, hit a dead end (orange) and back up.

placements 0 · backtracks 0

Hardware rig

I built the enclosure from LEGO so I could change the camera height and angle quickly and measure them exactly. Both had to stay fixed for the model to work. An iPad underneath acts as a backlit input surface for sharper digit contrast, and side openings keep the ESP32-CAM's reset button reachable when switching firmware. The camera streams over Wi-Fi from Arduino IDE, and I tuned its angle, zoom and lighting.

Hand sketch of the rig with dimensions 13×13×12 studs
Design sketch: 13 × 13 × 12 studs.
Laptop, oscilloscope and LEGO rig on a lab bench
Testing on the lab bench.

Three stages of the model

  1. Stage 1

    Single-digit transfer learning

    Trained an Edge Impulse transfer-learning model on MNIST handwritten digits. I tuned dataset size, training cycles, learning rate and batch size until it reached about 94% F1 without over- or underfitting, then deployed it as an Arduino library running on the ESP32-CAM, predicting every 7–8 seconds. Testing showed how much accuracy depends on digit size, stroke weight and screen brightness.

    Edge Impulse training dashboard showing model accuracy and a confusion matrix
    Edge Impulse training run for the single-digit classifier.

  2. Stage 2

    Whole-grid object detection

    To read the whole grid at once, I moved to an object-detection model that outputs each digit's class, confidence, position and size. Labeling every training image by hand was the bottleneck, so I used generative AI tools to speed it up. Accuracy fell well short of the single-digit model, limited by the more complex task and the free tier's training budget.

    Serial monitor output listing detected digits with confidence scores and bounding boxes
    Object-detection output: label, confidence and bounding box per digit.

  3. Stage 3

    Moving recognition into Python

    Working with students in the lab, I rebuilt recognition in Python so it could merge directly with my Python Sudoku solver. The current version reads a 3×3 grid and checks it against Sudoku rules. That's a narrower task, chosen on purpose to get the whole pipeline working end to end before scaling up.

    Arduino serial monitor printing per-digit class probabilities
    Per-digit probabilities streamed from the on-device model.

Takeaways & next steps

  • On-device ML is limited by data conditions (lighting, scale, contrast) as much as by model choice.
  • When a tool's ceiling became the bottleneck, I changed the toolchain instead of over-tuning.
  • Next: train a stronger recognizer in Python with more data, scale from 3×3 to the full 9×9 grid, and harden the rig for everyday users.