Introduction
When we started working on a computer vision problem, we spent a lot of time thinking about the model.
Which architecture should we use? How many images would we need? What image size should we train with? How many epochs would be enough? Should we use a larger model?
Those are all reasonable questions. And we spent time answering them.
But after going through several training and validation cycles, we realized something that changed the way we approached the project:
The model was not always the problem. The dataset was.
That sounds obvious when you say it afterward. It wasn’t obvious when we were in the middle of the project. This is what we learned while building and improving a computer vision dataset for technical drawings.
We Thought More Images Would Solve the Problem
Our initial thinking was fairly straightforward. We had a detection problem, so we needed images. We collected images, created annotations, and started training.
The natural assumption was: More training data → better model.
So when we saw predictions that weren’t quite right, our first instinct was to think about the model. Maybe we needed more training. Maybe the model was too small. Maybe we needed more epochs. Maybe we needed more examples.
But then we started looking at the predictions more carefully.
Instead of looking only at the metrics, we opened the actual images and looked at what the model was doing.
That changed the conversation.
Some predictions were genuinely model mistakes. But some were much more interesting.
The model was making mistakes that, after investigation, could be traced back to the data we had given it.
Before asking why the model made a mistake, check what we taught it.
A Bounding Box Can Be Valid and Still Be Wrong
This was probably one of the most important things we learned.
When working with YOLO annotations, it is easy to think of an annotation as a simple technical format. Something like:
class_id x_center y_center width height
The annotation tells the model both what the object is and where the object is located.
Correct:
┌─────────────────────────────┐ │ Target Object │ │ ┌────────────┐ │ │ │ │ │ │ └────────────┘ │ └─────────────────────────────┘
But if an unrelated visual element is accidentally included, the model may learn features that do not actually define the target object. This can eventually result in false positives or poor localization.
Incorrect: ┌─────────────────────────┐ │ │ │ Target Object │ │ ┌────────────┐ │ │ │ │ │ │ │ Target │ │ │ │ + unrelated│ │ │ │ content │ │ │ └────────────┘ │ │ │ └─────────────────────────┘
Annotation Errors Can Be Silent
Not every annotation problem causes a training error.
Some problems are structural. For example: invalid class IDs, malformed label files, missing labels, missing images, invalid coordinates, or zero-sized bounding boxes.
These can often be detected automatically.
But other problems are semantic. The wrong object may have been labeled, part of an object may have been missed, too much surrounding content may have been included, similar objects may have been labeled inconsistently, or an object that should not be labeled may have been included.
A dataset can therefore pass an automated validation script and still contain serious annotation problems.
This is why automated validation and visual validation should be used together.
Looking at the Images Was More Useful Than Looking at the Numbers
Model metrics are important. Precision, recall, mAP and other measurements tell us how the model is performing.
But during dataset development, we found that one of the most useful debugging tools was much simpler: look at the image.
We started reviewing predictions alongside the original annotations.
This sounds simple. It is. But it is also easy to skip when you’re focused on training runs and metrics.
A number can tell you that something is wrong. The image often gives you a clue about why.
Technical Drawings Make This Even Harder
This becomes particularly important when working with technical drawings.
A normal photograph might contain an object against a relatively simple background. A technical drawing is very different.
One page can contain text, dimensions, tables, borders, symbols, lines, hatching, repeated geometry, labels, components, and many other visual elements.
Some of these can look surprisingly similar to the objects we actually want to detect.
That means the model has to learn a fairly subtle distinction: What is the target, and what just happens to look like the target?
This is where dataset diversity becomes important.
We don’t just need more examples of the target. We need examples showing the model the situations in which the target appears and the situations in which it does not appear.
Then We Had to Bring Multiple Datasets Together
As the project evolved, we ended up with data coming from different stages of development.
Initially, this was useful because we could work on different detection problems independently. Eventually, though, we needed a unified dataset.
That created another set of problems.
It wasn’t enough to put all the images into one folder. We had to make sure that the datasets agreed on things such as class definitions, class IDs, annotation format, directory structure, image/label relationships, and dataset splits.
One of the easiest mistakes to make here is class mapping.
Both datasets can be individually correct. Put them together without remapping, and suddenly the combined dataset is teaching the model two different things.
This is why we started treating the dataset definition almost like an API contract. Once the class mapping was established, it had to remain consistent.
We Started Treating the Dataset Like Software
This was another mindset change.
In normal software development, we don’t casually change production code and then forget what changed. We use versions, change history, reviews, tests, and release notes.
The same thinking is useful for datasets.
Annotations change. Images get added. Some samples get removed. Classes get corrected. Splits may change.
So instead of thinking, “This is our dataset,” we started thinking, “This is dataset version X.”
That small change in thinking made experiments much easier to understand.
If a model changed, we could ask: Was the model different, or was the data different?
Without dataset versioning, answering that question becomes surprisingly difficult.
One Correction Can Matter More Than Another Training Run
At one point, we found annotation problems in specific samples.
The easy thing would have been to open the annotation files, make the correction, and continue. But we wanted to know exactly what had changed.
So the correction became its own controlled step.
The important part wasn’t just fixing the annotation. It was being able to explain later: What changed, why did we change it, and which model was trained using the corrected data?
That becomes increasingly important as the project gets bigger.
Not Every Problem Should Be Fixed by Adding Data
This was another assumption we had to rethink.
When a model struggles with a particular type of object, adding more examples is an obvious response. Sometimes it is exactly the right response.
But we learned to ask a few questions first.
- Is the object actually represented correctly in the existing dataset?
- Are similar objects being labeled consistently?
- Are we missing a particular visual variation?
- Is the model seeing enough examples of the difficult case?
- Are the failures concentrated in a particular layout or location?
Only after understanding the failure does it make sense to decide whether we need more data, better annotations, different data, or a model change.
The Test Set Needs Discipline
There is another temptation during development.
The model performs badly on some test examples, so we change those examples. Then we train again. Then we evaluate again.
This feels productive, but it slowly changes what the test set means.
The test set should ideally remain something we can use to answer: How well does this model perform on data it hasn’t been optimized against?
That means we need to be careful about when and why test data is changed.
For development, validation data is much more appropriate. The test set should be treated with a little more respect.
Our Dataset Workflow Became a Feedback Loop
Eventually, we stopped thinking about dataset preparation as something that happened before training.
It became part of the development loop.
This was probably the biggest change in our approach.
The model became a way of finding weaknesses in the dataset. And the dataset became something that could continuously improve.
What We Would Do Differently From Day One
If we were starting the project again, we would put more effort into dataset engineering from the beginning.
Not because the model isn’t important. It is. But because a lot of model-development time can be wasted when the data isn’t well understood.
We would establish these things early:
- Define the classes clearly — everyone working on annotations should have the same understanding of what belongs to each class.
- Create annotation guidelines — especially for difficult or ambiguous cases.
- Automate structural validation — catch malformed annotations before training.
- Build visual validation tools — make it easy to see annotations over the original images.
- Version the dataset — know exactly which data produced each model.
- Keep the test set controlled — don’t turn the test set into another development dataset.
- Analyze failures before collecting more data — understand the problem before deciding how to fix it.
- Collect difficult examples intentionally — the most valuable new image is not always another easy example.
The Part We Underestimated
When we started, we thought of the dataset as something we needed in order to train the model.
Today, we see it differently.
The dataset is part of the system.
It has its own structure, rules, quality checks, versions, failures, improvements, and lifecycle.
And perhaps the most important lesson is this:
When a model makes a mistake, don’t immediately ask how to make the model smarter. First ask whether the data gave it enough information to make the right decision.
The Core Workflow
The practical workflow that emerged from these lessons can be summarized as follows:
Dataset
↓
Training
↓
Predictions
↓
Error Analysis
↓
Dataset Investigation
↓
Corrections
↓
New Dataset Version
↓
Retraining
↓
Evaluation
Conclusion
Building a computer vision system is not simply a matter of collecting images and training a model.
The real challenge is creating a dataset that accurately represents the problem we want the model to solve.
A strong dataset engineering process should define the problem clearly, establish consistent annotation guidelines, validate annotations automatically and visually, maintain consistent class definitions, include representative positive and negative examples, create meaningful train/validation/test splits, protect the independence of the test set, version datasets and document changes, and use model errors to identify opportunities for dataset improvement.
Don’t just collect data. Engineer it.
Because in computer vision, the path to a better model often begins with a better understanding of the data.
