Let's approach this in a different sense. Imagine you are discovering math as it was in earlier times. The first problem you encounter is the necessity of counting. We go hunting, and you see a group of cattle (what did people used to hunt...?) and, upon returning to the hunting party, you want to notify the group on how many cattle there are. In this sense you say a number $n$ and then notify the group that it is $n$ cattle. The unit here would therefore be the head of cattle.
This is an interesting problem for a mathematician. We had to have come up with a counting system that is systematic throughout all units. For example, 1 cattle and 1 apple have the same count, but they have different units. As pure mathematics grew, the units became tools to relate these numbers. Essentially, we could use the natural numbers, $\mathbb{N}$, to express the count of objects in a set. This is referred to as the cardinality of a set.
Later, people realized that these counting numbers could also represent lengths. But another problem arose: how do you express a number in the middle of $0$ to $1$ or $1$ to $2$? In this sense we expanded the definition of the number system to include the positive rational numbers, $\mathbb{Q^+}$. As mathematics grew, and more people applied mathematics to the world around it, more units were added to give meaning to these numbers.
In an applied world, this is good. Finding meaning behind math and mathematical objects is a very natural idea, and as you have pointed out, this is very reasonable to see the relation between units and numbers. However, numbers extend further than our scope of universal intuition. In a pure sense, numbers are unitless objects. Mathematical objects are "ideas" such that in every sense we could think of the number "1" acts like the number "1". No matter it's shape or form.
As we expanded our set of numbers to include $\mathbb{C}, \mathbb{H}, \mathbb{O}$ it becomes difficult to relate these numbers to a universal system of measurement because what meaning does it make to say that we have $1+i$ cattle? It makes very little sense at all! If you are interested in this subject however, this (as I see it) is a good way to view the creation of set theory. Also, if you want to learn more about this type of thing, mathematics is the right place to look!