One of the differences between physics and mathematics is that mathematicians don’t tend to care about the units they are working with. In fact, they will usually consider all quantities as unitless^{1}. This makes it easy to compare quantities, because one only has to look at the number itself. If you have two numbers, 5 and 9, you know that 9 is the larger quantity.
In physics, however, the situation isn’t quite the same. That’s because our quantities have dimensions attached to them. As such, it doesn’t make sense to say that 5 L is larger than 3 m, since they don’t describe the same property of a system. Therefore, in physics we require more than just a comparison between the value of numbers themselves. We want the dimensions of the quantities to match up as well.
Things can get tricky though, because different people use different units to describe the same dimension. For our notion of length, we have plenty of units, from the metre to the yard to the light year. They all represent length, but there’s a huge difference between one metre and one light year. From this, we conclude that in addition to requiring quantities to be in the same dimension in order to compare them, they also need to have the same units.
A related notion is that of a quantity being “large”. If anyone tells you that a quantity is large, your first question should be, “Compared to what?”
There is no such thing as an “absolute” size. In other words, a quantity can only be large compared to something else. You might think that 300,000,000 m is an enormous length, but light travels that distance in about one second, which means that a light year is about 31.5 million times this length. As such, it doesn’t make sense to only say that 300,000,000 m is large. It needs to be compared to something else. Only then can the notion of “large” have meaning. (Think of it like an inequality. You can’t have an inequality with only one quantity. You need two.)
If this seems like it can get messy with all of the different units people use, you are correct. This is why many physicists like to use dimensionless quantities such as ratios. If the ratio involves two quantities with the same dimension, the ratio will “cancel out” the dimensions, leaving a dimensionless quantity. This is useful because it means one doesn’t have to worry about the units involved in the problem. No matter what units you use to measure my weight and your weight, the ratio of our weights will be the same no matter what instrument we use.
The next time you hear someone saying that a quantity is large, make sure to remind yourself what they are comparing their quantity too. Without doing this, there’s a chance for misunderstanding or manipulation. Therefore, don’t jump to conclusions when numbers are thrown around with the implication that they are large or small. Demand another number to compare it to!

Actually, theoretical physicists like to do this too, since everyone agrees that dealing with units can be annoying. This is why you might see physicists saying that the speed of light is c=1. ↩