I came across the concept of Root Mean Square Error (RMSE) two days ago at work, while investigating 3-D localization using Google Cartographer. I was familiar with the difference between RMSE and standard deviation previously, but at work I seemed to have gotten them mixed up (again). Like various other concepts that I frequently learned to forget, I figured I should write it down somewhere in case I need a quick refresher, hence this post.
To get started, let's begin with their similarities. The standard deviation, generally represented by the Greek lowercase sigma letter $\sigma$, and the RMSE are similar mathematically: they are both metrics to measure the variance of a variable from a reference. Put simply, they tell you how much a variable you are measuring/estimating1 vary from a reference. For example, if you get a high value in $\sigma$ or RMSE, you know that the variable that you are measuring/estimating tend to vary by a lot around the point of reference.
We only need to look at their formulas to see that they are very similar: $$\textrm{Standard deviation, }\sigma = \sqrt{\frac{1}{N}\sum^{N}_{i=1}(x_{i} - \bar{x})^2}$$ where $x_i$ is the measured value, and $\bar{x} = \frac{1}{N}\sum^{N}_{i=1}x_{i}$ is the mean. For simplicity sake I'm going to use $N$ instead of $N-1$ as the denominator; you can see here to learn more about their difference. $$\textrm{Root Mean Square Error, }RMSE = \sqrt{\frac{1}{N}\sum^{N}_{i=1}(x_{i} - \hat{x}_{i})^2}$$ where $x_i$ is the 'true'2 value, and $\hat{x}_{i}$ is the estimated value.
Through the mathematical formulas you can easily observe their differences: for $\sigma$, the reference is the mean of a sample set while for RMSE, the reference is the 'true' value.3 With this, we can re-visit the simple definitions again:
- $\sigma$ measures the variance about the mean.
- RMSE measures the variance about — or more appropriately — the deviation from the 'true' reference value.
An example will help outline the significance of these two formulas. Earlier in this post I spoke about the motivation for this post based on robot localization: to help a robot localize (know its position in the world), we implement algorithms that use sensors to estimate the robot's position; this is analogous to humans using senses to identify their position and orientation in the world.
However, the algorithms are imperfect, thus providing inaccurate robot position estimates. To quantify the inaccuracy, we need to know the robot's true position. (I'm not going to delve into how we get the robot's true position, so we can assume we have it readily and it's accurate enough to be considered the truth.) Then, we can move the robot around in a controlled environment and have both the information of its true and estimated position that we can use to compare.
To obtain the RMSE, we take the difference between the true and estimated values, i.e., the error $x_{i} - \hat{x}_{i}$ (E), square each of the errors $(x_{i} - \hat{x}_{i})^2$ (S), find the mean of all the squared errors $\frac{1}{N}\sum^{N}_{i=1}(x_{i} - \hat{x}_{i})^2$ (M), and then find the square root of the resultant value $\sqrt{\frac{1}{N}\sum^{N}_{i=1}(x_{i} - \hat{x}_{i})^2}$ (R) = RMSE (the order of operation is from the right to left).
Similarly, say we want to find the $\sigma$ of the error. In that case, we first find the mean of the error $\bar{e} = \frac{1}{N}\sum^{N}_{i=1}e_{i}$ where $e_{i} = x_{i} - \hat{x}_i$, and then take the root mean square of that difference $\sqrt{\frac{1}{N}\sum^{N}_{i=1}(e_{i} - \bar{e})^2}$ (see how similar that is to RMSE?).
Now that I've outlined how they are different, the next question becomes: how/where do we use them? In the same example, with the RMSE, I have a metric that tells me how much the estimation deviates from the ground truth reference. Meanwhile, with the standard deviation, I have a metric that tells me how much the estimation error deviates from the mean of the estimation error. For my use case, the RMSE means more to me than $\sigma$, since I want to know how good the estimated robot position is compared to the actual position; I don't really care about how much the error deviates from the mean. In a different use case however, you may care more about $\sigma$, e.g., examining the length of spaghetti coming out of the spaghetti machine: if $\sigma$ of the spaghetti length is high, that means you will get highly non-uniform spaghetti lengths, which may indicate a problem with your spaghetti machine (you can tell I'm a bit hungry now).
And that's the gist of it. Finally, I'd like to add on one thing that helps my intuition about $\sigma$: the 68-95-99.7 rule. Basically what this rule is about: say you found the mean to be $\mu$ and $\sigma = C$. This means ~68% of the time $x$ will fall in the range $\mu-C \leq x \leq \mu+C$. So in the spaghetti example, this means that if you get a mean length of 0.2 m and $\sigma$ is 0.05 m, that means about 680 out of 1000 spaghetti sticks is between the length 0.15 m to 0.25 m.4
1 I used 'estimating' because I'm talking with robotic applications in mind (e.g., localization estimates). 'Predicting' can also be used here, but that will apply better to data science applications. ↵
2 The term 'true' here really means ground truth, which are values we take to be absolutely correct: estimations that are close to ground truth to reflect good estimation. ↵
3 It may be clearer if you modify the formulas slightly so that the reference terms subtract the measurement or estimated values: $(\bar{x} - x_{i})$ and $(x_{i} - \hat{x}_{i})$ for $\sigma$ and $RMSE$ respectively. The exponent of 2 outside the parentheses allows this. ↵
4 This rule only applies to a normal (Gaussian) distribution. ↵
Comments
Post a Comment