Example
Suppose we want to model the \(sin\) function using an arbitrary method for solving the coefficient weights \(w\). To highlight the problems of curve fitting, we will only use a few points in our training data which are randomly sampled from \(sin\). In the figure below, the blue line represents the prediction of our model and the green line, the true sin function.
Using (\( \htmlId{tooltip-integer}{k} \) = 1), we only obtain a straight line. As expected, this does not model the \(sin\) function very well. Since the errors of the predictions are high concerning the training data, we refer to this phenomenon as underfitting.
With (\( \htmlId{tooltip-integer}{k} \) = 3), we achieve a good fit. The model's curve more closely follows the shape of the true sin function. This demonstrates how a well-chosen polynomial order can capture the underlying pattern in the data without excessive complexity.
When (\( \htmlId{tooltip-integer}{k} \) = 5), we observe overfitting. While the model might pass nearly perfectly through all the training points, its curve exhibits erratic fluctuations that deviate significantly from the true \(sin\) function. This highlights how an overly complex model can become too attuned to the noise in the training data, failing to generalize well to unseen examples.