np.polyfit giving incorrect coefficients for higher-order polynomial fits with noisy data
I've searched everywhere and can't find a clear answer. I'm trying to fit a polynomial to a dataset using `np.polyfit` in NumPy version 1.24.2, but the coefficients I'm getting seem incorrect, especially for higher-order fits. My dataset is somewhat noisy, and I expected the polynomial fitting to handle this, but the results are quite off from what I anticipated. Here's the code I'm using: ```python import numpy as np import matplotlib.pyplot as plt # Generate some noisy data np.random.seed(0) x = np.linspace(0, 10, 50) y = 2.5 * x ** 2 - 1.5 * x + 1 + np.random.normal(0, 5, size=x.shape) # Fit a 3rd degree polynomial coefficients = np.polyfit(x, y, 3) print("Coefficients:", coefficients) # Generate the polynomial function poly_func = np.poly1d(coefficients) # Plotting the results plt.scatter(x, y, label='Noisy data') plt.plot(x, poly_func(x), color='red', label='Fitted polynomial') plt.legend() plt.show() ``` When I run this code, the coefficients printed seem to suggest that the polynomial is fitting the data far worse than I expected. For instance, I get coefficients like `[0.05, -1.2, 3.0, 10.0]`, which don't capture the quadratic nature of the data at all. I've tried increasing the degree of the polynomial to 4 or 5, but the results are just as poor. Is there something I'm missing in terms of preprocessing the data or configuring the polynomial fitting? Should I be using any specific weighting or regularization techniques to improve the fit, particularly given the noise in my dataset? I am quite puzzled about how to proceed. My development environment is Linux. How would you solve this? I'm coming from a different tech stack and learning Python.