Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wonder if Kalman filter fits imputing with lots of missing #519

Open
XiaoJia849 opened this issue Dec 10, 2024 · 1 comment
Open

Wonder if Kalman filter fits imputing with lots of missing #519

XiaoJia849 opened this issue Dec 10, 2024 · 1 comment

Comments

@XiaoJia849
Copy link

i use Kalman filter to impute missing value in the time series, in the pic, the origin curve is the original curve, the red zone is missing, and i use UnscentedKalmanFilter to do the imputation, which generates the blue parts.

image

Obviously, in small missing area, the filter fits great, while in the big missing area, the imputation seems work bad. I wonder if Kalman filter fits imputing with lots of missing, or maybe i change some hyperparameters can fix this problem.

Here are my codes, part of them are copied from filterpy.kalman.UKF.UnscentedKalmanFilter. Thank you very very very very very much !!!!!!

def init_filter(fx,hx,initial_x):
    dt = 0.2
    # create sigma points to use in the filter. This is standard for Gaussian processes
    points = MerweScaledSigmaPoints(3, alpha=.1, beta=2., kappa=0)
    kf = UnscentedKalmanFilter(dim_x=3, dim_z=1, dt=dt, fx=fx, hx=hx, points=points)
    kf.x = np.array([initial_x, 0,0]) # initial state
    kf.P *= 0.2 # initial uncertainty
    z_std = 0.1 
    kf.R = np.diag([z_std**2]) 
    kf.Q = Q_discrete_white_noise(dim=3, dt=dt, var=0.01**2, block_size=1)
    return kf

def fx(x, dt):
    F = np.array([[1, dt,.5*dt*dt],
                  [0, 1,dt],
                  [0.,0.,1]], dtype=float)
    return np.dot(F, x)
def hx(x):
    # measurement function - convert state into a measurement
    # where measurements are [x_pos, y_pos]
    return np.array([x[0]])
    

def process_one_flux(fx,hx,input_flux):
    input_flux=input_flux.copy()
    idxs=np.argwhere(np.isnan(input_flux)).flatten()
    diff_idxs=np.diff(idxs)
    start_i,_=idxs[np.argmax(diff_idxs)]+1,idxs[np.argmax(diff_idxs)+1]
    initial_x=input_flux[start_i]
    kf=init_filter(fx,hx,initial_x)
    outs=[] 
    zs=input_flux.reshape((-1,1))[start_i:].tolist()
    for z in zs:
        if np.isnan(z):
            kf.predict()
            kf.update([kf.x[0]])
            outs.append(kf.x[0])
        else:
            kf.predict()
            kf.update(z)
            outs.append(kf.x[0])
    outs=np.array([np.nan]*(start_i)+outs)
    mask=np.isnan(input_flux)
    input_flux[mask]=outs[mask]
    if start_i>0:
        initial_x=input_flux[-1]
        # print("initial_x",initial_x)
        kf=init_filter(fx,hx,initial_x)
        outs=[] 
        zs=input_flux.reshape((-1,1)).tolist()
        for z in zs[::-1]:
            if np.isnan(z):
                kf.predict()
                # kf.update([kf.x[0]])
                outs.append(kf.x[0])
            else:
                kf.predict()
                kf.update(z)
                outs.append(kf.x[0])
        outs=np.array(outs[::-1])
        mask=np.isnan(input_flux)
        input_flux[mask]=outs[mask]
    return input_flux
    
@XiaoJia849
Copy link
Author

for up pic, in fact many small missing zone didn't successfully show, the pic next show all the missing zones:
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant