Why am I unable to detect spikes and only noise?

[Auxcy1ene]

Hi there,

I am a masters student at the University of Exeter studying Natural Sciences. I am currently working on my masters project in which I am comparing MEA recordings of healthy recordings and diseased recordings and investigating how the models for each differ, using the model laid out in the paper by Zaytsev et al.: https://pmc.ncbi.nlm.nih.gov/articles/PMC4493949/pdf/10827_2015_Article_565.pdf.

I am using kilosort to detect spikes in my data before feeding it into the modelling code. After a while I was finally able to get kilosort to process my data. Little did I know that at that time although kilosort gui had completed without any software error, the number of clusters detected was far fewer than expected. Only three were detected originally and the recordings were a 4x4 electrode array recording between 50 to 100 neurons. I have been able to increase the number of clusters detected to around 17 by tweaking the kilosort settings, but then I realised another problem.

When viewing the results in phy I realised that the spike waveforms were not actually spikes, it appeared that all the spikes in each cluster were not spikes. I don’t know if noise is the right term for them but they were all just slightly different zigzags. I have uploaded a picture below for reference.

My question is what is going wrong here? Is the issue with the parameters I have selected in kilosort, or is there an error somewhere in the pipeline whereby the raw recordings, in axion biosystems format, are extracted in matlab and then written to a nwb file in python. Another possible source of the error could be the step where the nwb file is converted to a binary file to facilitate use with kilosort gui.

I will a link to a zip folder containing the kilosort results, alongside the recordings at different stages of processing in case that can help anyone isolate the source of the issue.

https://drive.google.com/file/d/1zV2lamdQ0E5tIWg_jgTOvUzG52A6K6Kz/view?usp=sharing

If anyone can help me resolve this issue I would really appreciate it. Also have a happy Christmas.

Many Thanks,

From Sean.H

[jacobpennington]

Can you please upload kilosort4.log from the results directory separately in a comment? A screenshot of the KS4 GUI with your data loaded would also be helpful. You don't need to run the sorting again, just open it and click "load."

[Auxcy1ene]

Will do. Not at my desk at the moment but I'll see if I can do it remote from my phone.

[Auxcy1ene]

here is the link to the kilosort.log. I was unable to upload the file as I couldn't download it to my phone. Also here is a screenshot of kilosort gui with the same data loaded.

[Auxcy1ene]

https://drive.google.com/file/d/19pwwptO6neo5Mb7NgX2IB134REgpnYGl/view?usp=drivesdk sorry the link didn't work in the last comment.

[jacobpennington]

Please try sorting using the default settings aside from n_chan_bin and fs, and set nblocks = 0 to disable drift correction. In particular, setting Th_universal and Th_learned to smaller values than the defaults is likely to resort in more "spikes" detected that are actually noise. You should not set the values based on previous versions of Kilosort, start with the defaults for Kilosort4.

[Auxcy1ene]

Thank you for the advice I will run it again and post the results here.

[Auxcy1ene]

I got the same error I got a while ago again. It is an error with truncatesvd . Ages ago when I kept getting this error I eventually determined it was occurring because no spikes were being detected by kilosort which is why the array had one dimension with a size of 0. This is why I progressively lowered all of the thresholds until I was able to detect any 'spikes' (noise). The problem is when I viewed a portion of one of the traces from a single channel using the axion bio systems mat lab code I could personally see some very clear spikes so I know they are there but they are not being picked up by kilosort.

[jacobpennington]

Please upload kilosort4.log from the latest run that generated the error.

…

On Sat, Dec 28, 2024, 10:04 AM Auxcy1ene ***@***.***> wrote: I got the same error I got a while ago again. It is an error with truncatesvd 9D54336A-DACC-4327-A670-0709B184EF77.jpeg (view on web) <https://github.com/user-attachments/assets/f124b2a4-787b-4832-98da-43e3fbe288a7>. Ages ago when I kept getting this error I eventually determined it was occurring because no spikes were being detected by kilosort which is why the array had one dimension with a size of 0. This is why I progressively lowered all of the thresholds until I was able to detect any 'spikes' (noise). The problem is when I viewed a portion of one of the traces from a single channel using the axion bio systems mat lab code I could personally see some very clear spikes so I know they are there but they are not being picked up by kilosort. — Reply to this email directly, view it on GitHub <#840 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIQ6WYFT4WD732W3KQNAMIL2H3RZPAVCNFSM6AAAAABTZQ5TAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRUGM4TENBWHA> . You are receiving this because you commented.Message ID: ***@***.***>

[Auxcy1ene]

Here is the log file from the last run.
kilosort4 (1).log

[jacobpennington]

@Auxcy1ene There's some kind of issue with your data.bin file, when I load it myself with numpy (i.e. not using Kilosort4 at all) it's just noise. I scanned through several 100000 sample segments, and they all look like this:

Some things to check:

• Make sure the file is converted correctly from NWB.
• Check that you're using the correct dtype for the data.

[Auxcy1ene]

Thank you Jacob, I suspected there was an issue someone in my conversion pipeline. I have recently validated the integrity of the nwb file by directly comparing it with the source file voltage plots and they are almost exactly the same. I will look into the conversion from nwb file to data.bin. Thanks for the advice. I will let you know if I have any problems tomorrow. Also happy new year and thank you for the help.
From Sean.H

[Auxcy1ene]

Hello again,
Sorry its been a while but it took me a while to fix the issue. When you told me that it was likely a datatype mismatch I was certain that must be the case so I went through everything to make sure it didn't change but I still got the same issue. Eventually I decided to stop using float datatype and just convert to int16. In an effort too preserve the amplitude resolution I ran a script to find the largest absolute value in the recording, divided 32000 by that number and then multiplied the recording data by that scaling factor before converting to int16. After following the same steps I previously used to load it into kilosort gui, except this time I used kilosort gui to convert it to a binary file (note this didnt fix the issue when I tried to do the same with different datatypes, only int16), and I have finally managed to obtain visible waveforms in phy. I have uploaded a couple examples below.

There are still some issues though. My understanding of the waveform view panel is that the different numbered waveform plots show the waveforms that are associated with the selected cluster across the different channel numbers. I would expect there to be more than one channel where the spike 'fingerprint' has been identified but this doesn't appear to be the case. Are my parameters potentially the cause of this?
Another issue is that some of the waveforms do appear to have the correct spike shape, where the voltage dips drastically before rising a little above baseline and returning to normal, but some have this the other way around with the rise coming before the dip. As the recording is of extracellular data I would assume that the spikes should be a dip followed by a rise so I was wondering if there is a way to tell kilosort to ignore these instances or should I manually omit them from my results?
Finally I was wondering if you have any advice or could point me to some resources that I could use to refine my parameters further now that I have been able to detect spikes as the detected waveforms are still not clean and clear enough for me to use. In my recent experiments with different parameters I was unable to get more than 3 spike clusters detected but I was eventually able to detect more after increasing the 'x centers' value to 10. As far as I understand it this value represents the spacing between the centers used when looking for spikes that potentially fit the detected waveform. Increasing this value manually feels like a brute force method that might cause problems further down the line to me. Are my fears regarding this valid?

By the way I want to thank you again for the help you've given me, with it I've already managed to remove a bottleneck which has prevented me from progressing in my project, and if you could give me some advice to improve it further I would greatly appreciate it.
Many Thanks
From Sean.H

P.S. I just remembered I should probably give you the data for the last run I did so here it is https://drive.google.com/file/d/1HtC24khUSPv_EEumsYLjn_krLC-rKqzE/view?usp=sharing