[Neuralynx] Add multiple streams

[JuliaSprenger]

To load multiple ncs files with different sampling rates multiple signals streams are required on the RawIO level. Here, for each set of channels sharing (sampling rate, n_samples, t_start) encountered in the files a dedicated signal stream is created to permit loading the data. Stream are ordered by descending sampling rate.

This PR also changes the event, epoch and spikes loaded for a segment. Previously all time point data were loaded for all segments, extending beyond the continuous signal of that segment. Now only time point data in the corresponding time frame of a segment are considered. If ncs data are available these define the segment structure. To load all time point data in a single segment all ncs files can be excluded resulting in a single segment containing all event, epoch and spiking data.

This PR is building on top of #990..
For the original discussion, see #1038

[pep8speaks]

Hello @JuliaSprenger! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-05-13 11:17:03 UTC

[TheChymera]

I tried this out and I get:

chymera@neurohost ~/src/neuralynx_nwb/neuralynx_nwb $ python -c 'import convert; convert.reposit_data()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/chymera/src/neuralynx_nwb/neuralynx_nwb/convert.py", line 44, in reposit_data
    reader = neo.io.NeuralynxIO(dirname=session_data, keep_original_times=True) # TODO: newer version should support: , keep_original_times=True)
  File "/home/chymera/src/python-neo/neo/io/neuralynxio.py", line 51, in __init__
    BaseFromRaw.__init__(self, dirname)
  File "/home/chymera/src/python-neo/neo/io/basefromrawio.py", line 74, in __init__
    self.parse_header()
  File "/home/chymera/src/python-neo/neo/rawio/baserawio.py", line 185, in parse_header
    self._parse_header()
  File "/home/chymera/src/python-neo/neo/rawio/neuralynxrawio/neuralynxrawio.py", line 291, in _parse_header
    self._sigs_memmaps, ncsSegTimestampLimits = self.scan_ncs_files(self.ncs_filenames)
  File "/home/chymera/src/python-neo/neo/rawio/neuralynxrawio/neuralynxrawio.py", line 635, in scan_ncs_files
    if v[0].is_equivalent(latest_sections, abs_tol=tolerance):
AttributeError: 'NcsSections' object has no attribute 'is_equivalent'

[JuliaSprenger]

Hi @TheChymera, sorry for not making this clear in the first place: I opened this PR as a Draft as it is not complete yet. I am still working on the issue and will finalize the PR this week.

[TheChymera]

@JuliaSprenger no worries, I just wanted to share feedback in case it might useful :) Also, what example data do you work with, did you get the full vStr_phase_stim dataset, or just one of the M* subsets?

[JuliaSprenger]

@samuelgarcia I resolved conflicts with the current master and tests should pass now. Can you review if you have time?

[samuelgarcia]

Hi Julia,
this looks good.
I made some comments in parse header. Maybe we can enforce check to be sure that channel in the same stream have same sampling rate(done already) but also same lenght and same t_start.

Do we have somewhere something that check that the same stream have the same gaps ?
How do we handle if diffrents stream have differents gaps ? (this would be a noghtmare no ?)

[JuliaSprenger]

Hi @samuelgarcia thanks for the feedback. You are right, the gap handling should happen first on a per-stream basis and then across streams as the requirements are a bit different. I will update the PR soon.

[yarikoptic]

Hi @JuliaSprenger , we are hacking in DANDI/NWB hackathon and @TheChymera apparently still uses his own, more hacky version. It would be great to see this PR to finalize/gets merged so we could use it instead. Please let us know if there is anything specific we could be of help with if anything.

[JuliaSprenger]

Failing IO tests are due to a certificate update on the gin side and outdated cached versions. Hopefully this issue will resolve itself in the next hours, see https://gin.g-node.org/G-Node/Info/issues/59

[JuliaSprenger]

@samuelgarcia Thanks for having a first look. I addressed your comments. Can you have a look again? I am only waiting for confirmation to put some corresponding test files on gin to add a minimal stream test.

[JuliaSprenger]

TODO: Ensure consistent ordering of streams - done

[JuliaSprenger]

@samuelgarcia I rebased on master and ordered stream by descending sampling rate.

[samuelgarcia]

Perfect. Can I merge now ?

[JuliaSprenger]

If the tests still pass: Yes please merge it :)

[samuelgarcia]

Salut Julia,
test are not passing. Could you have a look ?

[JuliaSprenger]

@samuelgarcia I rebased on master and tests are passing now. Ready to merge.

[JuliaSprenger]

@samuelgarcia ping?