Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Real-world benchmarking for (Py)PREP #98

Open
a-hurst opened this issue Jun 28, 2021 · 0 comments
Open

Real-world benchmarking for (Py)PREP #98

a-hurst opened this issue Jun 28, 2021 · 0 comments
Labels
enhancement New feature or request
Milestone

Comments

@a-hurst
Copy link
Collaborator

a-hurst commented Jun 28, 2021

As discussed in #74 and #41, for future improvements in PyPREP it would be handy to have some sort of real-world test battery that measures how well PyPREP does its job. That way we can better tune PREP's default settings, test proposed changes to see if they actually improve the resulting signal/noise ratio, and have a cool way of empirically showing off the utility of the software (and comparing it to other approaches). This wouldn't be something to add to CI as it would almost certainly be pretty slow, but rather a script or set of scripts to run locally for the sake of testing.

One of the ways I think we could do this would be using BCI datasets and algorithms, where the quality of data cleaning could be measured in terms of how much PyPREP improves (or doesn't) the classification accuracy of a range of different BCI algorithms. A good candidate for facilitating this is moabb, which is a package designed specifically for large-scale comparisons of various EEG classification algorithms across a range of different datasets (see section 23 of the Python notebook here for a good illustration of what I mean). I hacked together a basic script for testing PyPREP against the BCI2000 dataset with this package a while ago but had to drop it for other projects, will report back here sometime soon with some preliminary results!

There's also the .ipynb for comparing Python EEG cleaning tools in a similar manner here, which @sappelhoff shared a while ago in the Zenodo issue. It might be worthwhile adding PyPREP into that to see how it stacks up against other Python methods for the same purpose.

@a-hurst a-hurst added this to the 0.5.0 milestone Jun 28, 2021
@a-hurst a-hurst added the enhancement New feature or request label Jun 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant