Add 'Electrophysiology' page.

.github/workflows/docs_build_and_deploy.yml

@@ -27,7 +27,9 @@ jobs:
     needs: linting
     runs-on: ubuntu-latest
     steps:
-      - uses: neuroinformatics-unit/actions/build_sphinx_docs@v2
+      - uses: neuroinformatics-unit/actions/build_sphinx_docs@main
+        with:
+          python-version: 3.11
 
   deploy_sphinx_docs:
     name: Deploy Sphinx Docs
@@ -37,6 +39,6 @@ jobs:
     if: github.event_name == 'push' && github.ref_name == 'main'
     runs-on: ubuntu-latest
     steps:
-      - uses: neuroinformatics-unit/actions/deploy_sphinx_docs@v2
+      - uses: neuroinformatics-unit/actions/deploy_sphinx_docs@main
         with:
           secret_input: ${{ secrets.GITHUB_TOKEN }}

.gitignore

@@ -1,3 +1,6 @@
+docs/source/ephys_at_swc/example_pipelines/auto_examples_ephys/
+sg_execution_times.rst
+
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]

docs/requirements.txt

@@ -1,8 +1,11 @@
 linkify-it-py
+matplotlib
 myst-parser
 nbsphinx
 numpydoc
 pydata-sphinx-theme
+
 sphinx
 sphinx-copybutton
 sphinx-design
+sphinx-gallery

docs/source/data_analysis/HPC-module-SLEAP.md → docs/source/behav_at_swc/HPC-module-SLEAP.md

@@ -12,31 +12,11 @@
 | [SLEAP](https://sleap.ai/)                                      | Social LEAP Estimates Animal Poses           |
 | [SWC](https://www.sainsburywellcome.org/web/)                   | Sainsbury Wellcome Centre                    |
 | [HPC](https://en.wikipedia.org/wiki/High-performance_computing) | High Performance Computing                   |
-| [IT](https://en.wikipedia.org/wiki/Information_technology)      | Information Technology                       |
 | [GUI](https://en.wikipedia.org/wiki/Graphical_user_interface)   | Graphical User Interface                     |
 | [SLURM](https://slurm.schedmd.com/)                             | Simple Linux Utility for Resource Management |
 
 ## Prerequisites
 
-::: {dropdown} Note on managed Linux desktops
-:color: info
-:icon: info
-
-The SWC's IT team offers managed desktop computers equipped with a Linux image. These machines are already part of SWC's trusted domain and have direct access to SLURM, the HPC modules, and the SWC filesystem.
-
-If you have access to one of these desktops,
-you can skip the pre-requisite steps.
-You may simply open a terminal, type `module load SLEAP`,
-and start using SLEAP directly, as you would on any local
-Linux machine. All SLEAP commands should work as expected,
-including `sleap-label` for launching the GUI.
-
-That said, you may still want to offload GPU-intensive tasks to an HPC node (e.g. because the desktop's GPU is not powerful enough or because you need to run many jobs in parallel). In that case, you may
-still want to read the sections on [model training](sleap-training)
-and [inference](sleap-inference).
-:::
-
-(access-to-the-hpc-cluster)=
 ### Access to the HPC cluster
 Verify that you can access HPC gateway node (typing your `<SWC-PASSWORD>` both times when prompted):
 ```{code-block} console
@@ -53,17 +33,15 @@ $ module avail
 ...
 SLEAP/2023-03-13
 SLEAP/2023-08-01
-SLEAP/2024-08-14
 ...
 ```
 - `SLEAP/2023-03-13` corresponds to `SLEAP v.1.2.9`
 - `SLEAP/2023-08-01` corresponds to `SLEAP v.1.3.1`
-- `SLEAP/2024-08-14` corresponds to `SLEAP v.1.3.3`
 
 We recommend always using the latest version, which is the one loaded by default
 when you run `module load SLEAP`. If you want to load a specific version,
 you can do so by typing the full module name,
-including the date e.g. `module load SLEAP/2023-08-01`.
+including the date e.g. `module load SLEAP/2023-03-13`.
 
 If a module has been successfully loaded, it will be listed when you run `module list`,
 along with other modules it may depend on:
@@ -83,8 +61,34 @@ While you can delegate the GPU-intensive work to the HPC cluster,
 you will need to use the SLEAP GUI for some steps, such as labelling frames.
 Thus, you also need to install SLEAP on your local PC/laptop.
 
-We recommend following the official [SLEAP installation guide](https://sleap.ai/installation.html).
-To minimise the risk of issues due to incompatibilities between versions, ensure the version of your local installation of SLEAP matches the one you plan to load in the cluster.
+We recommend following the official [SLEAP installation guide](https://sleap.ai/installation.html). If you already have `conda` installed, you may skip the `mamba` installation steps and opt for installing the `libmamba-solver` for `conda`:
+
+```{code-block} console
+$ conda install -n base conda-libmamba-solver
+$ conda config --set solver libmamba
+```
+This will get you the much faster dependency resolution that `mamba` provides, without having to install `mamba` itself.
+From `conda` version 23.10 onwards (released in November 2023), `libmamba-solver` [is anyway the default](https://conda.org/blog/2023-11-06-conda-23-10-0-release/).
+
+After that, you can follow the [rest of the SLEAP installation guide](https://sleap.ai/installation.html#conda-package), substituting `conda` for `mamba` in the relevant commands.
+
+::::{tab-set}
+
+:::{tab-item} Windows and Linux
+```{code-block} console
+$ conda create -y -n sleap -c conda-forge -c nvidia -c sleap -c anaconda sleap=1.3.1
+```
+:::
+
+:::{tab-item} MacOS X and Apple Silicon
+```{code-block} console
+$ conda create -y -n sleap -c conda-forge -c anaconda -c sleap sleap=1.3.1
+```
+:::
+
+::::
+
+You may exchange `sleap=1.3.1` for other versions. To be on the safe side, ensure that your local installation version matches (or is at least close to) the one installed in the cluster module.
 
 ### Mount the SWC filesystem on your local PC/laptop
 The rest of this guide assumes that you have mounted the SWC filesystem on your local PC/laptop.
@@ -110,14 +114,12 @@ $ rsync -avz <LOCAL-DIR> <SWC-USERNAME>@ssh.swc.ucl.ac.uk:/ceph/scratch/neuroinf
 ```
 :::
 
-(sleap-training)=
 ## Model training
-This will consist of two parts: [preparing a training job](prepare-the-training-job)
-(on your local SLEAP installation) and [running a training job](run-the-training-job)
+This will consist of two parts - [preparing a training job](#prepare-the-training-job)
+(on your local SLEAP installation) and [running a training job](#run-the-training-job)
 (on the HPC cluster's SLEAP module). Some evaluation metrics for the trained models
-can be [viewed via the SLEAP GUI](model-evaluation) on your local SLEAP installation.
+can be [viewed via the SLEAP GUI](#evaluate-the-trained-models) on your local SLEAP installation.
 
-(prepare-the-training-job)=
 ### Prepare the training job
 Follow the SLEAP instructions for [Creating a Project](https://sleap.ai/tutorials/new-project.html)
 and [Initial Labelling](https://sleap.ai/tutorials/initial-labeling.html).
@@ -132,7 +134,6 @@ i.e. *Predict* -> *Run Training…* -> *Export Training Job Package…*.
 - Make sure to save the exported training job package (e.g. `labels.v001.slp.training_job.zip`) in the mounted SWC filesystem, for example, in the same directory as the project file.
 - Unzip the training job package. This will create a folder with the same name (minus the `.zip` extension). This folder contains everything needed to run the training job on the HPC cluster.
 
-(run-the-training-job)=
 ### Run the training job
 Login to the HPC cluster as described above.
 ```{code-block} console
@@ -345,8 +346,7 @@ If you encounter out-of-memory errors, keep in mind that there two main sources
 - If requesting more memory doesn't help, you can try reducing the size of your SLEAP models. You may tweak the model backbone architecture, or play with *Input scaling*, *Max stride* and *Batch size*. See SLEAP's [documentation](https://sleap.ai/) and [discussion forum](https://github.com/talmolab/sleap/discussions) for more details.
 ```
 
-(model-evaluation)=
-## Model evaluation
+### Evaluate the trained models
 Upon successful completion of the training job, a `models` folder will have
 been created in the training job directory. It contains one subfolder per
 training run (by default prefixed with the date and time of the run).
@@ -385,7 +385,6 @@ The SLEAP GUI on your local machine can be used to quickly evaluate the trained
 
 For more detailed evaluation metrics, you can refer to [SLEAP's model evaluation notebook](https://sleap.ai/notebooks/Model_evaluation.html).
 
-(sleap-inference)=
 ## Model inference
 By inference, we mean using a trained model to predict the labels on new frames/videos.
 SLEAP provides the [`sleap-track`](https://sleap.ai/guides/cli.html?#inference-and-tracking) command line utility for running inference
@@ -483,7 +482,7 @@ the training-inference cycle. The basic steps are:
 In this section, we will describe how to test that the SLEAP module is loaded
 correctly for you and that it can use the available GPUs.
 
-Login to the HPC cluster as described [above](access-to-the-hpc-cluster).
+Login to the HPC cluster as described [above](#access-to-the-hpc-cluster).
 
 Start an interactive job on a GPU node. This step is necessary, because we need
 to test the module's access to the GPU.
@@ -535,23 +534,23 @@ name, temperature, memory usage, etc. If you see an error message instead,
 Next, load the SLEAP module.
 ```{code-block} console
 $ module load SLEAP
-Loading SLEAP/2024-08-14
+Loading SLEAP/2023-08-01
   Loading requirement: cuda/11.8
 ```
 
 To verify that the module was loaded successfully:
 ```{code-block} console
 $ module list
 Currently Loaded Modulefiles:
- 1) SLEAP/2024-08-14
+ 1) SLEAP/2023-08-01
 ```
 You can essentially think of the module as a centrally installed conda environment.
 When it is loaded, you should be using a particular Python executable.
 You can verify this by running:
 
 ```{code-block} console
 $ which python
-/ceph/apps/ubuntu-20/packages/SLEAP/2024-08-14/bin/python
+/ceph/apps/ubuntu-20/packages/SLEAP/2023-08-01/bin/python
 ```
 
 Finally we will verify that the `sleap` python package can be imported and can
@@ -572,7 +571,7 @@ This is normal. Subsequent imports should be faster.
 >>> import sleap
 
 >>> sleap.versions()
-SLEAP: 1.3.3
+SLEAP: 1.3.1
 TensorFlow: 2.8.4
 Numpy: 1.21.6
 Python: 3.7.12
@@ -604,7 +603,10 @@ $ exit()
 If you encounter troubles with using the SLEAP module, contact
 Niko Sirmpilatze of the SWC [Neuroinformatics Unit](https://neuroinformatics.dev/).
 
-To completely exit the HPC cluster, you will need to type `exit` or
-`logout` until you are back to the terminal prompt of your local machine.
+To completely exit the HPC cluster, you will need to logout of the SSH session  twice:
+```bash
+$ logout
+$ logout
+```
 See [Set up SSH for the SWC HPC cluster](../programming/SSH-SWC-cluster.md)
 for more information.

docs/source/behav_at_swc/index.md

@@ -0,0 +1,10 @@
+# Behaviour
+
+Guides related to the analysis of behavioural data.
+The focus may be on the use of specific software tools, or on more general analysis tasks and concepts.
+
+```{toctree}
+:maxdepth: 1
+
+HPC-module-SLEAP
+```

docs/source/conf.py

@@ -42,6 +42,7 @@
     "myst_parser",
     "numpydoc",
     "nbsphinx",
+    "sphinx_gallery.gen_gallery",
 ]
 
 # Configure the myst parser to enable cool markdown features
@@ -74,6 +75,10 @@
     # to ensure that include files (partial pages) aren't built, exclude them
     # https://github.com/sphinx-doc/sphinx/issues/1965#issuecomment-124732907
     "**/includes/**",
+    # exclude .py and .ipynb files in auto_examples generated by sphinx-gallery
+    # this is to prevent sphinx from complaining about duplicate source files
+    "ephys_at_swc/example_pipelines/auto_examples_ephys/*.ipynb",
+    "ephys_at_swc/example_pipelines/auto_examples_ephys/*.py",
 ]
 
 # Ignore certain URLs from being checked
@@ -91,6 +96,7 @@
 linkcheck_anchors_ignore_for_url = [
     "https://gin.g-node.org/G-Node/Info/wiki",
     "https://gin.g-node.org/G-Node/info/wiki",  # ignore both spellings
+    "https://github.com/stephenlenzi/npix_lse",  # ignore private repository
 ]
 linkcheck_retries = 2
 
@@ -148,3 +154,9 @@
 # Configure the code block copy button
 # don't copy line numbers, prompts, or console outputs
 copybutton_exclude = ".linenos, .gp, .go"
+
+# Configure sphinx-gallery
+sphinx_gallery_conf = {
+     'examples_dirs': ["ephys_at_swc/example_pipelines/examples"],   # path to your example scripts
+     'gallery_dirs': ["ephys_at_swc/example_pipelines/auto_examples_ephys"],    # path to where to save gallery generated output
+}

docs/source/ephys_at_swc/community.md

@@ -0,0 +1,21 @@
+# Community
+
+This site is maintained by the
+[NeuroInformatics Unit](https://neuroinformatics.dev/).
+Please don't hesitate to contact us
+(in particular Joe Ziminski) on Slack with any questions or feedback.
+
+For information and advice on electrophysiology from the SWC community, the
+best place to go is the `#forum-extracellular-ephys` channel on the SWC Slack.
+
+Outside the SWC, you can address any questions or issues about
+[SpikeInterface](https://github.com/SpikeInterface)
+by raising an issue on their
+[GitHub repository](https://github.com/SpikeInterface/spikeinterface/issues).
+They are very friendly and
+happy to answer any questions!
+
+In addition, the
+[Neuropixels](https://neuropixelsgroup.slack.com/)
+Slack channel is a great resource, with an active community discussing
+extracellular electrophysiology acquisition and analysis.

docs/source/ephys_at_swc/example_pipelines/examples/README.rst

@@ -0,0 +1,39 @@
+:orphan:
+
+Python Pipelines
+================
+
+Multiple probes (Cambridge Neurotech)
+-------------------------------------
+
+Dammy Onih (Akrami lab) is running a multi-session paradigm with two
+`Cambridge Neurotech <https://www.cambridgeneurotech.com/neural-probes>`__
+probes. In this multi-session task, mice learn a statistical learning paradigm to a
+reward-associated auditory stimulus, recording from the hippocampus.
+The pipeline uses SpikeInterface for preprocessing,
+sorting, and analysis and can be found `here <https://github.com/AOONIH/ephys/tree/master>`__.
+
+The IBL analysis pipeline
+-------------------------
+
+Nate Miska (Mrsic-Flogel lab) is a member of the
+`International Brain Laboratory
+(IBL) <https://www.internationalbrainlab.com/>`_
+running the
+`IBL's standardised behavioural task <https://elifesciences.org/articles/63711>`_
+with acute Neuropixels 1.0 recordings. Details of the
+`analysis pipeline code <https://github.com/int-brain-lab/ibl-neuropixel>`__
+on the IBL data management system can be found
+`here <https://int-brain-lab.github.io/iblenv/index.html>`_.
+
+Integrated analysis with Neuropixels 2.0
+----------------------------------------
+
+Steve Lenzi (Margrie Lab) has an integrated pipeline for automated behavioural,
+electrophysiological, and anatomical analysis. Steve works with a
+multi-session escape paradigm, recording stimulus-responsive neurons in the posterior
+striatum. He is using NeuroPixels 2.0 and SpikeGLX with a SpikeInterface-based
+pipeline; the code can be found `here <https://github.com/stephenlenzi/npix_lse>`__.
+
+Python Scripts
+--------------

docs/source/ephys_at_swc/example_pipelines/examples/notes-on-docs-structure.txt

@@ -0,0 +1,5 @@
+Here, the 'Python Examples' page is rendered as a sphinx gallery. The
+first section is python examples that link to outside repos. The second
+section is the sphinx gallery entries. The 'README.rst' is the
+'Python Examples' page, it must be called 'README.rst' for sphinx-related
+reasons.