Add jupyter notebooks (#3)

muellerflorian · web-flow · commit 38fdb68f927c · 2020-03-06T10:39:29.000+01:00
* Added notebooks

- updated docs
- added notebooks

* Remove workflows in favor of notebooks

* Update segment-cells-nuclei.ipynb

corrected typo

* [docs] add summary for how to use ImJoy

* [docs] improved overview

* corrected typo in index.md

* Added section for how to create single-channel images

* Update data.md

corrected typos

* changed imjoy ws
diff --git a/docs/create-fq-outlines.md b/docs/create-fq-outlines.md
diff --git a/docs/data.md b/docs/data.md
@@ -2,7 +2,8 @@
 
 ## Data organization
 We strongly recommend the following data-organization on which this workflow has been tested.
-1. Images are store as single-channel multi-z-stack tif files, e.g on tif per position and channel.
+
+1. Images are store as single-channel multi-z-stack tif files, e.g on tif per position and channel. If your data are not single-channel, see the section on how to split channels with [Fiji](fiji-split-channels.md).
 2. All raw 3D images are stored in a folder `acquisition`
 3. All analysis results are stored in subfolder `analysis`, where each analysis step has a separate subfolder.
 
diff --git a/docs/diverse-workflows-overview.md b/docs/diverse-workflows-overview.md
@@ -0,0 +1,7 @@
+
+Here we provide a description of less frequently used workflows.  
+
+* [**Split multi-channel images**](fiji-split-channels.md) to obain single-channel images for analysis. 
+* [**Convert segmentation results in FQ outlines**](create-fq-outlines.md) that can be used by the older Matlab version of FISH-quant. 
+
+Each workflow is documented in a dedicated section.
diff --git a/docs/fiji-split-channels.md b/docs/fiji-split-channels.md
@@ -0,0 +1,34 @@
+# Split channeles
+This workflow requires that each channel is stored as a separate z-stack. 
+If your images are stored as multi-channel z-stacks you have to split these images into 
+individual channels. This can be done with different software packages, below we describe some options.
+
+## Manual conversion with Fiji
+For this you need to install [Fiji](https://fiji.sc/).
+
+1.  Create a new folder for each multi-channel image.
+2.  Open image stack (`.dv`) in FIJI.
+3.  Split channels (FQ only supports mono-channel input images)
+    a. From menu: `Image` > `Color` > `Split channels`
+    a. Save each channels with a unique channel identifier, e.g. `C1-` or `DAPI_`.
+
+## Batch conversion with Fiji
+For this you need to 
+* install [Fiji](https://fiji.sc/).
+* Use the provided macro `MacroBatchSplitChannel.ijm`, provided in the folder `workflows`.
+  
+### Workflow of macro
+1. Scan recursively a specified folder for files ending in a specified suffix.
+2. Images will be opened, split in different channels and maximum intensity projections (MIP) created
+3. Results will be saved in a new folder
+      1. Macro will replace specified string in full file path by "acquisition", and created this folder to save the data. 
+      2. Will create additional subfolders 'MIP' to store the MIPs.
+
+
+### How to use the macro
+1.  By default your data has to be in a folder `multi_channel_stacks`
+0.  Open Fiji and drag the macro file in Fiji.
+0.  This will open the macro, here you can modify two parameters
+     * `suffix` (default `ome.tif`): last part of the file-name. Only files containing this string will be processed.
+     * `folder_img` (default `multi_channel_stacks`): part of the file-name that will be replaced by `acquisition` in order to save the results. 
+0.  Press the `Run` button and you will be asked to specify the folder that should be processed. 
diff --git a/docs/imjoy-overview.md b/docs/imjoy-overview.md
@@ -1,7 +1,7 @@
 
 We provide several ImJoy plugins to 
 
-* Pre-process 3D images
-* Perform segmentation 
+* [**Pre-process 3D images**](imjoy-pre-processing.md) to obain 2D images ready for segmentation. 
+* Perform [**segmentation**](imjoy-segmentation.md) of 2D images. 
 
 Each workflow is documented in a dedicated section.
diff --git a/docs/imjoy-pre-processing.md b/docs/imjoy-pre-processing.md
@@ -4,7 +4,7 @@ Segmentation is done on 2D images. In this step, 3D images are transformed into
 2D images by applying a projection. 
 
 Preprocessing is done with the ImJoy plugin `PreProcess`:  
-<a href="https://imjoy.io/#/app?w=cellpose&plugin=fish-quant/segmentation:PreProcess@stable&upgrade=1" target="_blank">**install from here.**</a>
+<a href="https://imjoy.io/#/app?w=fq-seg&plugin=fish-quant/segmentation:PreProcess@stable&upgrade=1" target="_blank">**install from here.**</a>
 
 1. Before running the plugin, you have to specify a few parameters. Note that you have to perform this 
    projection for each channel-type. This allows to use different projection methods for a channel. This can be set in the plugin interface, avaible after clicking on the arrow down next to the plugin name.
diff --git a/docs/imjoy-segmentation.md b/docs/imjoy-segmentation.md
@@ -2,7 +2,7 @@
 # Segmentation of cells and nuclei
 
 Preprocessing is done with the ImJoy plugin `SegmentCellsNuclei`: 
-<a href="https://imjoy.io/#/app?w=cellpose&plugin=fish-quant/segmentation:SegmentCellsNuclei@stable&upgrade=1" target="_blank">**install from here.**</a>
+<a href="https://imjoy.io/#/app?w=fq-seg&plugin=fish-quant/segmentation:SegmentCellsNuclei@stable&upgrade=1" target="_blank">**install from here.**</a>
 
 
 **Resizing to speed up prediction**: segmentation speed depends on the image size. In our experience, resizing the images
diff --git a/docs/index.md b/docs/index.md
@@ -1,11 +1,17 @@
 
 This repository provides wrapper code to use the generalist cell/nuclei segmentation package [**Cellpose**](https://github.com/mouseland/cellpose). 
 
+In the other **overview sections** below we describe
+
+* How to install the **tools** needed for this analysis. 
+* How to organize the **data**.
+
 Example of **nuclear segmentation**:
 ![segmentation__nuclei](img/segmentation__nuclei.png)
 
-
 Example of **cytoplasmic segmentation**:
 ![segmentation__cells](img/segmentation__cells.png)
 
 
+You can then find a **detailed description of each workflow** in dedicated sections accessible from the menu above.  
+
diff --git a/docs/tools.md b/docs/tools.md
@@ -1,18 +1,26 @@
 # Tools
-Here, we describe the different software packages that are used.
+Here, we describe the different software packages that are used, and how to install them.
 
-We provide for all analysis workflows dedicated **ImJoy plugins**, to faciliate their use. 
-The ImJoy plugins are build on Python and require that you connect to a Jupyter Notebook. 
-Once you followed the instructions below you can connect ImJoy to a Jupyter notebook with:
+* Most tasks are performed by **Python plugins running in ImJoy**, and described in the section [**ImJoy analysis**](imjoy-overview.md).
+* For less frequently tasks, we provide some explanations in  [**Diverse workflows**](Diverse-workflows.md).
 
-1. **Activate the environment**: `conda activate cellpose`
-2. **Start Jupyter notebook**: `jupyter notebook --NotebookApp.allow_origin='*' --no-browser`
-3. Copy the provided URL including the token, in ImJoy connect to this Jupyter notebook. 
+A detailed description of each analysis workflow is provided in the dedicated sections. 
 
-We also provide Python source code illustrating the basic usage in the folder `workflows`.
-
-A more detailed description of each workflow is provided in the dedicated sections. 
+## Summary
+Here, you find a brief summary of the installation steps detailed in the sections below.
 
+* The analysis workflows are provided as dedicated **ImJoy plugins**. 
+* Once the plugin are installed in ImJoy, you can access them in a dedicated **workspace** `fq-seg`. 
+* To use the plugins them you have to connect to the **Jupyter engine**
+    1. **Activate the environment**: 
+        ```
+        conda activate cellpose
+        ```
+    1. **Start Jupyter notebook**: 
+       ```
+       jupyter notebook --NotebookApp.allow_origin='*' --no-browser
+       ```
+    3. Copy the provided URL including the token. In ImJoy connect to this Jupyter notebook. 
 
 ## Installation
 In order to use the workflows in this repository, you need to follows these steps, which we detail below: 
@@ -40,10 +48,11 @@ conda activate cellpose
 
 ## ImJoy
 [**ImJoy**](https://imjoy.io/docs/#/) is image processing platform with an easy
- to use interface. ImJoy can be used directly in your browser, withoyt any prior installation. 
- While ImJoy is running in the browser, NO data will be transferred. 
+ to use interface. ImJoy can be used directly in your browser, without any prior installation. 
+
+ While ImJoy is an app running in the browser, **NO** user data will be transferred over the internet. 
  
- Some important features
+ Some important **features**:
 
  2. Specific functionality is provided by plugins, which can be installed with simple links. Available 
     plugins are listed in the plugin list on the left part of the interface. Depending on the implementation 
@@ -67,9 +76,10 @@ dialog asking if you want to install the specified plugin. To confirm, press the
 Once installed, ImJoy remembers the workspaces and plugins and you simply have to
 open the ImJoy app and select the workspace (which will be `cellpose`): [https://imjoy.io/#/app](https://imjoy.io/#/app)
 
-### Jupyter notebook for Python plugins 
-Some of the provided plugins use code written in Python. In order for ImJoy this code, it can connect 
-to a **Jupyter notebook**, which can be installed via Miniconda.
+### Running Python plugins 
+Most of the provided plugins use Python for the processing. In order for ImJoy these plugins, you have 
+to connect ImJoy to a **Jupyter notebook server**, which can be installed via Miniconda. Please note
+that this "server" can run on your local machine, so no data-transfer over the internet is taking place. 
     
 Once you have the computational environment set up (see Installation), you start an Jupyter Notebook, 
 to which ImJoy can connect: 
@@ -89,11 +99,12 @@ to which ImJoy can connect:
     next to the plugin name, and selecting the Juypyter Notebook as engine.  
 
 
-## Workflows
-We also provide the Python code that is called in the ImJoy plugins, as simple
-Python files in the folder `workflow`. 
+## Jupyter notebooks 
+We also illustrate the workflow in the ImJoy Python plugins in dedicated Jupyter notebooks, which can 
+be found in the folder `notebooks`. 
 
-To run this code, install this repository in a dedicated conda environment, e.g. with 
+To run this code, we recommend following the instructions above to create a dedicated conda environment, 
+and install the code of this plugin in this env, e.g. with 
 ```
 pip install git+https://github.com/muellerflorian/segmentation/ --upgrade
 ```
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -59,13 +59,15 @@ nav:
     - Tools: tools.md  # This is strange but only with this organization this file shows upd in the overview together with the index.md
     - Data: data.md  # This is strange but only with this organization this file shows upd in the overview together with the index.md
 
-    - Analysis with ImJoy:
+    - ImJoy Analysis:
       - Overview: imjoy-overview.md
       - Preprocessing: imjoy-pre-processing.md
       - Segmentation: imjoy-segmentation.md
 
-    - Post-processing:
-      - Create FQ outline files: post-processing-fq.md
+    - Diverse workflows:
+      - Overview: diverse-workflows-overview.md
+      - Split channels: fiji-split-channels.md
+      - Create FQ outline files: create-fq-outlines.md
 
     - Advanced:
       - Developer notes: developer-notes.md
diff --git a/notebooks/pre-processing.ipynb b/notebooks/pre-processing.ipynb
@@ -0,0 +1,72 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Notebook to perform preprocessing of images for segmentation\n",
+    "* Takes 3D images and performs a 2D projection to obtain 2D png that can be used in the segmentation algorithm.\n",
+    "* Requires that `cellpose` and the code of this respository are installed. One way to do this is with a pip install (recommended in a dedicated conda environment as explained in the documentation).\n",
+    "   ```\n",
+    "   pip install git+https://github.com/muellerflorian/segmentation/ --upgrade\n",
+    "   ```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# >>> Imports \n",
+    "from pathlib import Path\n",
+    "from segwrap import utils_segmentation  # Either on sys path or pip installed"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# >>> Function call\n",
+    "\n",
+    "# Parameters\n",
+    "path_process = Path(r'paste-path-to-data')       # For example data: \\example_data\\acquisition\n",
+    "path_save = Path(r'paste-path-to-save-results')  # For example data: example_data\\analysis\\segmentation-input\n",
+    "channel_ident = 'dapi'                           # Identifier of channel that should be pre-processed \n",
+    "projection_type = 'max'                          # Projection type (mean, max, indiv)\n",
+    "\n",
+    "# Call pre-processing function\n",
+    "utils_segmentation.folder_prepare_prediction(\n",
+    "                    path_process=path_process,\n",
+    "                    search_type=None,\n",
+    "                    channel_ident=channel_ident,\n",
+    "                    projection_type=projection_type,\n",
+    "                    path_save=path_save,\n",
+    "                    callback_log=None)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3.7.6 64-bit ('cellpose': conda)",
+   "language": "python",
+   "name": "python37664bitcellposecondafe33b8fac7d34fcb85d1f0e07041e8c2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/notebooks/segment-cells-nuclei.ipynb b/notebooks/segment-cells-nuclei.ipynb
@@ -0,0 +1,76 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Notebook to perform segmentation of cells and nuclei with cellpose.\n",
+    "* Requires that `cellpose` and the code of this respository are installed. One way to do this is with a pip install (recommended in a dedicated conda environment as explained in the documentation).\n",
+    "   ```\n",
+    "   pip install git+https://github.com/muellerflorian/segmentation/ --upgrade\n",
+    "   ```"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Imports \n",
+    "from pathlib import Path\n",
+    "import utils_cellpose  # Either on sys path or pip installed"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# >> Function call\n",
+    "\n",
+    "# Parameters\n",
+    "path_scan = Path(r'paste-path-to-data')            # For example data: example_data\\analysis\\segmentation-input\n",
+    "path_save = Path(r'paste-path-to-save-results')    # For example data: example_data\\analysis\\segmentation-results\n",
+    "str_cyto = 'cy3'                                   # Identifier of channel for cytoplasmic segmentation\n",
+    "str_nuclei = 'dapi'                                # Identifier of channel for nuclear segmentation\n",
+    "img_ext = '.png'                                   # Extension of images to be segmented\n",
+    "new_size = (512, 512)                              # Size of images (when resize should be applied, empty tuple otherwise)\n",
+    "\n",
+    "size_cells = 100                                   # Typical size (diameter) of cells\n",
+    "size_nuclei = 50                                   # Typical size (diameter) of nuclei\n",
+    "\n",
+    "# Call segmentation function\n",
+    "utils_cellpose.segment_cells_nuclei_indiv(path_scan=path_scan, \n",
+    "                                strings=(str_cyto, str_nuclei), \n",
+    "                                img_ext=img_ext, \n",
+    "                                new_size=new_size,\n",
+    "                                sizes=(size_cells, size_nuclei), \n",
+    "                                models=('cyto','nuclei'),\n",
+    "                                path_save=path_save)"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3.7.6 64-bit ('cellpose': conda)",
+   "language": "python",
+   "name": "python37664bitcellposecondafe33b8fac7d34fcb85d1f0e07041e8c2"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/readme.md b/readme.md
@@ -20,8 +20,7 @@ Example of **cytoplasmic segmentation**:
 
 # Licensing
  
-
-- **Cellpose** is (general) are BSD-licenced (3 clause)
+- **Cellpose** is BSD-licenced (3 clause)
 
 > Copyright © 2020 Howard Hughes Medical Institute
 >
diff --git a/workflows/MacroBatchSplitChannel.ijm b/workflows/MacroBatchSplitChannel.ijm
diff --git a/workflows/preprocessing.py b/workflows/preprocessing.py
diff --git a/workflows/segment_cells_nuclei.py b/workflows/segment_cells_nuclei.py

Original file line number	Diff line number	Diff line change
`@@ -20,8 +20,7 @@ Example of cytoplasmic segmentation:`
`20`	`20`
`21`	`21`	`# Licensing`
`22`	`22`
`23`		`-`
`24`		`-- Cellpose is (general) are BSD-licenced (3 clause)`
	`23`	`+- Cellpose is BSD-licenced (3 clause)`
`25`	`24`
`26`	`25`	`> Copyright © 2020 Howard Hughes Medical Institute`
`27`	`26`	`>`