ISC 2024

.gitignore

@@ -5,3 +5,4 @@
 *.pyo
 *__pycache__*
 *.ipynb_checkpoints*
+environ

.scripts/requirements/python-basics-requirements.txt

@@ -1,3 +1,3 @@
 # These should match the versions in the version of jaspy which will be used for the course.
-matplotlib==3.5.3
-pandas==2.0.3
+matplotlib==3.8.4
+pandas==2.2.2

README.md

@@ -1,26 +1,6 @@
-# Introduction to Scientific Computing course  
+# Introduction to Scientific Computing course
 This repository holds teaching materials for the NCAS Introduction to Scientific Computing course.  
 
-## Overview  
-
-The course covers:  
-- Introduction to the Linux shell
-  - [Presentations and Exercises](https://ncasuk.github.io/ncas-isc-shell/)
-- Python Setup
-  - [Logging in to the JASMIN Notebook Service ](https://github.com/ncasuk/ncas-isc/blob/main/setup/Logging_in_to_the_JASMIN_Notebook_Service.pdf)
-- Git and GitHub
-  - [Presentation](https://github.com/ncasuk/ncas-isc/tree/main/version_control)  
-  - [Exercise](https://github.com/ncasuk/ncas-isc/tree/main/version_control)
-- Introduction to Python
-  - [Python Introduction Slides](https://github.com/ncasuk/ncas-isc/blob/main/python-intro/presentations.md) 
-  - [Exercises - Jupyter Notebooks](https://github.com/ncasuk/ncas-isc/tree/main/python-intro/exercises) | [Solutions](https://github.com/ncasuk/ncas-isc/tree/main/python-intro/solutions) 
-- Data manipulation and visualisation in Python (Working with Data)
-  - [Python working with data Slides](https://github.com/ncasuk/ncas-isc/tree/main/python-data/slides) 
-  - [Exercises - Jupyter Notebooks](https://github.com/ncasuk/ncas-isc/tree/main/python-data/notebooks) | [Solutions](https://github.com/ncasuk/ncas-isc/tree/main/python-data/solutions)
-- Example code for all python modules
-  - [Example code for Python (Working with Data)](https://github.com/ncasuk/ncas-isc/tree/main/python-data/example_code)
-  - [Example data for Python (Working with Data)](https://github.com/ncasuk/ncas-isc/tree/main/python-data/example_data)
-
 ## Index  
 ### Overview Presentations  
 * [Algorithmic thinking](https://github.com/ncasuk/ncas-isc/blob/main/working_practices/Algorithmic_thinking.pdf)  
@@ -37,50 +17,41 @@ The course covers:
 * [Exercise and solutions](https://github.com/ncasuk/ncas-isc/blob/main/version_control/01_git_exercise.md) 
 
 ## Introduction to Python
-1. [Running and Quitting](http://swcarpentry.github.io/python-novice-gapminder/01-run-quit.html)  
-2. [Variables and Assignment](http://swcarpentry.github.io/python-novice-gapminder/02-variables.html)  
-3. [Data Types and Type Conversion](http://swcarpentry.github.io/python-novice-gapminder/03-types-conversion.html)  
-4. [Built-in Functions and Help](http://swcarpentry.github.io/python-novice-gapminder/04-built-in.html)
-5. [Libraries](http://swcarpentry.github.io/python-novice-gapminder/06-libraries.html)  
-6. [Reading Tabular Data into DataFrames](http://swcarpentry.github.io/python-novice-gapminder/07-reading-tabular.html)
-7. [Pandas DataFrames](http://swcarpentry.github.io/python-novice-gapminder/08-data-frames.html)  
-8. [Plotting](http://swcarpentry.github.io/python-novice-gapminder/09-plotting.html)   
-9. [Lists](http://swcarpentry.github.io/python-novice-gapminder/11-lists.html)  
-10. [For Loops](http://swcarpentry.github.io/python-novice-gapminder/12-for-loops.html)  
-11. [Conditionals](http://swcarpentry.github.io/python-novice-gapminder/13-conditionals.html)
-12. [Looping Over Data Sets](http://swcarpentry.github.io/python-novice-gapminder/14-looping-data-sets.html)
-13. [Writing Functions](http://swcarpentry.github.io/python-novice-gapminder/16-writing-functions.html)
-14. [Variable Scope](http://swcarpentry.github.io/python-novice-gapminder/17-scope.html)
-15. [Programming Style](http://swcarpentry.github.io/python-novice-gapminder/18-style.html)
 
-* [Exercises](https://github.com/ncasuk/ncas-isc/blob/main/python-intro/notebooks) and [Solutions](https://github.com/ncasuk/ncas-isc/blob/main/python-intro/solutions)
+| Lesson | Exercise | Solution |
+| ------ | -------- | -------- |
+| [Running and quitting](https://swcarpentry.github.io/python-novice-gapminder/01-run-quit.html) | [Exercise 01](/python-intro/exercises/ex01_running_notebooks.ipynb) | [Solution 01](/python-intro/solutions/ex01_running_notebooks.ipynb) |
+| [Variables and assignment](https://swcarpentry.github.io/python-novice-gapminder/02-variables.html) | [Exercise 02](/python-intro/exercises/ex02_variables_assignment.ipynb) | [Solution 02](/python-intro/solutions/ex02_variables_assignment.ipynb) |
+| [Data types and type conversion](https://swcarpentry.github.io/python-novice-gapminder/03-types-conversion.html) | [Exercise 03](/python-intro/exercises/ex03_data_types.ipynb) | [Solution 03](/python-intro/solutions/ex03_data_types.ipynb) |
+| [Built-in functions and Help](https://swcarpentry.github.io/python-novice-gapminder/04-built-in.html) | [Exercise 04](/python-intro/exercises/ex04_built_in_functions.ipynb) | [Solution 04](/python-intro/solutions/ex04_built_in_functions.ipynb) |
+| [Libraries](https://swcarpentry.github.io/python-novice-gapminder/06-libraries.html) | [Exercise 05](/python-intro/exercises/ex05_libraries.ipynb) | [Solution 05](/python-intro/solutions/ex05_libraries.ipynb) |
+| [Reading tabular data into data frames](https://swcarpentry.github.io/python-novice-gapminder/07-reading-tabular.html) | [Exercise 06](/python-intro/exercises/ex06_dataframes.ipynb) | [Solution 06](/python-intro/solutions/ex06_dataframes.ipynb) |
+| [Pandas data frames](https://swcarpentry.github.io/python-novice-gapminder/08-data-frames.html) | [Exercise 07](/python-intro/exercises/ex07_pandas_dataframes.ipynb) | [Solution 07](/python-intro/solutions/ex07_pandas_dataframes.ipynb) |
+| [Plotting](https://swcarpentry.github.io/python-novice-gapminder/09-plotting.html) | [Exercise 08](/python-intro/exercises/ex08_plotting.ipynb) | [Solution 08](/python-intro/solutions/ex08_plotting.ipynb) |
+| [Lists](https://swcarpentry.github.io/python-novice-gapminder/11-lists.html) | [Exercise 09](/python-intro/exercises/ex09_lists.ipynb) | [Solution 09](/python-intro/solutions/ex09_lists.ipynb) |
+| [For loops](https://swcarpentry.github.io/python-novice-gapminder/12-for-loops.html) | [Exercise 10](/python-intro/exercises/ex10_for_loops.ipynb) | [Solution 10](/python-intro/solutions/ex10_for_loops.ipynb) |
+| [Conditionals](https://swcarpentry.github.io/python-novice-gapminder/13-conditionals.html) | [Exercise 11](/python-intro/exercises/ex11_conditionals.ipynb) | [Solution 11](/python-intro/solutions/ex11_conditionals.ipynb) |
+| [Looping over data sets](https://swcarpentry.github.io/python-novice-gapminder/14-looping-data-sets.html) | [Exercise 12](/python-intro/exercises/ex12_looping_data_sets.ipynb) | [Solution 12](/python-intro/solutions/ex12_looping_data_sets.ipynb) |
+| [Writing functions](https://swcarpentry.github.io/python-novice-gapminder/16-writing-functions.html) | [Exercise 13](/python-intro/exercises/ex13_writing_functions.ipynb) | [Solution 13](/python-intro/solutions/ex13_writing_functions.ipynb) |
+| [Variable scope](https://swcarpentry.github.io/python-novice-gapminder/17-scope.html) | [Exercise 14](/python-intro/exercises/ex14_variable_scope.ipynb) | [Solution 14](/python-intro/solutions/ex14_variable_scope.ipynb) |
+| [Programming style](https://swcarpentry.github.io/python-novice-gapminder/18-style.html) | [Exercise 15](/python-intro/exercises/ex15_programming_style.ipynb) | [Solution 15](/python-intro/solutions/ex15_programming_style.ipynb) |
+| [Wrap Up / Summary](/python-intro/exercises/ex16_wrap_up.ipynb) |
 
 ## Python - Working with Data
 
-### Handling arrays  
-* [Numpy](https://github.com/ncasuk/ncas-isc/blob/main/python-data/slides/numpy.pdf)  
-* [Exercises](https://github.com/ncasuk/ncas-isc/blob/main/python-data/notebooks/ex01_numpy_arrays.ipynb) and [Solutions](https://github.com/ncasuk/ncas-isc/blob/main/python-data/solutions/ex01_numpy_arrays_solutions.ipynb)
-
-### Visualisation  
-* [Matplotlib and cartopy](https://github.com/ncasuk/ncas-isc/blob/main/python-data/slides/matplotlib_and_cartopy.pdf)  
-* [Exercises](https://github.com/ncasuk/ncas-isc/blob/main/python-data/notebooks/ex02_matplotlib.ipynb) and [Solutions](https://github.com/ncasuk/ncas-isc/blob/main/python-data/solutions/ex02_matplotlib_solutions.ipynb)
+| Lesson | Exercise | Solution |
+| ------ | -------- | -------- |
+| __xarray:__ Introduction to [multidimensional arrays](https://tutorial.xarray.dev/fundamentals/01_data_structures.html), [xarray data structures](https://tutorial.xarray.dev/fundamentals/01_datastructures.html) and [indexing](https://tutorial.xarray.dev/fundamentals/02.1_indexing_Basic.html) | [Exercise 01](/python-data/exercises/ex01_xr_intro.ipynb) [Exercise 01.5](/python-data/exercises/ex01.5_xr_label_based_indexing.ipynb)| [Solution 01](/python-data/solutions/ex01_xarray_intro.ipynb) [Solution 01.5](/python-data/solutions/ex01.5_xr_label_based_indexing.ipynb)|
+| __xarray:__ [Plotting](https://tutorial.xarray.dev/fundamentals/04.1_basic_plotting.html) and [Aggregation](https://tutorial.xarray.dev/fundamentals/03.1_computation_with_xarray.html) | [Exercise 02](/python-data/exercises/ex02_xr_plotting.ipynb) [Exercise 02.5](/python-data/exercises/ex02.5_xr_aggregation.ipynb)| [Solution 02](/python-data/solutions/ex02_plotting.ipynb) [Solution 02.5](/python-data/solutions/ex02.5_xr_aggregation.ipynb)|
+| __xarray:__ [GroupBy processing](https://tutorial.xarray.dev/fundamentals/03.2_groupby_with_xarray.html) and [masking](https://tutorial.xarray.dev/intermediate/indexing/boolean-masking-indexing.html) | [Exercise 03](/python-data/exercises/ex03_xr_groupby.ipynb) [Exercise 03.5](/python-data/exercises/ex03.5_xr_masking.ipynb)| [Solution 03](/python-data/solutions/ex03_groupby.ipynb) [Solution 03.5](/python-data/solutions/ex03.5_masking.ipynb)|
+| [cf-python]() | [Exercise 04](/python-data/exercises/ex04_cf_python.ipynb) | [Solution 04](/python-data/solutions/ex04_cf_python.ipynb) |
+| [matplotlib](https://matplotlib.org/stable/users/explain/quick_start.html) | [Exercise 05](/python-data/exercises/ex05_matplotlib.ipynb) | [Solution 05](/python-data/solutions/ex05_matplotlib.ipynb) |
+| [numpy](https://numpy.org/doc/stable/user/quickstart.html) | [Exercise 06](/python-data/exercises/ex06_numpy.ipynb) | [Solution 06](/python-data/solutions/ex06_numpy.ipynb) |
+| [netCDF4 basics](https://unidata.github.io/netcdf4-python/#tutorial) | [Exercise 07](/python-data/exercises/ex07_netcdf4_basics.ipynb) | [Solution 07](/python-data/solutions/ex07_netcdf4_basics.ipynb) | 
+| [netCDF advanced](https://unidata.github.io/netcdf4-python/#dealing-with-time-coordinates) | [Exercise 08](/python-data/exercises/ex08_netcdf4_advanced.ipynb) | [Exercise 08](/python-data/solutions/ex08_netcdf4_advanced.ipynb) |
+| Weather Exercise | [Exercise 09a](/python-data/exercises/ex09a_weather_api.ipynb) | [Solution 09b](/python-data/solutions/ex09a_weather_api.ipynb) |
+| Sentinel Data Exercise | [Exercise 09b](/python-data/exercises/ex09b_satellite_data.ipynb) | [Solution 09b](/python-data/solutions/ex09b_satellite_data.ipynb) |
 
-### Read and Write data  
-1. [Data formats and metadata - why?](https://github.com/ncasuk/ncas-isc/blob/main/python-data/slides/01_data_formats.pdf)  
-2. [Text formats](https://github.com/ncasuk/ncas-isc/blob/main/python-data/slides/02_python_text_formats.pdf)  
-3. [Some more common text formats (at CEDA)](https://github.com/ncasuk/ncas-isc/blob/main/python-data/slides/03_text_formats_ceda.pdf)  
-4. [Binary formats](https://github.com/ncasuk/ncas-isc/blob/main/python-data/slides/04_binary_formats.pdf)  
-5. [Overview of NetCDF](https://github.com/ncasuk/ncas-isc/blob/main/python-data/slides/05_netcdf_overview.pdf)  
-6. [The structure of "Classic" NetCDF files](https://github.com/ncasuk/ncas-isc/blob/main/python-data/slides/06_netcdf_structure.pdf)  
-7. [`ncgen` and `ncdump` to create/export NetCDF and CDL](https://github.com/ncasuk/ncas-isc/blob/main/python-data/slides/07_ncgen_ncdump_cdl.pdf)  
-8. [The CF Metadata Conventions (for NetCDF)](https://github.com/ncasuk/ncas-isc/blob/main/python-data/slides/08_cf_metadata_conventions.pdf)  
-9. [Checking CF-compliance: `cf-checker`](https://github.com/ncasuk/ncas-isc/blob/main/python-data/slides/09_cfchecker.pdf)  
-10. [Reading NetCDF files with Python: `netCDF4`](https://github.com/ncasuk/ncas-isc/blob/main/python-data/slides/10_read_netcdf_python.pdf)  
-11. [Creating NetCDF files with Python](https://github.com/ncasuk/ncas-isc/blob/main/python-data/slides/11_create_netcdf_python.pdf)  
-12. [Reading and writing other formats](https://github.com/ncasuk/ncas-isc/blob/main/python-data/slides/12_python_and_other_formats.pdf)  
-13. [Viewing NetCDF: `Ncview` and `ncBrowse`](https://github.com/ncasuk/ncas-isc/blob/main/python-data/slides/13_ncview_ncbrowse.pdf)  
-* [Exercises](https://github.com/ncasuk/ncas-isc/blob/main/python-data/notebooks/ex03_netcdf.ipynb) and [Solutions](https://github.com/ncasuk/ncas-isc/blob/main/python-data/solutions/ex03_netcdf_solutions.ipynb)
-* [Weather API Exercise](https://github.com/ncasuk/ncas-isc/blob/main/python-data/notebooks/ex04_weather_api.ipynb) and [Solution](https://github.com/ncasuk/ncas-isc/blob/main/python-data/solutions/ex04_weather_api_solutions.ipynb)
 
 ## Useful materials and resources
 
@@ -91,4 +62,4 @@ See the [Resources page](resources.md) for links to useful related sites and mat
 Feel free to fork this repository on GitHub and re-use these materials however you like.  
 
 ### Acknowledgements  
-The foundations of our course are based on the superb materials provided by [Software Carpentry](https://software-carpentry.org/) who we are eternally grateful to.  
+The foundations of our course are based on the superb materials provided by [Software Carpentry](https://software-carpentry.org/) who we are eternally grateful to. 

python-data/notebooks/ex01_numpy_arrays.ipynb → old_material/data_old_materials/notebooks/ex01_numpy_arrays.ipynb

@@ -53,7 +53,7 @@
    "source": [
     "### Let's create a numpy array from a list.\n",
     "\n",
-    "Create a with values 1 to 10 and assign it to the variable `x`"
+    "Create a range with values 1 to 10 and assign it to the variable `x`"
    ]
   },
   {
@@ -179,7 +179,7 @@
     "Create an array from the list `[2, 3.2, 5.5, -6.4, -2.2, 2.4]` and assign it to the variable `a`\n",
     "\n",
     "- Do you know what `a[1]` will equal? Print to see.\n",
-    "- Try print `a[1:4]` to see what that equals."
+    "- Try printing `a[1:4]` to see what that equals."
    ]
   },
   {
@@ -261,7 +261,7 @@
     "\n",
     "### Let's interrogate an array to find out it's characteristics\n",
     "\n",
-    "Create a 2-D array of shape (2, 4) containing two lists `range(4)` and `range(10, 14)`, assign it to the vairable `arr`\n",
+    "Create a 2-D array of shape (2, 4) containing two lists `range(4)` and `range(10, 14)`, assign it to the variable `arr`\n",
     "\n",
     "- Print the shape of the array\n",
     "- Print the size of the array\n",
@@ -342,7 +342,7 @@
     "\n",
     "### Let's perform some array calculations\n",
     "\n",
-    "Create a 2-D array of shape (2, 4) containing two lists `range(4)` and `range(10, 14)`, assign it to the vairable `a`\n",
+    "Create a 2-D array of shape (2, 4) containing two lists `range(4)` and `range(10, 14)`, assign it to the variable `a`\n",
     "\n",
     "Create an array from a list `[2, -1, 1, 0]` and assign it to the variable `b`\n",
     "\n",
@@ -385,7 +385,7 @@
     "\n",
     "Create an array of values 0 to 9 and assign it to the variable `arr`\n",
     "\n",
-    "- Print two different way of expressing the condition where the array is less than 3.\n",
+    "- Print two different ways of expressing the condition where the array is less than 3.\n",
     "- Create a numpy condition where `arr` is less than 3 OR greater than 8.\n",
     "- Use the `where` function to create a new array where the value is `arr*5` if the above condition is `True` and `arr-5` where the condition is `False`"
    ]
@@ -418,7 +418,7 @@
    "source": [
     "### Let's implement a mathematical function that works on arrays.\n",
     "\n",
-    "Write a function that takes a 2-D array of horizontal zonal (east-west) wind components (`u`, in m/s) and a 2-D array of horizontal meridional (north-south) wind componenets (`v`, in m/s)\n",
+    "Write a function that takes a 2-D array of horizontal zonal (east-west) wind components (`u`, in m/s) and a 2-D array of horizontal meridional (north-south) wind components (`v`, in m/s)\n",
     "and returns an array of the magnitudes of the total wind.\n",
     "Include a test for the overall magnitude: if it is less than 0.1 then set it equal to 0.1 (We might presume this particular domain has no non-zero winds and that only winds above 0.1 m/s constitute \"good\" data while those below are indistinguishable from the minimum due to noise)\n",
     "\n",
@@ -534,7 +534,7 @@
     "- Print the array to view its values.\n",
     "- Print its missing value (i.e. `narr.fill_value`)\n",
     "- Print an array that converts `narr` so that the missing values are represented by the missing value (i.e. `MA.filled`). Assign it to `farr`\n",
-    "- What is the type of the `farr`"
+    "- What is the type of `farr`?"
    ]
   },
   {

python-data/notebooks/ex02_matplotlib.ipynb → old_material/data_old_materials/notebooks/ex02_matplotlib.ipynb

@@ -237,7 +237,7 @@
    "source": [
     "## 3. Plotting gridded data on a map\n",
     "\n",
-    "In this section, we will use `cartopy` - a python module that supports maps and usage with `matplotlib`.\n",
+    "In this section, we will use `cartopy` - a Python module that supports maps and usage with `matplotlib`.\n",
     "\n",
     "First, let's grab some data from a NetCDF file and quickly plot it.\n",
     "\n",

python-data/notebooks/ex03_netcdf.ipynb → old_material/data_old_materials/notebooks/ex03_netcdf.ipynb

@@ -70,7 +70,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Loop through and print Dataset `variables` names, this is the ID that act like the key to access the Variable."
+    "Loop through and print Dataset `variables` names, this is the ID that acts like the key to access the variable."
    ]
   },
   {
@@ -314,7 +314,7 @@
    "metadata": {},
    "source": [
     "Create four new Dimensions to `myds` from your previous slices. Re-use the names from the last section.\n",
-    "Note that the \"level\" and \"time\" Dimensions should have length, \"lat\" length 10 and \"lon\" length 5.\n",
+    "Note that the \"level\" and \"time\" dimensions should have length 1, \"lat\" length 10 and \"lon\" length 5.\n",
     "To create a new Dimension use `myds.createDimension(name, size)`"
    ]
   },