Code

ML DataLoaders#115

Code

Merged

al-rigazzi merged 125 commits intoCrayLabs:developCrayLabs/SmartSim:developfrom al-rigazzi:traineral-rigazzi/SmartSim:trainerCopy head branch name to clipboardFeb 3, 2022

+2,941-12

Commits

Commits on Jan 17, 2022

• First implementation of Keras loader and generator

  al-rigazzi

  committed
  d797589

  Copy full SHA for d797589

• Improve naming conventions

  al-rigazzi

  committed
  13ba12f

  Copy full SHA for 13ba12f

• Enhance TF generator flexibility

  al-rigazzi

  committed
  5fe08eb

  Copy full SHA for 5fe08eb

• Adjustments to TF API

  al-rigazzi

  committed
  252c981

  Copy full SHA for 252c981

• Add Torch dataset

  al-rigazzi

  committed
  3c83441

  Copy full SHA for 3c83441

• Add Pytorch Dataset

  al-rigazzi

  committed
  e2f96d8

  Copy full SHA for e2f96d8

• Add torch iterable and multiprocess load

  al-rigazzi

  committed
  78dddb7

  Copy full SHA for 78dddb7

• Add Horovod to Torch loader example

  al-rigazzi

  committed
  ff805b3

  Copy full SHA for ff805b3

• Unify data loader API

  al-rigazzi

  committed
  6d4cdc2

  Copy full SHA for 6d4cdc2

• Add superclass to ml loaders

  al-rigazzi

  committed
  1ed2bb5

  Copy full SHA for 1ed2bb5

• Added docstrings to base downloader class

  al-rigazzi

  committed
  b683bb2

  Copy full SHA for b683bb2

• Update ML docs and TF tutorial

  al-rigazzi

  committed
  ef00694

  Copy full SHA for ef00694

• Apply style

  al-rigazzi

  committed
  b74796e

  Copy full SHA for b74796e

• Update Torch ML training tutorial

  al-rigazzi

  committed
  4094333

  Copy full SHA for 4094333

• Add documentation of ML training

  al-rigazzi

  committed
  daade7c

  Copy full SHA for daade7c

• Add ML API to docs

  al-rigazzi

  committed
  19a58b9

  Copy full SHA for 19a58b9

• Fix typo in docs

  al-rigazzi

  committed
  74cd1a9

  Copy full SHA for 74cd1a9

• Add parameterized command line arguments

  al-rigazzi

  committed
  62fc65b

  Copy full SHA for 62fc65b

• Fix generator to accept multiple tags on same line

  al-rigazzi

  committed
  f212e25

  Copy full SHA for f212e25

• Apply style

  al-rigazzi

  committed
  fb1fd36

  Copy full SHA for fb1fd36

• Update comments of ensemble generation

  al-rigazzi

  committed
  92b07e6

  Copy full SHA for 92b07e6

• Include reviewer's suggestions

  al-rigazzi

  committed
  ee8827e

  Copy full SHA for ee8827e

• implement create_run_settings

  Show description for e41202c
  

  Sam Partee

  authored and

  al-rigazzi

  committed
  e41202c

  Copy full SHA for e41202c

• Split out create_run_settings

  Show description for 375bced
  

  Sam Partee

  authored and

  al-rigazzi

  committed
  375bced

  Copy full SHA for 375bced

• Migrate tests to use create_run_settings

  

  Sam Partee

  authored and

  al-rigazzi

  committed
  6c3d415

  Copy full SHA for 6c3d415

• Account for null case in create_run_settings

  Show description for fb5f3c2
  

  Sam Partee

  authored and

  al-rigazzi

  committed
  fb5f3c2

  Copy full SHA for fb5f3c2

• Format and make style

  

  Sam Partee

  authored and

  al-rigazzi

  committed
  05403f7

  Copy full SHA for 05403f7

• Remove pandas from experiment.py

  

  MattToast

  authored and

  al-rigazzi

  committed
  31af7ef

  Copy full SHA for 31af7ef

• Change table fmt to match pd.df, run black

  

  MattToast

  authored and

  al-rigazzi

  committed
  9c1547f

  Copy full SHA for 9c1547f

• ake style

  

  MattToast

  authored and

  al-rigazzi

  committed
  47dd489

  Copy full SHA for 47dd489

• refactor test to work with str from summary

  

  MattToast

  authored and

  al-rigazzi

  committed
  616e2b0

  Copy full SHA for 616e2b0

• add summary test to test_testexperiment.py, refactor test_simple_entity_launch.py summary test

  

  MattToast

  authored and

  al-rigazzi

  committed
  04602ae

  Copy full SHA for 04602ae

• Experiment.create_batch_settings

  Show description for cd6977f
  

  Sam Partee

  authored and

  al-rigazzi

  committed
  cd6977f

  Copy full SHA for cd6977f

• Add generic batch test with create_batch_settings

  Show description for 64699f4
  

  Sam Partee

  authored and

  al-rigazzi

  committed
  64699f4

  Copy full SHA for 64699f4

• Add LSF test to test_create_batch

  Show description for 59d97ff
  

  Sam Partee

  authored and

  al-rigazzi

  committed
  59d97ff

  Copy full SHA for 59d97ff

• Add format to summary as an optional parameter

  

  MattToast

  authored and

  al-rigazzi

  committed
  94e9a55

  Copy full SHA for 94e9a55

• add tabulate format link to docstring

  

  MattToast

  authored and

  al-rigazzi

  committed
  9f35503

  Copy full SHA for 9f35503

• Add missing functions to LSFSettings API

  al-rigazzi

  committed
  66e6727

  Copy full SHA for 66e6727

• Fix incorrect node setting for BsubBatchSettings

  al-rigazzi

  committed
  52e44e8

  Copy full SHA for 52e44e8

• Integrate new batch commands

  al-rigazzi

  committed
  b5e5621

  Copy full SHA for b5e5621

• Patched broken tests for LSF

  al-rigazzi

  committed
  34fb974

  Copy full SHA for 34fb974

• Unify batch settings creation

  al-rigazzi

  committed
  474d81c

  Copy full SHA for 474d81c

• Fix LSF problem with kwargs and project

  al-rigazzi

  committed
  f56c546

  Copy full SHA for f56c546

• Fix multiple account for LSF batch settings

  al-rigazzi

  committed
  d4194bc

  Copy full SHA for d4194bc

• Remove print statements

  al-rigazzi

  committed
  a94cb8f

  Copy full SHA for a94cb8f

• Add check of qstat output length in pbsParser

  al-rigazzi

  committed
  06e9e64

  Copy full SHA for 06e9e64

• Apply style

  al-rigazzi

  committed
  b0220f2

  Copy full SHA for b0220f2

• Migrate tests to use create_run_settings

  

  Sam Partee

  authored and

  al-rigazzi

  committed
  8689a69

  Copy full SHA for 8689a69

• Format and make style

  

  Sam Partee

  authored and

  al-rigazzi

  committed
  92bdbec

  Copy full SHA for 92bdbec

• Add missing functions to LSFSettings API

  al-rigazzi

  committed
  7033dfe

  Copy full SHA for 7033dfe

• Fix incorrect node setting for BsubBatchSettings

  al-rigazzi

  committed
  a10e94b

  Copy full SHA for a10e94b

• Add Generator Support for Directories (CrayLabs#88)

  Show description for bbf13b4
  

  MattToast

  authored and

  al-rigazzi

  committed
  bbf13b4

  Copy full SHA for bbf13b4

• Unify data loader API

  al-rigazzi

  committed
  70e20f7

  Copy full SHA for 70e20f7

• Apply style

  al-rigazzi

  committed
  b44e879

  Copy full SHA for b44e879

• Apply style

  al-rigazzi

  committed
  f99f210

  Copy full SHA for f99f210

• Migrate tests to use create_run_settings

  

  Sam Partee

  authored and

  al-rigazzi

  committed
  2bd460e

  Copy full SHA for 2bd460e

• Format and make style

  

  Sam Partee

  authored and

  al-rigazzi

  committed
  6739b7b

  Copy full SHA for 6739b7b

• Add missing functions to LSFSettings API

  al-rigazzi

  committed
  432c8ff

  Copy full SHA for 432c8ff

• Rebased and updated ML loader

  al-rigazzi

  committed
  282c44c

  Copy full SHA for 282c44c

• Apply style

  al-rigazzi

  committed
  b11208c

  Copy full SHA for b11208c

• Migrate tests to use create_run_settings

  

  Sam Partee

  authored and

  al-rigazzi

  committed
  03fd007

  Copy full SHA for 03fd007

• Format and make style

  

  Sam Partee

  authored and

  al-rigazzi

  committed
  c71d091

  Copy full SHA for c71d091

• Add missing functions to LSFSettings API

  al-rigazzi

  committed
  579632e

  Copy full SHA for 579632e

• Fix incorrect node setting for BsubBatchSettings

  al-rigazzi

  committed
  8913ccb

  Copy full SHA for 8913ccb

• Add Generator Support for Directories (CrayLabs#88)

  Show description for 57696db
  

  MattToast

  authored and

  al-rigazzi

  committed
  57696db

  Copy full SHA for 57696db

• Unify data loader API

  al-rigazzi

  committed
  9cf3e21

  Copy full SHA for 9cf3e21

• Apply style

  al-rigazzi

  committed
  4c86d86

  Copy full SHA for 4c86d86

• Apply style

  al-rigazzi

  committed
  e6ecf4e

  Copy full SHA for e6ecf4e

• Format and make style

  

  Sam Partee

  authored and

  al-rigazzi

  committed
  8c7e0f1

  Copy full SHA for 8c7e0f1

• Add missing functions to LSFSettings API

  al-rigazzi

  committed
  4d84ac4

  Copy full SHA for 4d84ac4

• Add missing functions to LSFSettings API

  al-rigazzi

  committed
  4d348f6

  Copy full SHA for 4d348f6

• Add Generator Support for Directories (CrayLabs#88)

  Show description for ee90aab
  

  MattToast

  authored and

  al-rigazzi

  committed
  ee90aab

  Copy full SHA for ee90aab

• Add parameterized command line arguments

  al-rigazzi

  committed
  0d7bf8d

  Copy full SHA for 0d7bf8d

• Fix generator to accept multiple tags on same line

  al-rigazzi

  committed
  4034163

  Copy full SHA for 4034163

• Apply style

  al-rigazzi

  committed
  ac8a437

  Copy full SHA for ac8a437

• Update comments of ensemble generation

  al-rigazzi

  committed
  6f72ba8

  Copy full SHA for 6f72ba8

• Include reviewer's suggestions

  al-rigazzi

  committed
  1349bec

  Copy full SHA for 1349bec

• implement create_run_settings

  Show description for c627ac4
  

  Sam Partee

  authored and

  al-rigazzi

  committed
  c627ac4

  Copy full SHA for c627ac4

• Split out create_run_settings

  Show description for f344f0b
  

  Sam Partee

  authored and

  al-rigazzi

  committed
  f344f0b

  Copy full SHA for f344f0b

• Migrate tests to use create_run_settings

  

  Sam Partee

  authored and

  al-rigazzi

  committed
  2015b68

  Copy full SHA for 2015b68

• Account for null case in create_run_settings

  Show description for 20d4fc5
  

  Sam Partee

  authored and

  al-rigazzi

  committed
  20d4fc5

  Copy full SHA for 20d4fc5

• Format and make style

  

  Sam Partee

  authored and

  al-rigazzi

  committed
  23ae580

  Copy full SHA for 23ae580

• Remove pandas from experiment.py

  

  MattToast

  authored and

  al-rigazzi

  committed
  77927a1

  Copy full SHA for 77927a1

• Change table fmt to match pd.df, run black

  

  MattToast

  authored and

  al-rigazzi

  committed
  1a90947

  Copy full SHA for 1a90947

• ake style

  

  MattToast

  authored and

  al-rigazzi

  committed
  9034f47

  Copy full SHA for 9034f47

• refactor test to work with str from summary

  

  MattToast

  authored and

  al-rigazzi

  committed
  72638ef

  Copy full SHA for 72638ef

• add summary test to test_testexperiment.py, refactor test_simple_entity_launch.py summary test

  

  MattToast

  authored and

  al-rigazzi

  committed
  8c3088a

  Copy full SHA for 8c3088a

• Experiment.create_batch_settings

  Show description for 268763d
  

  Sam Partee

  authored and

  al-rigazzi

  committed
  268763d

  Copy full SHA for 268763d

• Add generic batch test with create_batch_settings

  Show description for 5fcf288
  

  Sam Partee

  authored and

  al-rigazzi

  committed
  5fcf288

  Copy full SHA for 5fcf288

• Add LSF test to test_create_batch

  Show description for 3ea5e66
  

  Sam Partee

  authored and

  al-rigazzi

  committed
  3ea5e66

  Copy full SHA for 3ea5e66

• Add format to summary as an optional parameter

  

  MattToast

  authored and

  al-rigazzi

  committed
  d56dfe0

  Copy full SHA for d56dfe0

• add tabulate format link to docstring

  

  MattToast

  authored and

  al-rigazzi

  committed
  d59f7ab

  Copy full SHA for d59f7ab

• Add missing functions to LSFSettings API

  al-rigazzi

  committed
  0a2e127

  Copy full SHA for 0a2e127

• Fix incorrect node setting for BsubBatchSettings

  al-rigazzi

  committed
  f5b9845

  Copy full SHA for f5b9845

• Integrate new batch commands

  al-rigazzi

  committed
  e79751f

  Copy full SHA for e79751f

• Patched broken tests for LSF

  al-rigazzi

  committed
  3056241

  Copy full SHA for 3056241

• Unify batch settings creation

  al-rigazzi

  committed
  ead3673

  Copy full SHA for ead3673

• Fix LSF problem with kwargs and project

  al-rigazzi

  committed
  7a932e0

  Copy full SHA for 7a932e0

• Fix multiple account for LSF batch settings

  al-rigazzi

  committed
  31a543f

  Copy full SHA for 31a543f

• Remove print statements

  al-rigazzi

  committed
  5236867

  Copy full SHA for 5236867

• Add Generator Support for Directories (CrayLabs#88)

  Show description for d266b2d
  

  MattToast

  authored and

  al-rigazzi

  committed
  d266b2d

  Copy full SHA for d266b2d

• Remove redundant code from ml loaders

  al-rigazzi

  committed
  3517ae7

  Copy full SHA for 3517ae7

• Remove old code from ml.torch

  al-rigazzi

  committed
  8d6bfb4

  Copy full SHA for 8d6bfb4

• Rebase on develop

  al-rigazzi

  committed
  29ab033

  Copy full SHA for 29ab033

• Merge branch 'develop' into trainer

  al-rigazzi

  committed
  2072328

  Copy full SHA for 2072328

Commits on Jan 31, 2022

• Address reviewers' comment on code

  al-rigazzi

  committed
  a5bb69d

  Copy full SHA for a5bb69d

• Add workaround for old SmartRedis versions

  al-rigazzi

  committed
  cba90b9

  Copy full SHA for cba90b9

• Fix docstring

  al-rigazzi

  committed
  bdb687a

  Copy full SHA for bdb687a

• Add Build Matrix for CI (CrayLabs#130)

  Show description for b25fc23
  

  EricGustin

  authored and

  al-rigazzi

  committed
  b25fc23

  Copy full SHA for b25fc23

• Remove numpy as a dependency (CrayLabs#132)

  Show description for a2d1f89
  

  EricGustin

  authored and

  al-rigazzi

  committed
  a2d1f89

  Copy full SHA for a2d1f89

• Update Tutorials for Release (CrayLabs#132)

  Show description for 8284a22
  al-rigazzi

  committed
  8284a22

  Copy full SHA for 8284a22

• Docker Documentation Builds (CrayLabs#133)

  Show description for 2676c91
  

  Sam Partee

  authored and

  al-rigazzi

  committed
  2676c91

  Copy full SHA for 2676c91

• Update build_docs.yml

  

  Sam Partee

  authored and

  al-rigazzi

  committed
  c88ce3c

  Copy full SHA for c88ce3c

• Remove Duplicate Tutorials

  Show description for 34ca199
  

  MattToast

  authored and

  al-rigazzi

  committed
  34ca199

  Copy full SHA for 34ca199

• Update Tutorials for Release (CrayLabs#132)

  Show description for d3d1eb0
  

  MattToast

  authored and

  al-rigazzi

  committed
  d3d1eb0

  Copy full SHA for d3d1eb0

• Docker Documentation Builds (CrayLabs#133)

  Show description for 853e8c6
  

  Sam Partee

  authored and

  al-rigazzi

  committed
  853e8c6

  Copy full SHA for 853e8c6

• Update build_docs.yml

  

  Sam Partee

  authored and

  al-rigazzi

  committed
  b85619b

  Copy full SHA for b85619b

• Merge branch 'develop' into trainer

  al-rigazzi

  authored
  353269f

  Copy full SHA for 353269f

• Reduce torch nn size in dataloader test

  al-rigazzi

  committed
  fa440ef

  Copy full SHA for fa440ef

• Remove logging from test

  al-rigazzi

  committed
  a3f4a09

  Copy full SHA for a3f4a09

• Address reviewers' comments on documentation

  al-rigazzi

  committed
  dba0cac

  Copy full SHA for dba0cac

• Modify docstrings and adapt class names

  al-rigazzi

  committed
  3ad1f1a

  Copy full SHA for 3ad1f1a

• Update documentation

  al-rigazzi

  committed
  d96b7bb

  Copy full SHA for d96b7bb

• Use SmartSim logger

  al-rigazzi

  committed
  28c8933

  Copy full SHA for 28c8933

• Add deprecation warning and make import not fatal

  al-rigazzi

  committed
  2fd37d1

  Copy full SHA for 2fd37d1