Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration Test class for Stratified W7 setup. #558

Merged
merged 13 commits into from
May 12, 2016
3 changes: 3 additions & 0 deletions tardis/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ def pytest_addoption(parser):
parser.addoption("--slow", action="store_true",
help="include running slow tests during run")

parser.addoption("--slow-test-data", dest="slow-test-data",
help="path to directory having baseline data for slow tests")


def pytest_report_header(config):

Expand Down
6 changes: 5 additions & 1 deletion tardis/tests/coveragerc
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,8 @@ exclude_lines =
def main\(.*\):

# Ignore branches that don't pertain to this version of Python
pragma: py{ignore_python_version}
pragma: py{ignore_python_version}

omit =
# Slow Tests should be run but need not appear in "html report"
tests_slow
4 changes: 3 additions & 1 deletion tardis/tests/setup_package.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
def get_package_data():
return {
_ASTROPY_PACKAGE_NAME_ + '.tests': ['coveragerc', 'data/*.h5',
'data/*.dat', 'data/*.npy']}
'data/*.dat', 'data/*.npy',
'tests_slow/w7/*.yml',
'tests_slow/w7/*.dat']}
Empty file.
175 changes: 175 additions & 0 deletions tardis/tests/tests_slow/test_w7.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
import os
import yaml
import h5py
import numpy as np
import pytest
from numpy.testing import assert_allclose
from astropy import units as u

from tardis.simulation.base import Simulation
from tardis.model import Radial1DModel
from tardis.io.config_reader import Configuration


def data_path(fname):
return os.path.join(os.path.dirname(os.path.realpath(__file__)), "w7", fname)


@pytest.mark.skipif(not pytest.config.getoption("--slow"),
reason="slow tests can only be run using --slow")
@pytest.mark.skipif(not pytest.config.getvalue("slow-test-data"),
reason="--slow-test-data was not specified")
@pytest.mark.skipif(not pytest.config.getvalue("atomic-dataset"),
reason="--atomic-dataset was not specified")
class TestW7(object):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can likely be subclassed.

"""
Slow integration test for Stratified W7 setup.

Assumed two compressed binaries (.npz) are placed in `slow-test-data/w7`
directory, whose path is provided by command line argument:

* ndarrays.npz | * quantities.npz
Contents (all (.npy)): | Contents (all (.npy)):
* last_interaction_type | * t_rads
* last_line_interaction_out_id | * luminosity_inner
* last_line_interaction_in_id | * montecarlo_luminosity
* j_estimators | * montecarlo_virtual_luminousity
* j_blue_estimators | * time_of_simulation
* last_line_interaction_shell_id | * montecarlo_nu
* nubar_estimators | * last_line_interaction_angstrom
* ws | * j_blues_norm_factor
"""
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docstring for explanation related to work in progress.


@classmethod
@pytest.fixture(scope="class", autouse=True)
def setup(self):
"""
This method does initial setup of creating configuration and performing
a single run of integration test.
"""
self.config_file = data_path("config_w7.yml")
self.abundances = data_path("abundancies_w7.dat")
self.densities = data_path("densities_w7.dat")

# First we check whether atom data file exists at desired path.
self.atom_data_filename = os.path.expanduser(os.path.expandvars(
pytest.config.getvalue('atomic-dataset')))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may want to add a check that the atomic database is the same as used for the baseline data. Otherwise, this test most likely won't pass. Not sure if we have to include this check in the current PR, but something to keep in mind, though. Thoughts, @wkerzendorf, @karandesai-96, @yeganer?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This didn't even come to my mind ! Using a different atomic database would generate different results. So far there is kurucz_cd23_chianti_H_He.h5 everywhere. If this check is added when need arrives, then the purpose of adding that additional check would be more clear and could be specified in PR description more effectively. Waiting for others' opinions 😄

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you just check the uuid of the atomic data set - that would be good.

assert os.path.exists(self.atom_data_filename), \
"{0} atom data file does not exist".format(self.atom_data_filename)

# We now check that the baseline data for slow tests was obtained using
# the same atomic dataset as the one used in current test run.
# TODO: This is a quick workaround, generalize it.

kurucz_data_file_uuid = "5ca3035ca8b311e3bb684437e69d75d7"
Copy link
Contributor

@unoebauer unoebauer May 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not too happy about this quick workaround. You can directly access the UUID and the MD5 checksum of the atomic data file via model.atom_data.uuid1 and model.atom_data.md5. I suggest you add these to your baseline data and then check them against the one of the test run.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, so after the run, I obtain the radial1d_model and that would enable comparison, right ? But wouldn't it be a bit time expensive ? Waiting for twenty minutes or more and finally seeing that the test failed because atom data files weren't same ? Is it fine ?

Copy link
Author

@kdexd kdexd May 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest you add these to your baseline data and then check them against the one of the test run.

Add them in the sense, like a json or yml file in the directory containing baseline data for slow tests, right ?

atom_data:
    name: kurucz_cd23_chianti_H_He.h5
    uuid1: <uuid1>
    md5: <md5>

I think this would conflict with --atomic-dataset argument. In that case we can give the yml file a priority and NOT skip this test if --atomic-dataset is explicitly not provided. Does it sound like a plan ?
@wkerzendorf @yeganer Your thoughts ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@unoebauer I suggested this quick workaround. I agree that in the long run we want to have this done properly, but my suggestion is to build parts of the framework and then refine. Would that work for you?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wkerzendorf, @karandesai-96 - sure thing. Sorry for holding up the process. Just wanted to flag this again.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay if we not do apply this change, though please do let me know about this idea of putting in a yml file like this. If it is good enough, I would keep it in my mind and apply it immediately when required 😉

Copy link
Contributor

@yeganer yeganer May 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't have to wait for the run to be completed to access AtomData.
Afaik Atomdata is already an object inside tardis_config(the parsed, validated one) so doing something like tardis_config.atom_data.uuid should work.

About storing the reference: I think hardcoding it into the tests is fine for now, as there won't be any other database in the near future and this is just a simple implementation detail which can be changed anytime in the future.

Copy link
Author

@kdexd kdexd May 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

config_yaml = yaml.load(open(self.config_file))
config_yaml['atom_data'] = self.atom_data_filename

We open the config file and do an overwrite with the new path of atom data filename (the one which we provided from command line). It doesn't seem like an object. Did I miss anything ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh sorry, it was a misunderstanding, @yeganer I did as you suggested and it works fine. :)

with h5py.File(self.atom_data_filename, 'r') as slow_test_atom_data_file:
slow_test_data_file_uuid = slow_test_atom_data_file.attrs['uuid1']
assert slow_test_atom_data_file == kurucz_data_file_uuid

# The available config file doesn't have file paths of atom data file,
# densities and abundances profile files as desired. We form dictionary
# from the config file and override those parameters by putting file
# paths of these three files at proper places.
config_yaml = yaml.load(open(self.config_file))
config_yaml['atom_data'] = self.atom_data_filename
config_yaml['model']['abundances']['filename'] = self.abundances
config_yaml['model']['structure']['filename'] = self.densities

# The config hence obtained will be having appropriate file paths.
tardis_config = Configuration.from_config_dict(config_yaml)
Copy link
Contributor

@yeganer yeganer May 11, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's where the config get's validated and where atom_data becomes an object.

If I'm wrong, you can "load" the atomdata manually and pass the object to the config or the model. IIRC they all accept either a string or an AtomData object.
This might be easier anyway, instead of manipulating the config, load the AtomData directly.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This strategy sounds better. I am trying it.


# We now do a run with prepared config and get radial1d model.
self.obtained_radial1d_model = Radial1DModel(tardis_config)
simulation = Simulation(tardis_config)
simulation.legacy_run_simulation(self.obtained_radial1d_model)

# The baseline data against which assertions are to be made is ingested
# from already available compressed binaries (.npz). These will return
# dictionaries of numpy.ndarrays for performing assertions.
self.slow_test_data_dir = os.path.join(os.path.expanduser(
os.path.expandvars(pytest.config.getvalue('slow-test-data'))), "w7")

self.expected_ndarrays = np.load(os.path.join(self.slow_test_data_dir,
"ndarrays.npz"))
self.expected_quantities = np.load(os.path.join(self.slow_test_data_dir,
"quantities.npz"))

def test_j_estimators(self):
assert_allclose(
self.expected_ndarrays['j_estimators'],
self.obtained_radial1d_model.j_estimators)

def test_j_blue_estimators(self):
assert_allclose(
self.expected_ndarrays['j_blue_estimators'],
self.obtained_radial1d_model.j_blue_estimators)

j_blues_norm_factor = self.expected_quantities['j_blues_norm_factor']
j_blues_norm_factor = j_blues_norm_factor * u.Unit('1 / (cm2 s)')

assert_allclose(
j_blues_norm_factor,
self.obtained_radial1d_model.j_blues_norm_factor)

def test_last_line_interactions(self):
assert_allclose(
self.expected_ndarrays['last_line_interaction_in_id'],
self.obtained_radial1d_model.last_line_interaction_in_id)

assert_allclose(
self.expected_ndarrays['last_line_interaction_out_id'],
self.obtained_radial1d_model.last_line_interaction_out_id)

assert_allclose(
self.expected_ndarrays['last_line_interaction_shell_id'],
self.obtained_radial1d_model.last_line_interaction_shell_id)

last_line_interaction_angstrom = self.expected_quantities['last_line_interaction_angstrom']
last_line_interaction_angstrom = last_line_interaction_angstrom * u.Unit('Angstrom')

assert_allclose(
last_line_interaction_angstrom,
self.obtained_radial1d_model.last_line_interaction_angstrom)

def test_nubar_estimators(self):
assert_allclose(
self.expected_ndarrays['nubar_estimators'],
self.obtained_radial1d_model.nubar_estimators)

def test_ws(self):
assert_allclose(
self.expected_ndarrays['ws'],
self.obtained_radial1d_model.ws)

def test_spectrum(self):
luminosity_inner = self.expected_quantities['luminosity_inner']
luminosity_inner = luminosity_inner * u.Unit('erg / s')

assert_allclose(
luminosity_inner,
self.obtained_radial1d_model.luminosity_inner)

def test_montecarlo_properties(self):
montecarlo_luminosity = self.expected_quantities['montecarlo_luminosity']
montecarlo_luminosity = montecarlo_luminosity * u.Unit('erg / s')

montecarlo_virtual_luminosity = self.expected_quantities['montecarlo_virtual_luminosity']
montecarlo_virtual_luminosity = montecarlo_virtual_luminosity * u.Unit('erg / s')

montecarlo_nu = self.expected_quantities['montecarlo_nu']
montecarlo_nu = montecarlo_nu * u.Unit('Hz')

assert_allclose(
montecarlo_luminosity,
self.obtained_radial1d_model.montecarlo_luminosity)

assert_allclose(
montecarlo_virtual_luminosity,
self.obtained_radial1d_model.montecarlo_virtual_luminosity)

assert_allclose(montecarlo_nu, self.obtained_radial1d_model.montecarlo_nu)

def test_shell_temperature(self):
t_rads = self.expected_quantities['t_rads']
t_rads = t_rads * u.Unit('K')
assert_allclose(t_rads, self.obtained_radial1d_model.t_rads)
Loading