check_estimator#

pytorch_forecasting.utils._estimator_checks.check_estimator(estimator, raise_exceptions=False, tests_to_run=None, fixtures_to_run=None, verbose=True, tests_to_exclude=None, fixtures_to_exclude=None)[source]#

Run all tests on one single estimator or pytorch-forecasting object.

This utility runs all tests from the unified API conformance suites applying to the estimator, including tests for the specific subtype and all supertypes.

If estimator is an instance, tests are run on the specific instance and its class; if estimator is a class, tests are run on the class, and all instances constructed via its create_test_instances_and_names method.

For packaged objects such as neural network models, fetches the package class via the pkg attribute and also runs all tests on the package class.

Parameters:

  • estimator (estimator class or estimator instance) – can be any object from pytorch-forecasting for which suite tests exist.

  • raise_exceptions (bool, optional, default=False) –

    whether to return exceptions/failures in the results dict, or raise them

    • if False: returns exceptions in returned results dict

    • if True: raises exceptions as they occur

  • tests_to_run (str or list of str, optional. Default = run all tests.) – Names (test/function name string) of tests to run. sub-sets tests that are run to the tests given here.

  • fixtures_to_run (str or list of str, optional. Default = run all tests.) – pytest test-fixture combination codes, which test-fixture combinations to run. sub-sets tests and fixtures to run to the list given here. If both tests_to_run and fixtures_to_run are provided, runs the union, i.e., all test-fixture combinations for tests in tests_to_run, plus all test-fixture combinations in fixtures_to_run.

  • verbose (int or bool, optional, default=1.) –

    verbosity level for printouts from tests run.

    • 0 or False: no printout

    • 1 or True (default): print summary of test run, but no print from tests

    • 2: print all test output, including output from within the tests

  • tests_to_exclude (str or list of str, names of tests to exclude. default = None) – removes tests that should not be run, after subsetting via tests_to_run.

  • fixtures_to_exclude (str or list of str, fixtures to exclude. default = None) – removes test-fixture combinations that should not be run. This is done after subsetting via fixtures_to_run.

Returns:

results – dictionary of results of the tests in self keys are test/fixture strings, identical as in pytest, e.g., test[fixture]; entries are the string "PASSED" if the test passed, or the exception raised if the test did not pass. returned only if all tests pass, or raise_exceptions=False

Return type:

dict

Raises:

  • if raise_exceptions=True, –

  • raises any exception produced by the tests directly –

Examples

>>> from pytorch_forecasting.models import NBeats
>>> from pytorch_forecasting.utils import check_estimator

Running all tests for NBeats class, this uses all instances from get_test_params and compatible scenarios

>>> results = check_estimator(NBeats)
All tests PASSED!

Running specific test (all fixtures) for NBeats

>>> results = check_estimator(NBeats, tests_to_run="test_pkg_linkage")
All tests PASSED!

{‘test_pkg_linkage[NBeats-0]’: ‘PASSED’, ‘test_pkg_linkage[NBeats-1]’: ‘PASSED’}

Running one specific test-fixture-combination for NBeats

>>> check_estimator(
...    NBeats, fixtures_to_run="test_pkg_linkage[NBeats_pkg-NBeats]"
... )
All tests PASSED!
{'test_pkg_linkage[NBeats_pkg-NBeats]': 'PASSED'}