jerry/OpenBB
1
0
Fork 0
mirror of https://github.com/OpenBB-finance/OpenBB.git synced 2026-10-27 12:40:35 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update test documentation (#1630)

Browse Source 
Co-authored-by: Colin Delahunty <72827203+colin99d@users.noreply.github.com>
Co-authored-by: Theodore Aptekarev <aptekarev@gmail.com>
This commit is contained in:
minhhoang1023
2022-04-05 02:15:05 +07:00
committed by GitHub
parent 01391c2564
commit b4e44d23d0
1 changed files with 211 additions and 78 deletions
Download Patch File Download Diff File Expand all files Collapse all files

289
tests/README.md vendored
View File

@@ -2,17 +2,19 @@
This document is part of the `OpenBB Terminal` library documentation.
It aims to provide necessary information in order :
It aims to provide necessary information in order to:
- Run `unit tests`
- Build `unit tests`
- Fix failed existing `unit tests`
- Maintain `unit tests`
## 1.1. Why having unit tests ?
Here `Unit tests` purpose is to allow : update without fear.
`Unit tests` purpose is to allow developers to update without fear.
Writing tests that are easy to run will help the person modifying the code to do so quick and with more confidence. Having tests available also decreases the chances that bugs will make it into production and break other parts of the code.
Insuring with enough confidence that : addition, removal or update of a `module/function` won't break other parts of the code.
To do that we need `unit tests` to be :
@@ -21,11 +23,11 @@ To do that we need `unit tests` to be :
**FAST TO WRITE**
The `unit tests` will only tests the most common cases.
The `unit tests` is used for testing a single functionality for its correctness of input and the corresponding output. Unit tests are about ensuring that low-level implementation details are rock solid.
**FAST TO RUN**
Slow part of the code like `network` or `database` access should be either `mocked` or `marked` as `slow`.
Tests can be slow at times, due to the need to connect with external services. This can be sending a HTTP request or connecting to a database. We can speed this up by using mocks. In case this is not possible, make sure to `marked` it as `slow`
# 2. Run `unit tests`
@@ -33,38 +35,36 @@ In this section we will explain everything you need to run the `unit tests` on `
## 2.1. How to install tests dependencies ?
To run the tests you will need the `dev-dependencies`.
By default poetry installs the `dev-dependencies` when you run this command :
To run the tests you will need first to install the `dev-dependencies`. By default poetry installs the `dev-dependencies` when you run this command :
```bash
poetry install
```
If you dont want to install the `dev-dependencies` you will have to add the option `--no-dev` like this :
If you do not want to install the `dev-dependencies` you will have to add the option `--no-dev` like this :
```bash
poetry install --no-dev
```
## 2.2. How to run `tests` : by `module` ?
## 2.2. How to run `tests` :
### By `module`
You can run tests on a specific package/module by specifying the path of this package/module, like this :
You can run tests on a specific package/module by specifying the path of this package/module, as follows:
```bash
pytest tests/openbb_terminal/some_package
pytest tests/openbb_terminal/some_package/some_module.py
```
### By test `name`
## 2.3. How to run `tests` : by `name` ?
You can run tests by their name :
You can run tests by their name with the argument `-k`
```
pytest -k "test_function1"
```
## 2.4. How to run `tests` : by `markers` ?
### By `markers`
You can run tests only on specific markers, like this :
@@ -80,7 +80,11 @@ You can list the available markers using this command :
pytest --markers
```
## 2.5. How to skip tests function ?
## 2.3. How to skip tests
### Skipping a specific test
You can use the decorator `@pytest.skip` as below:
```python
import pytest
@@ -94,7 +98,7 @@ def test_another_function(mocker):
pass
```
## 2.6. How to skip tests modules ?
### Skipping the entire test module
```python
import pytest
@@ -105,38 +109,72 @@ def test_some_function(mocker):
pass
```
# 3. Build `unit tests`
# 3. How to build `unit tests`
## 3.1. Where to add my `test module` ?
When you contribute a new feature to the Terminal, it's important that tests are added for this particular feature. It is a part of the Checklist for the PR to be approved.
All the `unit tests` should be insides the `tests` folder.
All the `unit tests` should be insides the `tests` folder. There should be at most one `test module` for each `module` of `OpenBB Terminal`.
There should be at most one `test module` for each `module` of `OpenBB Terminal`.
Each `test module` should follow the same path of the `module` that it is `testing`.For instance,
- in order to test the following module `openbb_terminal/stocks/due_diligence/dd_controller.py`
- a `test module` should be added here: `tests/openbb_terminal/stocks/due_diligence/test_dd_controller.py`
Each `test module` should follow the same path than the `module` it is `testing`.
Now that you know where to add tests, let's go through 3 types of unit tests that we have:
For instance to `test` the following module :
- `openbb_terminal/stocks/due_diligence/dd_controller.py`
## 3.1. Model Test
A `test module` should be added here :
As mentioned in the main repo's README, we are following the MVC pattern. The `_model.py`'s job to get the required data, e.g. from an external APIs or scraping.
- `tests/openbb_terminal/stocks/due_diligence/test_dd_controller.py`
Therefore, in unit tests for `_model.py`, the function output is captured and saved when the test is first run. Then, the next time the test is run, the function's returned value is compared with the captured value in the first run, to ensure parity.
## 3.2. How to record network ?
Depending on the data type of function output, we can generate the following files:
- csv : for pandas.DataFrame
- json : for python native data type (list, dict, tuple)
- txt : store str or printed output
Network exchange can be recording inside a `test function` using the `@pytest.mark.vcr` `marker`.
Here is an example :
Examples:
```
@pytest.fixture(scope="module")
def vcr_config():
return {
"filter_query_parameters": [
("token", "MOCK_TOKEN"),
],
}
```python
@pytest.mark.vcr
def test_single():
assert requests.get("http://httpbin.org/get").text == '{"get": true}'
@pytest.mark.parametrize(
"ticker",
[
("BTC"),
("BTC-USD"),
],
)
def test_check_supported_ticker(ticker, recorder):
df = sentimentinvestor_model.check_supported_ticker(ticker)
recorder.capture(df)
```
You can choose the `cassette` name using the `@pytest.mark.default_cassette` marker.
Here is an example :
Several things worth noting here:
1. If the model utilizes an API key or token, make sure to filter it insider the `vcr_config` fixture inside your test module, similar to the example. This will hide your actual API key / token for security purposes.
2. Record HTTP requests using the decorator `@pytest.mark.vcr`
1. When run the first time, the VCR will record the requests to `cassettes/test_*.yml`
2. Run it again, and VCR will replay the recorded response when the HTTP request is made. This test is now fast (no real HTTP requests are made anymore), deterministic (the test will continue to pass, even if you have no internet connection or that external APIs go down) and accurate (the response will contain the same headers and body you get from a real request).
3. Test out multiple input variables to ensure tests cover multiple edge cases
1. Use the decorator `@pytest.mark.parametrize`to specify the inputs for your tests.
4. Capture the returned df using `recorder.capture(<return_value>)`
1. Next time the test run, return output will be compared against the captured file
5. Each fixture / cassette file for a test module should be stored within the same folder. An example with `tests/openbb_terminal/some_package/some_module.py` :
- `tests/openbb_terminal/some_package`/cassettes/`some_module`/test_function1.yaml
- `tests/openbb_terminal/some_package`/txt/`some_module`/test_function1.csv
- `tests/openbb_terminal/some_package`/csv/`some_module`/test_function1.csv
- `tests/openbb_terminal/some_package`/json/`some_module`/test_function1.json
6. You can choose the `cassette` name using the `@pytest.mark.default_cassette` marker. Here is an example :
```python
@pytest.mark.default_cassette("example.yaml")
@@ -145,46 +183,139 @@ def test_default():
assert requests.get("http://httpbin.org/get").text == '{"get": true}'
```
## 3.2. Where to put data files ?
## 3.2. View Test
Different from the `model.py`, `view.py` module handles how data is displayed, e.g.as a chart or a table.
Each data file should be related exactly to one `test module`.
The `view.py` has no return statement, and simply prints to stdout. Hence, our unit tests will capture stdout values, and compare against the recorded fixture in subsequent runs.
The purpose is to have each `test module` as self-contained as possible.
Example:
Data related to a `test module` should be stored within the same folder than this `test module`.
Example with `tests/openbb_terminal/some_package/some_module.py` :
- `tests/openbb_terminal/some_package`/cassettes/`some_module`/test_function1.yaml
- `tests/openbb_terminal/some_package`/txt/`some_module`/test_function1.csv
- `tests/openbb_terminal/some_package`/csv/`some_module`/test_function1.csv
- `tests/openbb_terminal/some_package`/json/`some_module`/test_function1.json
## 3.3. How to filter `api keys` on `cassettes` ?
The filtering of `api keys` from a `cassette` should be done directly inside the `test module` using this `cassette`.
The purpose is to have each `test module` as self-contained as possible.
To filter `api keys` from a `cassettes` you can define a `vcr_config` fixture inside your `test module`.
Here is an example of filtering :
```python
@pytest.fixture(scope="module")
def vcr_config():
return {
"filter_query_parameters": [
("token", "MOCK_TOKEN"),
]
}
```
@pytest.mark.vcr
@pytest.mark.record_stdout
def test_display_historical(mocker):
# MOCK VISUALIZE_OUTPUT
mocker.patch(target="openbb_terminal.helper_classes.TerminalStyle.visualize_output")
sentimentinvestor_view.display_historical(
ticker="AAPL", start="2021-12-12", end="2021-12-15", export="", raw=True
)
```
More information about `vcr_config` fixture in the documentation of `pytest-recording` :
Important notes to go through:
1. Use `@pytest.mark.record_stdout`to capture `sdtout`.
1. Recorded stdout is saved as fixtures in a txt file
2. If your view function only plots a chart, and does not output anything in `sdtout`, you can skip unit testing your `view.py`
3. Make sure to mock `visualize_output` method.
1. Inside `visualize_output`, we use `ion` to enable interactive mode. However, under Windows, having `ion` enabled would make tests fail to run. Hence, you should always mock `visualize_output`, and hence `ion`
- https://github.com/kiwicom/pytest-recording
## 3.4. How to `mock` ?
## 3.3. Controller Test
Unit test for `_controller.py` is the last component you would need to add. Most of the time you won't need to create a `_controller.py` from scratch. Hence, we will only cover how to add tests for a new command to an existing `test_*_controller.py` file
#### Test print help
This test in the `test_*_controller.py` captured the help text printed to stdout.
Example:
```
@pytest.mark.vcr(record_mode="none")
@pytest.mark.record_stdout
def test_print_help():
controller = alt_controller.AlternativeDataController(queue=None)
controller.print_help()
```
Adding an additional feature to the `controller.py` will alter the help text. Make sure to run the tests with the argument `--rewrite-expected` to re-generate the text file.
Example: `pytest openbb_terminal.cryptocurrency.defi.defi_controller --rewrite-expected`
#### Test call function
This test is to check when `controller.py` calls a command, the corresponding function from a `view.py` will be called with the correct arguments.
Example:
```
@pytest.mark.vcr(record_mode="none")
@pytest.mark.parametrize(
"tested_func, other_args, mocked_func, called_args, called_kwargs",
[
(
"call_aterra",
[
"ust",
"--address=terra1wg2mlrxdmnnkkykgqg4znky86nyrtc45q336yv", # pragma: allowlist secret
],
"terraengineer_view.display_terra_asset_history",
[],
dict(),
),
(
"call_newsletter",
[],
"substack_view.display_newsletters",
[],
dict(),
),
],
)
def test_call_func(
tested_func, mocked_func, other_args, called_args, called_kwargs, mocker
):
path_controller = "openbb_terminal.cryptocurrency.defi.defi_controller"
if mocked_func:
mock = mocker.Mock()
mocker.patch(
target=f"{path_controller}.{mocked_func}",
new=mock,
)
controller = defi_controller.DefiController(queue=None)
getattr(controller, tested_func)(other_args)
if called_args or called_kwargs:
mock.assert_called_once_with(*called_args, **called_kwargs)
else:
mock.assert_called_once()
else:
controller = defi_controller.DefiController(queue=None)
getattr(controller, tested_func)(other_args)
```
When adding a new feature to an existing menu, there should already be a `test_*_controller.py` file with the `test_call_func()` function. You can simply add another tuple under `@pytest.mark.parametrize()`, which contains:
1. `call_<command>` from the controller.py file
2. Arguments you want to pass in the `call_<command>`
3. The corresponding view function, which will be mocked (Here we only want check if the view function is called at least once. We mock it such that the function won't execute.)
4. Arguments you want to pass to the view function. Note that the arguments here need to be the same as arguments in (2). This is to ensure that arguments passed from the controller.py file are passed correctly to function in `view.py`
## 3.4. Fixing existing tests
It is not uncommon that your new feature or bug fixes break the existing code. Understanding a few concepts can help you navigate and fix most common cases.
Before continuing this section, please have a look at our [FIXTURES](FIXTURES.md) documentation.
TL;DR:
You can run unit tests under a few mode:
- `--rewrite-expected` will re-generate the fixtures (csv, json and txt files) if there's detected change.
- `--record-mode` will impact both the cassette (yaml) and fixtures (csv, json, txt files)
1. `--record-mode=none` : searches for existing cassette / fixtures. If there's none, an exception will be thrown.
2. `--record-mode=once` : creates new fixtures and cassette if they did not exist. If there are already fixtures or cassette, it will take the existing one.
3. `--record-mode=rewrite` : this rewrites both the cassette and fixtures, even if they already exist.
### Scenario 1: Update Fixtures
Let's say you just updated the code in `_model.py` file, which then altered the recorded output. You can update the fixture files by running the tests with `--rewrite-expected` argument. This would update the fixtures with latest changes.
Example: `pytest openbb_terminal.cryptocurrency.defi.defi_controller --rewrite-expected`
### Scenario 2: Update both Fixtures and Cassettes
If you want to update existing cassettes and fixtures, use the argument `--record-mode=rewrite` when running tests.
As existing cassettes and fixtures will be overwritten (if there's any), make sure that you check both, and that they yield expected output.
## 3.5. How to `mock` ?
**PYTEST-MOCK**
@@ -214,7 +345,7 @@ More information about `monkeypatch` are available here :
- https://docs.pytest.org/en/6.2.x/monkeypatch.html
## 3.5. Which `helpers` are available ?
## 3.6. Which `helpers` are available ?
You can find the available helpers inside the following package/module :
@@ -223,7 +354,7 @@ You can find the available helpers inside the following package/module :
See also the `pytest fixtures` which are autoloaded helpers.
## 3.6. Which `fixtures` are available ?
## 3.7. Which `fixtures` are available ?
You can list all the available `pytest fixtures` using the following command :
@@ -235,7 +366,7 @@ More on custom fixtures here :
- [FIXTURES](FIXTURES.md)
## 3.7. Which `markers` are available ?
## 3.8. Which `markers` are available ?
You can list the available markers using this command :
@@ -247,11 +378,11 @@ More information on markers location are available in pytest documentation :
- https://docs.pytest.org/en/6.2.x/mark.html#mark
## 3.8. Known `issue` / `solution`
## 3.9. Known `issue` / `solution`
**ION Usage**
If you are testing for a `_view.py` module that uses `ION`, you would need to mock it. With Windows, the charts / graphs are unable to close the graph applications. This would make the tests failed when running with Windows.
As mentioned earlier, if you are testing for a `_view.py` module that uses `ION`, you would need to mock it. With Windows, the charts / graphs are unable to close the graph applications. This would make the tests failed when running with Windows.
You can use the following example below. Once you have mocked the graphs / charts, simply call the display function, which is now a mocked object.
@@ -259,10 +390,12 @@ You can use the following example below. Once you have mocked the graphs / chart
@pytest.mark.vcr
@pytest.mark.record_stdout
def test_display_defi_tvl(mocker):
# MOCK CHARTS
mocker.patch.object(target=llama_view.obbff, attribute="USE_ION", new=True)
mocker.patch(target="openbb_terminal.stocks.options.yfinance_view.plt.ion")
mocker.patch(target="openbb_terminal.stocks.options.yfinance_view.plt.show")
# MOCK EXPORT_DATA
mocker.patch(target="openbb_terminal.cryptocurrency.defi.llama_view.export_data")
# MOCK VISUALIZE_OUTPUT
mocker.patch(target="openbb_terminal.helper_classes.TerminalStyle.visualize_output")
llama_view.display_defi_tvl(20)
```
@@ -383,7 +516,7 @@ pytest tests/.../test_some_test_module.py --record-mode=once --rewrite-expected
pytest tests/.../test_some_test_module.py --record-mode=once --rewrite-expected
```
## 3.9. List of useful `vscode` tools for `unit tests`
## 3.10. List of useful `vscode` tools for `unit tests`
**VSCODE TESTING**
@@ -395,7 +528,7 @@ More information on this tool are available here :
- https://code.visualstudio.com/docs/python/testing
## 3.10. How to handle `dev-dependencies` ?
## 3.11. How to handle `dev-dependencies` ?
**UPDATE PYPROJECT**