| Good code :white_check_mark: | Bad code :x: |
| ```python def display_last_uni_swaps( top: int = 10, sortby: str = "timestamp", descend: bool = False, export: str = "",) -> None: ``` | ```python def display_last_uni_swaps( top: int, sortby: str, descend: bool, export: str,) -> None: ``` |
| Good code :white_check_mark: | Bad code :x: |
| ```python def get_coins( top: int = 250, category: str = "") -> pd.DataFrame: ``` | ```python def load( file: str, file_types: list, data_files: Dict[Any, Any], data_examples: Dict[Any, Any],) -> pd.DataFrame: ``` |
| Good code :white_check_mark: | Bad code :x: |
| ```python data: pd.Series, dataset_name: str, y_label: str, ``` | ```python data: pd.Series, dataset: str, column: str, ``` |
| Good code :white_check_mark: | Bad code :x: |
| ```python def get_gaintopain_ratio(portfolio: PortfolioEngine) -> pd.DataFrame: """...""" gtp_period_df = portfolio_helper.get_gaintopain_ratio( portfolio.historical_trade_data, portfolio.benchmark_trades, portfolio.benchmark_returns) return gtp_period_df ``` | ```python def get_gaintopain_ratio(self) -> pd.DataFrame: """...""" vals = list() for period in portfolio_helper.PERIODS: port_rets = portfolio_helper.filter_df_by_period(self.portfolio_returns, period) bench_rets = portfolio_helper.filter_df_by_period(self.benchmark_returns, period) ... ``` |
| Good code :white_check_mark: | Bad code :x: |
| ```python # [fred_view.py] def display_yieldcurve(country: str): df = fred_model.get_yieldcurve(country) … # [fred_model.py] def get_yieldcurve(country: str) -> pd.Dataframe: … ``` | ```python # [fred_view.py] def display_bondscrv(country: str): df = fred_model.get_yieldcurve(country) … # [fred_model.py] def get_yldcurve(country: str) -> pd.Dataframe: … ``` |
It is important to keep a coherent UI/UX throughout the terminal. These are the rules we must abide:
- There is 1 single empty line between user input and start of the command output.
- There is 1 single empty line between command output and the user input.
- The menu help has 1 empty line above text and 1 empty line below. Both still within the rectangular panel.
- From menu help rectangular panel there's no empty line below - this makes it more clear to the user that they are inside such menu.
## External API Keys
### Creating API key
OpenBB Terminal currently has over 100 different data sources. Most of these require an API key that allows access to some free tier features from the data provider, but also paid ones.
When a new API data source is added to the platform, it must be added through [config_terminal.py](/openbb_terminal/config_terminal.py). E.g.
```python
# https://messari.io/
API_MESSARI_KEY = os.getenv("OPENBB_API_MESSARI_KEY") or "REPLACE_ME"
```
Note that a `OPENBB_` is added so that the user knows that that environment variable is used by our terminal.
### Setting and checking API key
One of the first steps once adding a new data source that requires an API key is to add that key to our [keys_controller.py](/openbb_terminal/keys_controller.py). This menu allows the user to set API keys and check their validity.
The following code allows to check the validity of the Polygon API key.
```python
def check_polygon_key(show_output: bool = False) -> str:
"""Check Polygon key
Parameters
----------
show_output: bool
Display status string or not. By default, False.
Returns
-------
str
Status of key set
"""
if cfg.API_POLYGON_KEY == "REPLACE_ME":
logger.info("Polygon key not defined")
status = KeyStatus.NOT_DEFINED
else:
r = request(
"https://api.polygon.io/v2/aggs/ticker/AAPL/range/1/day/2020-06-01/2020-06-17"
f"?apiKey={cfg.API_POLYGON_KEY}"
)
if r.status_code in [403, 401]:
logger.warning("Polygon key defined, test failed")
status = KeyStatus.DEFINED_TEST_FAILED
elif r.status_code == 200:
logger.info("Polygon key defined, test passed")
status = KeyStatus.DEFINED_TEST_PASSED
else:
logger.warning("Polygon key defined, test inconclusive")
status = KeyStatus.DEFINED_TEST_INCONCLUSIVE
if show_output:
console.print(status.colorize())
return str(status)
```
Note that there are usually 3 states:
- **defined, test passed**: The user has set their API key and it is valid.
- **defined, test failed**: The user has set their API key but it is not valid.
- **not defined**: The user has not defined any API key.
Note: Sometimes the user may have the correct API key but still not have access to a feature from that data source, and that may be because such feature required an API key of a higher level.
A function can then be created with the following format to allow the user to change its environment key directly from the terminal.
```python
@log_start_end(log=logger)
def call_polygon(self, other_args: List[str]):
"""Process polygon command"""
parser = argparse.ArgumentParser(
add_help=False,
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
prog="polygon",
description="Set Polygon API key.",
)
parser.add_argument(
"-k",
"--key",
type=str,
dest="key",
help="key",
)
if not other_args:
console.print("For your API Key, visit: https://polygon.io\n")
return
if other_args and "-" not in other_args[0][0]:
other_args.insert(0, "-k")
ns_parser = self.parse_simple_args(parser, other_args)
if ns_parser:
self.status_dict["polygon"] = keys_model.set_polygon_key(
key=ns_parser.key, persist=True, show_output=True
)
```
# ADVANCED
## Important functions and classes
### Base controller class
This `BaseController` class is inherited by all controllers on the terminal.
This class contains both important variables and methods that are common across all terminal controllers.
**CHOICES_COMMON**: List of common commands across all controllers
- `cls`: clear screen
- `home`: go back to the main root
- `about`: allows to open our documentation directly on the menu or command
- `h`, `?` and `help`: display the help menu the user is in
- `q`, `quit` and `..`: go back to one menu above
- `exit`: exit the platform
- `r` and `reset`: reset the platform (reading code and settings again but going into the same state)
- `support`: create a support request ticket
All of these variables have a `call_FUNCTION` associated with them.
Worthy methods to mention are:
- `load_class`: Checks for an existing instance of the controller before creating a new one to speed up access to that menu.
- `custom_reset`: Should be used by controllers that rely on a state variable - meant to be overridden. They should add the commands necessary to have the same data loaded.
- `print_help`: Meant to be overridden by each controller
- `parse_input`: Processes the string the user inputs into a list of actionable commands
- `switch`: Acts upon the command action received
- `parse_known_args_and_warn`: Parses the command with the `-` and `--` flags and variables. Some built-in flags are:
- `export_allowed`: Which can be set to `_NO_EXPORT_`, `_EXPORT_ONLY_RAW_DATA_ALLOWED_`, `_EXPORT_ONLY_FIGURES_ALLOWED_` and `_EXPORT_BOTH_RAW_DATA_AND_FIGURES_`
- `raw`: Displaying the data raw
- `limit`: Number of rows to display
- `menu`: Most important method. When a menu is executed, the way to call it is through `stocks_menu.menu()`
## Default Data Sources
The document [data_sources_default.json](/data_sources_default.json) contains all data sources that the terminal has access to and specifies the data source utilized by default for each command.
The convention is as follows:
```python
{
"stocks": {
"search": ["FinanceDatabase"],
"quote": ["YahooFinance"],
"candle": [],
"load": [
"YahooFinance",
"AlphaVantage",
"Polygon",
"EODHD"
],
"options": {
"unu": ["FDScanner"],
"calc": [],
"screen": {
"view": ["Syncretism"],
"set": [],
"scr": ["Syncretism"]
},
"load": [
"YahooFinance",
"Tradier",
"Nasdaq"
],
"exp": [
"YahooFinance",
"Tradier",
"Nasdaq"
],
...
```
The way to interpret this file is by following the path to a data source, e.g.
- `stocks/search` relies on `FinanceDatabase`
- `stocks/candle` does not rely on any data source. This means that it relies on data that has been loaded before.
- `stocks/load` relies on `YahooFinance`, `AlphaVantage`, `Polygon` or `EODHD`.
- **The order is important as the first data source is the one utilized by default.**
- `stoks/options/unu` relies on `FDScanner`.
- `stocks/options/exp` relies on `YahooFinance` by default but `Tradier` and `Nasdaq` sources are allowed.
### Export Data
In the `_view.py` files it is common having at the end of each function `export_data` being called. This typically looks like:
```python
export_data(
export,
os.path.dirname(os.path.abspath(__file__)),
"contracts",
df_contracts
)
```
Let's go into each of these arguments:
- `export` corresponds to the type of file we are exporting.
- If the user doesn't has anything selected, then this function doesn't do anything.
- The user can export multiple files and even name the files.
- The allowed type of files `json,csv,xlsx` for raw data and `jpg,png,svg` for figures depends on the `export_allowed` variable defined in `parse_known_args_and_warn`.
- `os.path.dirname(os.path.abspath(__file__))` corresponds to the directory path
- This is important when `export folder` selected is the default because the data gets stored based on where it is called.
- If this is called from a `common` folder, we can use `os.path.dirname(os.path.abspath(__file__)).replace("common", "stocks")` insteaad
- `"contracts"` corresponds to the name of the exported file (+ unique datetime) if the user doesn't provide one
- `df_contracts` corresponds to the dataframe with data. Although we don't call this function with the figure reference, because it is open, we can use `plt.savefig` to achieve that.
If `export_allowed=EXPORT_BOTH_RAW_DATA_AND_FIGURES` in `parse_known_args_and_warn`, valid examples are:
- `cmd --export csv`
- `cmd --export csv,png,jpg`
- `cmd --export mydata.csv`
- `cmd --export mydata.txt,alsomydata.csv,alsoalsomydata.png`
Note that these files are saved on a location based on the environment variable: `EXPORT_FOLDER_PATH`. Which can be set in `settings/export`.
The default location is the `exports` folder and the data will be stored with the same organization of the terminal. But, if the user specifies the name of the file, then that will be dropped onto the folder as is with the datetime attached.
### Queue and pipeline
The variable `self.queue` contains a list of all actions to be run on the platform. That is the reason why this variable is always passed as an argument to a new controller class and received back.
```python
self.queue = self.load_class(
DarkPoolShortsController, self.ticker, self.start, self.stock, self.queue
)
```
Example:
If a user is in the root of the terminal and runs:
```shell
stocks/load AAPL/dps/psi -l 90
```
The queue created becomes:
`self.queue = ["stocks", "load AAPL", "dps", "psi -l 90"]`
And the user goes into the `stocks` menu and runs `load AAPL`. Then the queue is updated to
`self.queue = ["dps", "psi -l 90"]`
At that point the user goes into the `dps` menu and runs the command `psi` with the argument `-l 90` therefore displaying price vs short interest of the past 90 days.
### Auto Completer
In order to help users with a powerful autocomplete, we have implemented our own (which can be found [here](/openbb_terminal/custom_prompt_toolkit.py)).
This **STATIC** list of options is meant to be defined on the `__init__` method of a class as follows.
```python
if session and obbff.USE_PROMPT_TOOLKIT:
self.choices: dict = {c: {} for c in self.controller_choices}
self.choices["overview"] = {
"--type": {c: None for c in self.overview_options},
"-t": "--type",
}
self.choices["futures"] = {
"--commodity": {c: None for c in self.futures_commodities},
"-c": "--commodity",
"--sortby": {c: None for c in self.wsj_sortby_cols_dict.keys()},
"-s": "--sortby",
"--reverse": {},
"-r": "--reverse",
}
self.choices["map"] = {
"--period": {c: None for c in self.map_period_list},
"-p": "--period",
"--type": {c: None for c in self.map_filter_list},
"-t": "--type",
}
self.completer = NestedCompleter.from_nested_dict(self.choices)
```
Important things to note:
- `self.choices: dict = {c: {} for c in self.controller_choices}`: this allows users to have autocomplete on the command that they are allowed to select in each menu
- `self.choices["overview"]`: this corresponds to the list of choices that the user is allowed to select after specifying `$ overview`
- `"--commodity": {c: None for c in self.futures_commodities}`: this allows the user to select several commodity values after `--commodity` flag
- `"-c": "--commodity"`: this is interpreted as `-c` having the same effect as `--commodity`
- `"--reverse": {}`: corresponds to a boolean flag (does not expect any value after)
- `"--start": None`: corresponds to a flag where the values allowed are not easily discrete due to vast range
- `self.completer = NestedCompleter.from_nested_dict(self.choices)`: from the choices create our custom completer
In case the user is interested in a **DYNAMIC** list of options which changes based on user's state, then a class method must be defined.
The example below shows the `update_runtime_choices` method being defined in the options controller.
```python
def update_runtime_choices(self):
"""Update runtime choices"""
if self.expiry_dates and session and obbff.USE_PROMPT_TOOLKIT:
self.choices["exp"] = {str(c): {} for c in range(len(self.expiry_dates))}
self.choices["exp"]["-d"] = {c: {} for c in self.expiry_dates + [""]}
self.completer = NestedCompleter.from_nested_dict(self.choices)
```
This method should only be called when the user's state changes leads to the auto-complete not being accurate.
In this case, this method is called as soon as the user successfully loads a new ticker since the options expiry dates vary based on the ticker. Note that the completer is recreated from it.
### Logging
A logging system is used to help tracking errors inside the OpenBBTerminal.
This is storing every logged message inside the following location :
`$HOME/OpenBBUserData/logs`
Where $HOME is the user home directory, for instance:
- `C:\Users\foo` if your are in Windows and your name is foo
- `/home/bar/` if you are is macOS or Linux and your name is bar
The user can override this location using the settings key `OPENBB_USER_DATA_DIRECTORY`.
If you want to log a particular message inside a function you can do like so:
```python
import logging
logger = logging.getLogger(__name__)
def your_function() -> pd.DataFrame:
logger.info("Some log message with the level INFO")
logger.warning("Some log message with the level WARNING")
logger.fatal("Some log message with the level FATAL")
```
You can also use the decorator `@log_start_end` to automatically record a message every time a function starts and ends, like this:
```python
import logging
from openbb_terminal.decorators import log_start_end
logger = logging.getLogger(__name__)
@log_start_end(log=logger)
def your_function() -> pd.DataFrame:
pass
```
### Internationalization
WORK IN PROGRESS - The menu can be internationalised BUT we do not support yet help commands`-h` internationalization.
In order to add support for a new language, the best approach is to:
1. Copy-paste `i18n/en.yml`
2. Rename that file to a short version of language you are translating to, e.g. `i18n/pt.yml` for portuguese
3. Then just update the text on the right. E.g.
```text
stocks/NEWS: latest news of the company
```
becomes
```text
stocks/NEWS: mais recentes notĂcias da empresa
```
Note: To speed up translation, the team developed a [script](/i18n/help_translation.ipynb) that uses Google translator API to help translating the entire `en.yml` document to the language of choice. Then the output still needs to be reviewed, but this can be an useful bootstrap.
This is the convention in use for creating a new key/value pair:
- `stocks/search` - Under `stocks` context, short command `search` description on the `help menu`
- `stocks/SEARCH` - Under `stocks` context, long command `search` description, when `search -h`
- `stocks/SEARCH_query` - Under `stocks` context, `query` description when inquiring about `search` command with `search -h`
- `stocks/_ticker` - Under `stocks` context, `_ticker` is used as a key of a parameter, and the displayed parameter description is given as value
- `crypto/dd/_tokenomics_` - Under `crypto` context and under `dd` menu, `_tokenomics_` is used as a key of an additional information, and the displayed information is given as value
## Write Code and Commit
At this stage it is assumed that you have already forked the project and are ready to start working.
### Pre Commit Hooks
Git hook scripts are useful for identifying simple issues before submission to code review. We run our hooks on every
commit to automatically point out issues in code such as missing semicolons, trailing whitespace, and debug statements.
By pointing these issues out before code review, this allows a code reviewer to focus on the architecture of a change
while not wasting time with trivial style nitpicks.
Install the pre-commit hooks by running: `pre-commit install`.
### Coding
Although the Coding Guidelines section has been already explained, it is worth mentioning that if you want to be faster
at developing a new feature, you may implement it first on a `jupyter notebook` and then carry it across to the
terminal. This is mostly useful when the feature relies on scraping data from a website, or implementing a Neural
Network model.
### Git Process
1. Create your Feature Branch, e.g. `git checkout -b feature/AmazingFeature`
2. Check the files you have touched using `git status`
3. Stage the files you want to commit, e.g.
`git add openbb_terminal/stocks/stocks_controller.py openbb_terminal/stocks/stocks_helper.py`.
Note: **DON'T** add `config_terminal.py` or `.env` files with personal information, or even `feature_flags.py` which is user-dependent.
4. Write a concise commit message under 50 characters, e.g. `git commit -m "meaningful commit message"`. If your PR
solves an issue raised by a user, you may specify such issue by adding #ISSUE_NUMBER to the commit message, so that
these get linked. Note: If you installed pre-commit hooks and one of the formatters re-formats your code, you'll need
to go back to step 3 to add these.
### Branch Naming Conventions
The accepted branch naming conventions are:
- `feature/feature-name`
- `hotfix/hotfix-name`
- `release/2.1.0` or `release/2.1.0rc0`.
All `feature/feature-name` related branches can only have PRs pointing to `develop` branch. `hotfix/hotfix-name` and `release/2.1.0` or `release/2.1.0rc0` branches can only have PRs pointing to `main` branch.
## Add a Test
Unit tests minimize errors in code and quickly find errors when they do arise. Integration tests are standard usage examples, which are also used to identify errors.
A thorough introduction on the usage of unit tests and integration tests in OpenBBTerminal can be found on the following page respectively:
[Unit Test README](tests/README.md)
[Integration Test README](scripts/README.md)
In short:
- Pytest: is the tool we are using to run our tests, with the command: `pytest tests/`
- Coverage: can be checked like running `coverage run -m pytest` or `coverage html`
## Installers
When implementing a new feature or fixing something within the codebase, it is necessary to ensure that it is working
appropriately on the terminal. However, it is equally as important to ensure that new features or fixes work on the
installer terminal too. This is because a large portion of users utilize the installer to use OpenBB Terminal.
More information on how to build an installer can be found [here](build/README.md).