Files
OpenBB/cli/openbb_cli/controllers/utils.py
Danglewood 2232178fd8 [Release] OpenBB v4.7.0 (#7395)
* Merge Release/4.6.0 From Main Into Develop (#7301)

* add snowflake integration video (#6339)

* [HotFix] Fix broken URLs in docs page. (#6368)

* Fix broken URLs

* use full URL to other docs pages.

* [Release] 4.2.0 (#6422)

* [Docs] Remove Reference to `to_chart` in OBBject. (#6096)

* remove reference to to_chart in OBBject.

* put chart description back in OBBject properties

* [Feature] - Create openbb assets folder (#6101)

* create openbb assets folder

* fix unittest

* rebuild

---------

Co-authored-by: Igor Radovanovic <74266147+IgorWounds@users.noreply.github.com>

* [BugFix] Fixes to_df() where the date series contains multiple TZ-offsets. (#6099)

* allow mixed utc offset in a tz-aware date column

* enforce ascending from all providers for price.historical

* fix tests

* add comments and apply treatment to basemodel_from_df before JSON output

* yfinance consistency

* add unit test

* add unit test

* better test

* ruff

---------

Co-authored-by: Igor Radovanovic <74266147+IgorWounds@users.noreply.github.com>

* Update the quantitative extension to make more sense (#6087)

* Split out a rolling submenu for the rolling functions

* Make a performance and a stats submenu.

* Test the statistics functions

* lint

* lint

* dupe test

* pylint

* ruff

* Try tests quick

* black magic signature funcs

* fix my custom tests

* Fix the existing imports/urls

* push the api update

* okay I figured out whats going on

* this should be all  of them

* Correct docstringing examples

---------

Co-authored-by: Igor Radovanovic <74266147+IgorWounds@users.noreply.github.com>
Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* [BugFix] Fix ECB Yield Curve (#6108)

* fix ecb yield curve

* standard model update

* Update unit-test workflow to check for changes to openbb_platform (#6109)

* [Bug fix] - Fix package mypy (#6110)

* fix: package mypy

* types

* black

* [Feature] - Equity router descriptions & examples (#6103)

* feat: equity/calendar

* feat: compare router

* fix descriptions

* darkpool + discovery

* fix paragraph issues

* fix descriptions

* fundamental part 1

* fundamental part2

* transcipts

* ownership

* fix

* price

* shorts

* Update darkpool_router.py

* minor fix

* shorts examples

* examples

* ruff

* black

---------

Co-authored-by: Igor Radovanovic <74266147+IgorWounds@users.noreply.github.com>

* [Feature] - Add `exception_handler` decorator to static (#6112)

* add exception handler decorator

* add exception handler decorator code

* updated static

* rebuild to fix docstring indentation

---------

Co-authored-by: Diogo Sousa <montezdesousa@gmail.com>

* update extensions list (#6114)

* add missing field descriptions to yfinance equity price historical (#6116)

Co-authored-by: Igor Radovanovic <74266147+IgorWounds@users.noreply.github.com>

* remove charting extension env var (#6118)

* [BugFix] Fix Company News (#6111)

* fix company news

* update tests

* multiple symbols for polygon

* don't include pageSize in Benzinga querystring, fixed as 100

* remove extra space

* black

* unused imports

* make benzinga test cassette smaller

* missing params

* clear empty strings

---------

Co-authored-by: Igor Radovanovic <74266147+IgorWounds@users.noreply.github.com>

* [BugFix] Allow Multiple Symbols in `equity.fundamental.metrics()` from `yfinance` (#6124)

* allow multiple symbols in key_metrics from yfinance

* field order thing

* remove unnecessary function (#6125)

* [Docs] Update OpenBB Copilot docs (#6106)

* WIP: Tweak description.

* WIP: Experimenting with react-player

* WIP: Add copilot docs.

* fixes videos width/height

* WIP: Refine videos and copy.

* WIP: Add last videos and sections.

* Add bring-your-own-copilot section.

* Tweak grammar.

* Add copilot link.

* Fix typos.

---------

Co-authored-by: jose-donato <43375532+jose-donato@users.noreply.github.com>

* [Enhancement] Add Previous Close to DATA_DESCRIPTIONS (#6120)

* add data description for previous close

* ..just the one

* update the model tripped by this change

---------

Co-authored-by: montezdesousa <79287829+montezdesousa@users.noreply.github.com>

* added australia as a country also (#6131)

* [Bug fix] - Fix regulator cik map standard model (#6129)

* Fix regulator cik map standard model

* fix description

* Update cik_map.py

* increase timeout to 30 (#6115)

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* clean up some v3 deps (#6105)

* clean up some v3 deps

* forgot we need to do the requirements txt files

* not sure it matters, but run ruff + black

* am addicted to bug fixes

---------

Co-authored-by: Igor Radovanovic <74266147+IgorWounds@users.noreply.github.com>
Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* [Bug fix] - explicit multiple items docstring (#6127)

* explicit multiple items docstring

* rebuiild

* rebuild again

---------

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* [Feature] Pretty Pydantic `ValidationError` exceptions (#6130)

* fix validator in fmp IndexHistorical model

* make ValidationError pretty

* modify arg error

* linting fmp IndexHistorical

* black

* remove unused code

* tmx etf info (#6132)

Co-authored-by: montezdesousa <79287829+montezdesousa@users.noreply.github.com>

* [Feature] - Platform V4 Markdown Generator V2 (#6094)

* add recent `openapi.json`

* fix data types in models

* removed `openapi.json`

* set default date value as None in PolygonCurrencyPairs class

* add function to generate reference.json file

* reworked function to improve readability

* reword function to add seo metadata

* add functions for creating markdown sections

* add code to extract data card markdown title

* add additional functions for generating index and data models file

* extract first sentence of the description for cards

* code cleanup and documentation

* linting

* linting polygon models

* add openbb import statement in create_reference_markdown_examples function

* add POST method functions

* cleanup; reworked generate_reference_index_files function

* moved development section to 7th position in the sidebar

* fix POST function params default value

add standard flag to QueryParams and Data fields

cleanup

* add type expansion from package_builder.MethodDefinition

* sort data models cards alphabetically

make printing less verbose

* make MAX_CARDS global

display less content in cards in Commannds section

* cleanup

* Remove '_' from the cards under Commands section

* " to ' in econometrics/causality

* replace ' with " in ReferenceCard for reference dir index files

* remove extra . from the quantile function description

* shoutout to @deeleeramone for finding POST method description bug!

* set correct value for standard field

* handle BaseModel types in provider data fields

* unit tests for the platform markdown generator v2

* yeet 'Default' and 'Optional' columns in the 'Data' section

* last minute bug fix

* add info for multiple symbols

* make multiple items info same as platform static

* organize sections properly

* sort reference sub-directories alphabetically

* extra space in 'OBBject extra' description

* add type expansion for fields with multiple items

POST method cleanup

---------

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>
Co-authored-by: Igor Radovanovic <74266147+IgorWounds@users.noreply.github.com>

* [Docs] Adds Example Notebook for the Implied Earnings Move (#6128)

* adds example notebook for the implied earnings move

* remove commented out line

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* [Feature] - Sync with new openapi.json (#6137)

* feat: create data tabs

* fix: uncomment

* fix: remove unecessary comment

* sort side bar

* comment

* uncomment

* use .dev functions.json + clean traverse

* fix futures example

* fix example

* point to .co

* Fix the v3 markdown generation workflow (#6138)

* fix workflow

* debug time

* see if the docs generate at least

* ahhhh i found it

* getting there

* put the test first

* some keys cleanup

* small change to error message

* update functions.json (#6140)

* fix intraday check in charting ta_class (#6119)

Co-authored-by: Henrique Joaquim <h.joaquim@campus.fct.unl.pt>

* [BugFix] Display EconDB as source for macro (#6143)

* [BugFix] Disable Intrinio `InstitutionalOwnership` model (#6142)

* disable fetcher in init and fetcher tests

* disable provider test params from equity python and api tests

* updated static

* add more examples into findSymbols (#6139)

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* [BugFix] Fix SEC ETF Holdings Parsing Exception (#6121)

* fix etf holdings

* linting

* black

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* Docs/update xl docs (#6145)

* remove print

* update obb.get docs

* example

* md lint

* fix get reference (#6147)

* [BugFix] Filter OECD data using `start_date` and `end_date` parameters (#6144)

* move constant dicts to constants.py

filter by start_date and end_date

* black

* black again again

* Merge 'main' into 'develop' (#6151)

* Release/3.2.4 (#5696)

* test

* ultima description (#5541)

* Updating showcase page (#5543)

* updating showcase page

* use sdk instead

* add new bot command autopost docs (#5530)

add new bot command autopost docs to docs.openbb.co

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* Hotfix/ultima improvements (#5553)

* tweaks for monetization

* fix linting issue

* fix: Modify 'load' behavior to append data to existing sheet (#5503)

* fix: Modify 'load' behavior to append data to existing sheet

Details:
- Added logic to detect existing sheet and append data if present.
- There is no direct way to append on the sheet, so I am reading the sheet to get the last row number and then writing after that row number.

Resolves: #5472

* Fixed 'worksheet not found' error during export to an existing sheet

* Changes:
* Now if the sheet exists in the xlsx file it will ask the user if he
  wants to replace/append/create new sheet

---------

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* added platform API integration test to Nightly build (#5571)

* Add columns to `stocks/search` (#5565)

* Add columns to `stocks/search`

isin, cusip, figi, composite_figi, shareclass_figi

* Update test_search[False].txt

* Update test_search[True].txt

* Delete tests/openbb_terminal/stocks/cassettes/test_stocks_helper/test_search.yaml

* Create test_search.yaml

---------

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* updating ultima urls (#5587)

* fix: A value is trying to be set on a copy of a slice from a DataFrame #5582 (#5591)

* Update README.md (#5598)

* Hotfix/econometrics export filename fix (#5508)

* fix: Modify 'load' behavior to append data to existing sheet

Details:
- Added logic to detect existing sheet and append data if present.
- There is no direct way to append on the sheet, so I am reading the sheet to get the last row number and then writing after that row number.

Resolves: #5472

* Added --export option for exporting to a desired file in /econometrics menu

* Added a -f/--file option to provide the filename for export.
Made -t/--type and -f/--file mutually exclusive.
If -f is not given, -t is required and the filename will be generated based on the time and module.

* Revert `helper_funcs.py` to original state

* Made some arguments not required.

* Minor changes asked by @deeleeramone

* No sheet name

* black

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>
Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* FIX #5600 Updated the Twitter logo to X logo (#5601)

* Change Ultima paragraph (#5624)

* change ultima paragraph

* remove old paragraph

* Update CONTRIBUTING.md (#5623)

* mobile line-brake fixed (#5604)

In the mobile view (media max 400px ), the text line "Installation >>" appears broken into smaller segments.

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* Docs/terminalpro and improvement (#5622)

* add first iteration of content for Terminal Pro

* Update index.md

* Update dashboards.md

* Update folders.md

* Update grouping.md

* Update report.md

* Update templates.md

* Update quick-start.md

* Update index.md

* Update index.md

* Update home.md

* Update news.md

* Update index.md

* Update index.md

* Update index.md

* Update data-provider.md

* Update chat-with-widget.md

* Update data-manipulation.md

* Update forecasting.md

* fix: images self closing tags

* fix: only pro visitors can see pro

* inter font

* improve SEO content and restructuring

* add new package for toggle on tutorials

* move folder around

* improve main page of each prod

* small updates and improvements

* fix logo above

* small improvement in headtitle for portfolio funcs

* fix typos from OpenAI's GPT-4

* fix _category_ json

* remove generic SEO words

* add script that generates SEO for documentation

* add missing file

* remove sdk warning message

* fix links for andrew

---------

Co-authored-by: jose-donato <zmcdonato@gmail.com>
Co-authored-by: jose-donato <43375532+jose-donato@users.noreply.github.com>
Co-authored-by: andrewkenreich <andrew.kenreich@gmail.com>

* Add light blue banner, blue when clicking sidebar + fix link in docusaurus.config.js (#5627)

* typo

* pro docs tab coloring

* sidebar selection blue

* redirecting missing

* fix docs (#5646)

* Fix the redirect for intro to the correct url for docs (#5658)

* fix the redirect for intro to the correct url for docs

* fix code linting

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* Improve Pro documentation further (#5641)

* small improvement

* Fix TutorialVideo for mobile view

* main page terminal pro docs

* jose whelp

* placeholder index file that needs to be removed

* fix: fixes sidebar, removes unused index.md

* update link to api keys

* update api keys link

* fix hyperlinks in terminal helper

* fix a few more links

* fix some more links

* linter

* black linter

* improve main docs page for each product

* small refactor

* header to take to main page of product

* run OpenAI script to generate SEO for pro content

---------

Co-authored-by: jose-donato <43375532+jose-donato@users.noreply.github.com>

* Update folders.md

* Update integrate-your-own-backend.md

* Update index.md

* Update settings.md (#5665)

* fix front matter (#5667)

* #5633 adding comma every 1000 (#5652)

* #5633 adding comma every 1000

* pylint

---------

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* Hotfix/reports model render report sanitize inputs (#5669)

* Adding more robust special character handling to create_output_path().Addresses https://github.com/OpenBB-finance/OpenBBTerminal/issues/5299

* Running black linter on code to conform to OBB style guides.

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>
Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* hotfix/fix-gov-histcont: Fix #5650 - /stocks/gov/histcont (#5676)

* fix histcont

* some cleanup

* remove sort line

* 12-31 not 12-12

* columns_keep_types

* remove space in regex (#5675)

* Pin openai (#5685)

* Clarified stocks/ta empty recom object message (#5468)

* Clarified stocks/ta empty recom object message

* black

* lint

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* improved market cap value extraction (#5169)

* improved market cap value extraction

* pylint

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* 3.2.3->3.2.4

* Changelogs

* new changelog

---------

Co-authored-by: Henrique Joaquim <h.joaquim@campus.fct.unl.pt>
Co-authored-by: Andrew <andrew.kenreich@gmail.com>
Co-authored-by: Adi Sai <adithya.sairam1@gmail.com>
Co-authored-by: Sandip Saha <sandipsmoto@gmail.com>
Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>
Co-authored-by: Luqman <luqazino@gmail.com>
Co-authored-by: Adwait <111136306+AdwaitSalankar@users.noreply.github.com>
Co-authored-by: Abhishek Amar <66067391+abhishek-amar@users.noreply.github.com>
Co-authored-by: DidierRLopes <dro.lopes@campus.fct.unl.pt>
Co-authored-by: professssor <62937125+professssor@users.noreply.github.com>
Co-authored-by: Farookh Zaheer Siddiqui <129654632+FarukhS52@users.noreply.github.com>
Co-authored-by: MAX SHESTOV <57421753+maxxsh@users.noreply.github.com>
Co-authored-by: jose-donato <zmcdonato@gmail.com>
Co-authored-by: jose-donato <43375532+jose-donato@users.noreply.github.com>
Co-authored-by: Mohamad Fadil <fadil.parves@gmail.com>
Co-authored-by: Ramazan <guvencramazan@outlook.com>
Co-authored-by: joshuabuildsthings <joshuabuildsthings@users.noreply.github.com>
Co-authored-by: wojda <114031148+polooner@users.noreply.github.com>
Co-authored-by: Bim Phomthong <118195204+bimbolimbo@users.noreply.github.com>

* Release/4.0.1 (#5840)

* test

* ultima description (#5541)

* Updating showcase page (#5543)

* updating showcase page

* use sdk instead

* add new bot command autopost docs (#5530)

add new bot command autopost docs to docs.openbb.co

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* Hotfix/ultima improvements (#5553)

* tweaks for monetization

* fix linting issue

* fix: Modify 'load' behavior to append data to existing sheet (#5503)

* fix: Modify 'load' behavior to append data to existing sheet

Details:
- Added logic to detect existing sheet and append data if present.
- There is no direct way to append on the sheet, so I am reading the sheet to get the last row number and then writing after that row number.

Resolves: #5472

* Fixed 'worksheet not found' error during export to an existing sheet

* Changes:
* Now if the sheet exists in the xlsx file it will ask the user if he
  wants to replace/append/create new sheet

---------

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* added platform API integration test to Nightly build (#5571)

* Add columns to `stocks/search` (#5565)

* Add columns to `stocks/search`

isin, cusip, figi, composite_figi, shareclass_figi

* Update test_search[False].txt

* Update test_search[True].txt

* Delete tests/openbb_terminal/stocks/cassettes/test_stocks_helper/test_search.yaml

* Create test_search.yaml

---------

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* updating ultima urls (#5587)

* fix: A value is trying to be set on a copy of a slice from a DataFrame #5582 (#5591)

* Update README.md (#5598)

* Hotfix/econometrics export filename fix (#5508)

* fix: Modify 'load' behavior to append data to existing sheet

Details:
- Added logic to detect existing sheet and append data if present.
- There is no direct way to append on the sheet, so I am reading the sheet to get the last row number and then writing after that row number.

Resolves: #5472

* Added --export option for exporting to a desired file in /econometrics menu

* Added a -f/--file option to provide the filename for export.
Made -t/--type and -f/--file mutually exclusive.
If -f is not given, -t is required and the filename will be generated based on the time and module.

* Revert `helper_funcs.py` to original state

* Made some arguments not required.

* Minor changes asked by @deeleeramone

* No sheet name

* black

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>
Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* FIX #5600 Updated the Twitter logo to X logo (#5601)

* Change Ultima paragraph (#5624)

* change ultima paragraph

* remove old paragraph

* Update CONTRIBUTING.md (#5623)

* mobile line-brake fixed (#5604)

In the mobile view (media max 400px ), the text line "Installation >>" appears broken into smaller segments.

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* Docs/terminalpro and improvement (#5622)

* add first iteration of content for Terminal Pro

* Update index.md

* Update dashboards.md

* Update folders.md

* Update grouping.md

* Update report.md

* Update templates.md

* Update quick-start.md

* Update index.md

* Update index.md

* Update home.md

* Update news.md

* Update index.md

* Update index.md

* Update index.md

* Update data-provider.md

* Update chat-with-widget.md

* Update data-manipulation.md

* Update forecasting.md

* fix: images self closing tags

* fix: only pro visitors can see pro

* inter font

* improve SEO content and restructuring

* add new package for toggle on tutorials

* move folder around

* improve main page of each prod

* small updates and improvements

* fix logo above

* small improvement in headtitle for portfolio funcs

* fix typos from OpenAI's GPT-4

* fix _category_ json

* remove generic SEO words

* add script that generates SEO for documentation

* add missing file

* remove sdk warning message

* fix links for andrew

---------

Co-authored-by: jose-donato <zmcdonato@gmail.com>
Co-authored-by: jose-donato <43375532+jose-donato@users.noreply.github.com>
Co-authored-by: andrewkenreich <andrew.kenreich@gmail.com>

* Add light blue banner, blue when clicking sidebar + fix link in docusaurus.config.js (#5627)

* typo

* pro docs tab coloring

* sidebar selection blue

* redirecting missing

* fix docs (#5646)

* Fix the redirect for intro to the correct url for docs (#5658)

* fix the redirect for intro to the correct url for docs

* fix code linting

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* Improve Pro documentation further (#5641)

* small improvement

* Fix TutorialVideo for mobile view

* main page terminal pro docs

* jose whelp

* placeholder index file that needs to be removed

* fix: fixes sidebar, removes unused index.md

* update link to api keys

* update api keys link

* fix hyperlinks in terminal helper

* fix a few more links

* fix some more links

* linter

* black linter

* improve main docs page for each product

* small refactor

* header to take to main page of product

* run OpenAI script to generate SEO for pro content

---------

Co-authored-by: jose-donato <43375532+jose-donato@users.noreply.github.com>

* Update folders.md

* Update integrate-your-own-backend.md

* Update index.md

* Update settings.md (#5665)

* fix front matter (#5667)

* #5633 adding comma every 1000 (#5652)

* #5633 adding comma every 1000

* pylint

---------

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* Hotfix/reports model render report sanitize inputs (#5669)

* Adding more robust special character handling to create_output_path().Addresses https://github.com/OpenBB-finance/OpenBBTerminal/issues/5299

* Running black linter on code to conform to OBB style guides.

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>
Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* hotfix/fix-gov-histcont: Fix #5650 - /stocks/gov/histcont (#5676)

* fix histcont

* some cleanup

* remove sort line

* 12-31 not 12-12

* columns_keep_types

* remove space in regex (#5675)

* Pin openai (#5685)

* Clarified stocks/ta empty recom object message (#5468)

* Clarified stocks/ta empty recom object message

* black

* lint

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* improved market cap value extraction (#5169)

* improved market cap value extraction

* pylint

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* Release/3.2.4 (#5704)

* 3.2.3->3.2.4

* Changelogs

* new changelog

* OpenBB Platform v4 (#5225)

* change location

* check something

* add cache

* fixed auth w/@IgorWounds and @montezdesousa

* fixes for econometrics

* include Darren auth suggestions

* some routers included if dev + chg hub backend var

* coverage router should not be here

* update docker readme

* forgot to check auth here

* Change extension imports

* update docstrings and readme

* headers function wasnt being called, just referenced

* relative import

* econometrics add headers everywhere

* Fix QA

* Fix TA

* Fix test

* Fix python interface QA and TA

* Lint

* Cleanup

* Change path

* make the changes and some etf models

* tests for command_runner

* ruff

* solving naming issues on pytest

* Update some docstrings, fix typos and lint

* solving econometrics instantiating wrong openbb

* chaging the way pytest runs

* messing around with the imports

* ruff

* Change template to add platform considerations (#5533)

* change template to add platform considerations

* @the-praxs suggestions

* solving __pycache__ issue

* removing typo

* ci yml

* making linters happy (i guess)

* Improve Omega Ratio and docstrings

* chaging pro validation hash

* ruff

* Fix OLS Summary and lint

* Add OECD Provider for economy extension (#5534)

* Add record to docs

* update a couple of cboe models

* Fix econometrics bugs. Make tests pass. Lint.

* black

* ruff

* pydantic v2: remove deprecated Extra

* pydantic v2: self.dict -> self.model_dump

* revert pydantic

* revert the revert

* try 1: pin urllib3 < 2

* Revert "try 1: pin urllib3 < 2"

This reverts commit 3736c311cb.

* Revert dep

* Fix

* re-lock

* Skip all tests when doing cspell in pre-commit

* Resolve linter errors

* Add OECD to install script

* Add OECD

* Resolve more linter errors

* Exclude static files of v4 python from ruff

* More linting

* Another attempt to resolve even more linting errors

* Change how Literal is used

* ticker. symbol. consistent. @piiq

* change ticker field in Stock Search to is_symbol

* Update Cboe model for stocks_search -> is_symbol parameter.

* Fix issue

* Lint

* Small QA to rename SDK to Platform (#5544)

* SDK -> Platform in docs

* renamed in core

* renamed in openbb `init`

* `sdk` variable renamed as `platform`

---------

Co-authored-by: hjoaquim <h.joaquim@campus.fct.unl.pt>

* `validator` -> `field_validator` in standard models

* Standard convention for multi-ticker symbols  (#5549)

* standardize symbol input param in forex and crypto historical data

* static

* linting

* Fix stock_search integration test params

* Update dependencies and linting on the CI (#5550)

* Remove ruff and black from platform dependencies

* Update codespell configuration

* Lint python code

* Add toml and tomli to root dev dependencies

* Update requirements.txt files

* Fix buggy docstring in python package methods (#5548)

* fix buggy docstring for methods

* remove space

* fix typing

* avoid name collision with validator

* Revert "avoid name collision with validator"

This reverts commit 8a30a44555.

* typing

* rebuild

* python 3.8, ParamSpec not available

* lint

* ruff noqa

* Merge with develop (#5551)

* Charting extension integration tests (#5547)

* initial proposal for integration tests on the charting extension

* int tests for charting generation

* python generator

* ruff

* boilerplate templates

* python tests

* tests for the api

* using body instead so data don't get redefined

* unit tests for coverage

* ruff

* docstrings

* unnecessary access to keys()

* Add trading economics economic calendar

* dev install

* Data Models disposition on the website (#5538)

* pydantic v4

* flat structure

* md files

* Revert "md files"

This reverts commit bd2170e2ad.

* improve command descriptions

* nice title and description

* human readble data model names on left panel

* implementation details section

* content markdown files

* markdown versioned files

* showcase page

* updates

---------

Co-authored-by: tehcoderer <me@tehcoderer.com>

* everything except tests

* fix `datetime.date` import in fetcher test

* modified url to remove `api_key` placeholder

* record test

* api key is called c here

* fixed fetcher test for api key

* recorded tests

* refactor

* life is ruff

* codespell

* Make Intrinio options chains great again (#5540)

* add multithreading

* cleanup

* linting

* use dateutil.parser

* removed `TICKER_EXCEPTIONS`

* modified `date` and `type` fields in standard model

* set alias for `date` and `type` to match standards

* static

* static

* Hotfix/fix post tests (#5555)

* Fix econometrics

* Fix ta

* Fix qa. Remove intrinio to not be rate limited

* Lint

* Fix typo

* obbject tests (#5558)

* obbject tests

* ruff

* fix dev_install

* fix unittests

* black on nox

* black on nox

* nox

* fix fetcher test so it can compare dates

* fix intrinio tests

* ruff

* change to model_dum() instead (#5561)

* Passing tests (#5560)

* Passing tests

* fix options

* Fix most tests and bugs

* fix forex missing params

* make symbol upper in polygon fx pairs

* fix fixedincome integration tests (#5564)

* Fixing charting api integration tests (#5562)

* fixing charting api integration tests

* black

* fix stocks less five API fails

* fix the 5 failing stocks integration tests

* recapture fmp stock news unit test

* fix fmp forex api test

* couple more forex integration fixes

* changing the request to always use the legacy session. @jmaslek this is was broken for 3.8 and 3.9; with this change apparently all versions work - lemme know if you see any issue with this change

* ruff

* fix ta

* add missing param

* fix testers for the api

* fix globalnews to use images instead of image

* remove unncessary pandas import

* econcal int tests

* update gitignore

* fix get_querystring to support lists

* add missing parameters

---------

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>
Co-authored-by: Henrique Joaquim <h.joaquim@campus.fct.unl.pt>

* Stock Historical standardization and UX improvements (#5457)

* polygon add interval standard param

* attempt at standardization

* Alpha Vantage params standardization

* rename `get_intervals` and fixed bug for extracting wrong value

* missed changing it in `extract_data`

* linting

* polygon cleanup

* modified `start_date` & `end_date` type to `datetime` from `dateType`

* standardized intrinio stock historical params

* fixed yfinance

* linting

* revert start and end date to dateType from datetime

* fix alpha vantage

* changed param

* fix intrinio infinite loop

* `validator` is `field_validator`

* revert time params and fixed intervals

* `outputsize` alias set in `Field`

* set interval description using in-build dict

* yfinance stock param standardization

* static

* intrinio cleanup

* Polygon add api params as `PrivateAttr`

* modified yfinance params to `PrivateAttr`

* intrinio remove overriding date params

* fixed description for AV adjusted param

* fix alpha vantage intraday not working

* re-record and pass unit tests

* pass integration tests

* Update test

---------

Co-authored-by: Igor Radovanovic <74266147+IgorWounds@users.noreply.github.com>

* Fix econcal endpoint (#5567)

* fix econcal for python interface

* fix econcal tests

* fix non supported json output

* Fixes FMP index endpoint to get 1 day interval data (#5535)

* fix FMP endpoint for getting index data for `1day` interval

* fix `open` type to prevent validation error

* modifed `validator` to `field_validator`

* type of fields is `StrictFloat`

* re-record and pass test

* linting

* `validator` -> `field_validator`, a3'd everything

* Fix regression in tests. Lint

* updating the integration test actions

* Fix typo

* Fix typo

* Fix import

* Publish versions `a3` to Pypi (#5568)

* bump versions and script to publish

* small improvements

* uncomment exception

* openbb

* publishing guidelines

* improve docs

* add providers to economy api

* make symbol required in stock_quote

* Fix keys

* Fix oecd

* add comma

* revert

* Add quotes

* Sleep

* Improve tests

* Improve

* Feature/query transform (#5566)

* inject query into transform_data

* swap query and data

* update missing fetchers

* last test failing

* sig test fetcher

* black

* av add `query` to `transform_data` call

* transform_data: missing refactoring

* fix uvicorn command in the doc

* Automated examples (#5576)

* Automated examples

* Fix bug

* Lint

* static assets built on 3.8

* Revert "static assets built on 3.8"

This reverts commit fb82ff7bd9.

* remove assets

* static assets on 3.8

* static assets on 3.8

* Wrap in quotes

* Improve wrap

* minor fix

* rebuild 3.8

* Fix bug

* Fix edge case

* Lint

* rebuild

---------

Co-authored-by: hjoaquim <h.joaquim@campus.fct.unl.pt>
Co-authored-by: Diogo Sousa <montezdesousa@gmail.com>

* use SecretStr for sensitive fields (#5578)

* use SecretStr for sensitive fields

* fix

* docstring

* Add tests for provider (#5583)

* Add tests for provider

* Test standard models

* Lint

* Fix standard model issues.

Co-authored-by: @the-praxs

* linting

* `symbol` -> `is_symbol` in `economy.index_search`

* static

---------

Co-authored-by: Pratyush Shukla <ps4534@nyu.edu>

* feature/v4-biztoc-provider: Adds Biztoc provider and GlobalNews (#5546)

* add biztoc provider

* duplicate field

* black

* request headers

* recapture test

* move stuff around

* add integration tests

* duplicate OECD entry in dev_install

* duplicate entry for oecd in pyproject.toml

* Add Biztoc to workflow

* update readme: credentials

* Add Preferences to README

* improve tests

* fix biztoc

* obbject pylint

* Bug fix: serializ SecretStr before dumping to json

* export credentials as json (#5592)

* Check for wrong, changed, or outdated params. (#5597)

* Improving `economy.cpi` for the FRED data (#5602)

* remove multiindex using coding magic

* Update field descriptions

---------

Co-authored-by: Theodore Aptekarev <aptekarev@gmail.com>

* Improve error message.

* Improve

* Test tests (#5607)

* Test tests

* Add test

* 4.0.0a4 PyPI publishing (#5608)

* changes to publish

* fix mini conflict

* typo

* cleaned `descriptions.py`

* fix description in `cpi.py`

* Add ETF extension to v4 (#5609)

* Add etf extension with search data model

* Add FMP ETF search

* Add fetcher tests

* Add ETF historical price from yfinance (#5610)

* Add etf extension with search data model

* Add FMP ETF search

* Add fetcher tests

* Add etf historical price standard model and route

* Add etf historical data from yfinance

* Add unit tests

* Fix linter error. Organize imports

* Update test data

* Revert common descriptions erased by CPI PR

* remove `symbol` from data for `stocks.ca.peers`

* Test docstrings (#5613)

* Test docstrings

* fix minor bugs w/@IgorWounds

* static only w/@IgorWounds

* fix `importance` param in `economy.econcal`

* right static only w/@IgorWounds

* Fix econcal and add tests to noxfile

* fix econcal static

* re-record

---------

Co-authored-by: Pratyush Shukla <ps4534@nyu.edu>

* Clean up the newly created ETF extension (#5611)

* Add ETF to dev_install bundle

* Add integration tests for openbb-etf

* Patch integration test generator scripts to produce lintable code

* Add py.typed markers for extensions

* user preferences descriptions (#5593)

* user preferences descriptions

* typo

* Feature/v4-sec-provider: Add SEC provider to V4 (#5556)

* coerce string type, add default=None

* add sec provider and FTD function

* update description

* ruff

* data model definitions in standard model already

* duplicate oecd entries in dev_install and pyproject

* actually commit

* integration test

* add stock search to SEC

* add company filings for SEC provider

* integration tests

* remove duplicated line

* ruff

* **kwargs in Transform Data

* some helper functions for later

* black

* add helper for mutual fund and etf lookup

* add some SEC lookup helpers to a new regulators extension

* black

* ruff

* fix test param

* take institution filter out of sec stock_search

* ruff

* remove URL that codespell doesn't like

* add cik-to-ticker map

* add litigations rss feed

* make openbb-regulators a dependency of openbb-sec

* add SIC search

* improve ftd and remove BeautifulSoup from SEC dependencies

* fix parsing the zip files pre-2009

* ruff

* update descriptions

* recapture ftd test

* tests params

* typo

* change limit param to 1 in test_sec_stock_ftd

* Add didiers to_records method (#5588)

* Add didiers to_records

* unused import

* Bring back polars + do what igor suggested

* Make the output consistent with what it was.

* Black ?

* spelling.  oops.  almost like we have a pre commit or something

* Feature/v4-price-performance: % returns for various horizons, comparable across multiple tickers. (#5618)

* coerce string type, add default=None

* add price_performance endpoint from FMP

* black

* extra period in docstring

* ruff

* forgot one file

* Feature/ipo-calendar: V4 IPO Calendar (#5605)

* coerce string type, add default=None

* add intrinio ipo calendar

* fix typo

* sort __init__

* fix description

* make requested changes

* update test params

* black

* test updates

* etf integration

* Add disc router and WSJ provider

* OBBject extensions (#5612)

* changes to core

* bug?

* docstring

* doc

* Update credentials.py

* fix model_dump

* create extensions

* fix docstring

* doc

* revert change query_exc

* doc

* fix container test

* redirect obbject test patch

* doc

* rename method

* move decorator to extension file

* rename method extend_obbject

* changes in creds model

* avoid credential racing

* doc

* rename prop

* doc

* doc

* remove comment

* comment some code

* free extension names

* docstring

* doc

* docs

* Hotfix/fix-v4-economic-calendar: Add FMP to Econ Calendar and fix some TE bugs (#5580)

* coerce string type, add default=None

* adds fmp to economic calendar and fixes some things with TE

* ruff

* etf integration test

* missing provider for test

* rename te cassette to economic_calendar

* fix test

* rename tests

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* Add stocks disc endpoints (#5630)

* Add stocks disc endpoints

* Explicit

* Lint

* Add tests

* Bump timeout

* Feature/disc router (#5616)

* Add disc router and WSJ provider

* Lint

* Lint

* Move endpoints under ETF router

* Update etf_performance.py

* Add ETF info and sector exposure from FMP (#5620)

* Add ETF info from FMP

* Add integration tests

* Add unit tests

* Add etf sectors endpoint

* Add etf sectors fmp endpoint

* Omit nested sector list from fmp info data

* Update tests and test data

* Update test data

* Specify provider explicitly in integration tests

* Update tests

* Skip yfinance etf historical test

* Fix linter errors after the merge

* Fix more linter errors

* Fix failing and add missing tests

* Fix integration tests

* Fix calendar ipo tests

* Add ETF holdings and holding report dates from FMP (#5629)

* Add ETF info from FMP

* Add integration tests

* Add unit tests

* Add etf sectors endpoint

* Add etf sectors fmp endpoint

* Omit nested sector list from fmp info data

* Update tests and test data

* Update test data

* Specify provider explicitly in integration tests

* Update tests

* Skip yfinance etf historical test

* Add etf holdings and holdings date routes and standard models

* Add fmp holdings and holding dates fetchers

* Update holdings fetcher to properly handle date object inputs

* Add unit tests

* Add integration tests

* Black after merge

* Fix test input data format

* Disc router for equities. (#5640)

* Disc router for equities.

* add tests

* Bug fixes in financial statements (#5537)

* `validator` to `field_validator` in standard models

* Polygon add `ttm` to `period` param

* Revert "Polygon add `ttm` to `period` param"

* Polygon add `ttm` to param `period`

* `EarningsCallTranscript` standard model removed `quarter` param

* use `tag` instead of `name` for correct field name

* fix url for intrinio income statement

* modified intrinio cash flow `transform_data` function

* set optional data fields and validate dates

* fix insider trading not fetching recent data

* linting

* refactoring

* refactoring

* set param alias in fmp stock insider provider model

* get all quarter data for intrinio

* add `limit` param to `stocks.fa.ins`

* refactoring and linting

* removed additional comment

* linting

* re-recorded and fix tests

* cash flow standard model field type to `StrictFloat`

* Intrinio cash flow model fixed to get all quarters' data

* linting

* refactoring

* financial statement standard model field type to `StrictFloat`

* Intrinio financial statement model get all quarter data

* set `period` type to `Optional[str]`

* add `period` param to balance sheet provider models

* add `period` param to cash flow provider models

* use alias in model dump

* remove `ttm` from `period` param

* revamped intrinio financials

* fixed incorrect description for `limit` param

* modified tests for `balance`, `cash` and `income`

* re-recorded tests

* skipped intrinio financial statement fetcher tests

* Add ETF county weighting command with FMP data (#5645)

* Remove code that wrongfully infers the filing date as end of quarter

* Add etf countries command with fmp data

* Add countries command to etf router

* Update sector and country weighting fmp models

* Fix docstring typo

* Add unit tests and update mock data

* Add integration tests for etf.countries

* Bring back missing integration tests

* Fix economic calendar integration tests

* fix missing data in intrinio financials

* removed inutile params from Polygon financial statement models

* Intrinio IPO overhaul

* tests fixed and re-recorded

* Add ETF holdings performance with FMP data (#5659)

* Add etf holdings performance endpoint

* Add tests

* Fix failing polygon fetcher tests

* Split long holdings lists into chunks and update test data

* Change chunk length to 500 tickers

* Add stock search from FMP (#5642)

Co-authored-by: Theodore Aptekarev <aptekarev@gmail.com>

* Feature/v4 pypi (#5579)

* improvements to the publish script

* move scripts

* initial script that builds the package

* adjustments on the docs

* test pypi workflow

* add hour and minute to version

* black

* ruff

* duplicate file

* moving files instead

* build throught the script instead

* update nox file

* integration tests workflow

* change workflow

* fix dev_install

* fix in sed cmd

* testing the platform only

* commenting out the publishing

* fix run commands

* improvements to the publish script and doc

* fix publish script

* updating the script and the workflow

* feature/v4-pypi nightly (#5589)

* init

* Update nightly.py

* cleanup

* init file

* not referencing dev_utils

* updating obb platform key

* revert

* Increase warmup time

* project name on pyproject and adding the --pre flag on readme

* moving to the build folder

* fix noxfile

* reflecting the moves on the import statements

* fix pyproject path

* ruff

* fix paths

* fix super typo

* update workflows

* moving the pypi terminal files into its folder

---------

Co-authored-by: teh_coderer <me@tehcoderer.com>
Co-authored-by: Igor Radovanovic <74266147+IgorWounds@users.noreply.github.com>
Co-authored-by: Theodore Aptekarev <aptekarev@gmail.com>

* economy.calendar nasdaq (#5628)

* add quandl econcal

* unittest

* add integration tests

* replace &nbsp

* fix bug TE importance

* small fix

* remove stuff from quandl

* add nasdaq provider

* add nasdaq to install files

* Update test_economy_api.py

* test

* quandl leftovers

* remove quandl econcal test

* lint

* replace &nbsp

* ruff

* lint

* update nasdaq gmt

* validate countries

* here

* no need for this

* fix calendar tests

* static

* static

* remove error

* static

* readme + website

* readme

* remove changes TE

* typo

* fix unittest

* fix tests

* Update __init__.py

* Update economic_calendar.py

* revamped model and added helpers for date range

---------

Co-authored-by: Pratyush Shukla <ps4534@nyu.edu>

* Feature/v4-dividend-calendar: Adds Nasdaq dividend calendar and moves the command out of fundamental_analysis (#5672)

* add nasdaq dividend calendar and move out of fundamental_analysis

* black

* ruff

* add random-user-agent to pyproject.toml

* add nasdaq ipo calendar

* add nasdaq flag for SPO to IPO calendar

* add spo test param

* black

* dividend test

* fix transform_query - nasdaq_calendar_ipo

* bring back different headers for ipo calendar

* remove references to test

* Add disc endpoints (#5660)

* Add Quandl top retail

* Add tests

* Add Seeking Alpha

* Add Filings

* Change Provider

* Fix test

* Add dev_install

* static

* static correct

* y u forget me

* updated deps

* static

---------

Co-authored-by: Pratyush Shukla <ps4534@nyu.edu>

* using dev install (#5673)

* Expanding the `fixedincome` menu (#5655)

* models and fetcher

* making the fetcher visible

* std model

* adding the command

* improvements to the models

* typo

* adding the command

* imrpovements to dwpcr

* standard model for ecb int rates

* adding the fetcher for fred

* standard model of ice bofa

* add ice_bofa command

* ruff

* ice bofa fetcher

* moody std model

* moody

* std model

* add cp command

* unused import

* cp

* spot

* ruff

* hqm

* tmc

* ffrmc

* tbffr

* ruff

* typing fixes

* buggy dates

* fred fetcher tests

* ruff

* adding ecb as a provider

* adding new provider related stuff

* adding the new command eu_ycvr

* models

* test for ecb fetcher

* Feature/v4 fixedincome int tests (#5661)

* integration tests

* happy ruff

* mini adjustment

* ruff

* typo

* verbose type

* fix tests

* docstrings everywhere and an happy pydocstyle

* fix tests

* Feature/user-cache-dir: Add User Preference for the Cache Directory (#5621)

* add user preference for storing cached data

* ruff

* fix tests?

* fix test key error..?

* etf integration test fix

* add cache dir to SEC

* black

* improve docs structure for V4 (#5586)

* improve docs structure for V4

* moving over changes

* Add Docs to the navbar

* add some, re-order sidebar

* fix sentence

* add docs install to source instructions

* insert a couple of links, fix a typo

* big terminal docs usage revamps

* improve toggle component for tutorials

* get rid of TOCs for all terminal docs for consistency and simplicity

* improve platform docs

* improve bot docs content structure

* remove index from bot docs

* improve ON THIS PAGE to only display TOC if at least 1 element

* make edit this page text and icon on same line

* add author and date to docs

* add basic syntax page to usage

* sidebar order for usage section

* move api start command to the top with the import python import

* some install page additions

* small improvement

* refactor routines section

* Update introduction-to-routines.md

* Update pipeline-of-commands.md

* Update introduction-to-routines.md

* Update routines-for-power-users.md

* Update community-routines.md

* update keyboard hotkey macros

* askobb page

* Update askobb-feature.md

* Update index.md

* Update index.md

* Update index.md

* Update index.md

* fix images

* remove HeadTitle from all files where it exist

* get rid of extra spacing in markdown

* small update

* small edit

* need index at the start because of main path when homepage of docs

* use openAI to create SEO for front matter on my behalf

* remove table of content from sdk page

* finish improving metadata SEO for SDK docs

* update duplicate header

* update HeadTitle to be more specific on product

* remove index from bot telegram reference

* delete duplicated title for platform reference

* another update

* Hotfix/econometrics export filename fix (#5508)

* fix: Modify 'load' behavior to append data to existing sheet

Details:
- Added logic to detect existing sheet and append data if present.
- There is no direct way to append on the sheet, so I am reading the sheet to get the last row number and then writing after that row number.

Resolves: #5472

* Added --export option for exporting to a desired file in /econometrics menu

* Added a -f/--file option to provide the filename for export.
Made -t/--type and -f/--file mutually exclusive.
If -f is not given, -t is required and the filename will be generated based on the time and module.

* Revert `helper_funcs.py` to original state

* Made some arguments not required.

* Minor changes asked by @deeleeramone

* No sheet name

* black

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>
Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* FIX #5600 Updated the Twitter logo to X logo (#5601)

* Change Ultima paragraph (#5624)

* change ultima paragraph

* remove old paragraph

* Update CONTRIBUTING.md (#5623)

* mobile line-brake fixed (#5604)

In the mobile view (media max 400px ), the text line "Installation >>" appears broken into smaller segments.

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* start PR so screenshots can get attached via GitHub website

* Update qa.md

* Docs/terminalpro and improvement (#5622)

* add first iteration of content for Terminal Pro

* Update index.md

* Update dashboards.md

* Update folders.md

* Update grouping.md

* Update report.md

* Update templates.md

* Update quick-start.md

* Update index.md

* Update index.md

* Update home.md

* Update news.md

* Update index.md

* Update index.md

* Update index.md

* Update data-provider.md

* Update chat-with-widget.md

* Update data-manipulation.md

* Update forecasting.md

* fix: images self closing tags

* fix: only pro visitors can see pro

* inter font

* improve SEO content and restructuring

* add new package for toggle on tutorials

* move folder around

* improve main page of each prod

* small updates and improvements

* fix logo above

* small improvement in headtitle for portfolio funcs

* fix typos from OpenAI's GPT-4

* fix _category_ json

* remove generic SEO words

* add script that generates SEO for documentation

* add missing file

* remove sdk warning message

* fix links for andrew

---------

Co-authored-by: jose-donato <zmcdonato@gmail.com>
Co-authored-by: jose-donato <43375532+jose-donato@users.noreply.github.com>
Co-authored-by: andrewkenreich <andrew.kenreich@gmail.com>

* Add light blue banner, blue when clicking sidebar + fix link in docusaurus.config.js (#5627)

* typo

* pro docs tab coloring

* sidebar selection blue

* redirecting missing

* Update crypto introduction.md

* titles

* another title

* Update crypto dd.md

* Update defi.md

* Update disc.md

* Update nft.md

* Update onchain.md

* Update ov.md

* Update introduction.md

updates URL

* Update introduction.md

add HeadTitle

* Update ba.md

* Update comparison.md

* Update dark-pool-shorts.md

* Update disc.md

* fix docs (#5646)

* Update fa.md

* Update gov.md

* Update ins.md

* Update tradinghours.md

* Delete website/content/terminal/menus/stocks/options/screener.md

* Update Options introduction.md

* Update screener head title

* Update screener TOC

* Update econometrics.md

* Update economy head title

* Update etf.md

* codespell

* Fix the redirect for intro to the correct url for docs (#5658)

* fix the redirect for intro to the correct url for docs

* fix code linting

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>

* Contributing guidelines (#5615)

* intro section

* what's expected sections

* improve sections

* improve sections and move how to add a data point

* moving qa under contributor

* better intro

* moving the sharing section

* better expectations

* small intro to the contributor guidelines

* moving things under the contributor section

* there is no pre commits on the platform yet

* dependency management

* best practices for extensions

* typo

* collapse options subfolder

* codespell

* Improve Pro documentation further (#5641)

* small improvement

* Fix TutorialVideo for mobile view

* main page terminal pro docs

* jose whelp

* placeholder index file that needs to be removed

* fix: fixes sidebar, removes unused index.md

* update link to api keys

* update api keys link

* fix hyperlinks in terminal helper

* fix a few more links

* fix some more links

* linter

* black linter

* improve main docs page for each product

* small refactor

* header to take to main page of product

* run OpenAI script to generate SEO for pro content

---------

Co-authored-by: jose-donato <43375532+jose-donato@users.noreply.github.com>

* Update folders.md

* Update integrate-your-own-backend.md

* Update index.md

* Update fixedincome.md

* url

* Update forecast.md

* Update settings.md (#5665)

* Update forex.md

* fix front matter (#5667)

* Fix conflict markers that got into markdown during previous merges

* Fix front matter list formatting

* Fix typo

* small thing

* fix hyperlinks

* Update .codespell.ignore

* output_type options

* Architectural considerations

* Update sidebar to hide pro section by default

* Minor copy changes

* TET pattern

* import statements

* clarification on preferences

* Minor copy changes

* Lint markdown and python

* fix: open index on collapsible root elements

* remove old content

* options title

* Fix linting errors

* Refactor sidebar label to url mapping condition

---------

Co-authored-by: andrewkenreich <andrew.kenreich@gmail.com>
Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>
Co-authored-by: hjoaquim <h.joaquim@campus.fct.unl.pt>
Co-authored-by: Sandip Saha <sandipsmoto@gmail.com>
Co-authored-by: James Maslek <jmaslek11@gmail.com>
Co-authored-by: professssor <62937125+professssor@users.noreply.github.com>
Co-authored-by: Farookh Zaheer Siddiqui <129654632+FarukhS52@users.noreply.github.com>
Co-authored-by: MAX SHESTOV <57421753+maxxsh@users.noreply.github.com>
Co-authored-by: jose-donato <zmcdonato@gmail.com>
Co-authored-by: jose-donato <43375532+jose-donato@users.noreply.github.com>
Co-authored-by: Mohamad Fadil <fadil.parves@gmail.com>
Co-authored-by: Theodore Aptekarev <aptekarev@gmail.com>

* Add multi-ticker support to etf_countries (#5670)

* add multi-ticker support to etf_countries

* remove commented-out lines

* normalized percent points

* percent -> percentage

* Fix Nasdaq Econ Calendar Fetcher (#5668)

* fix nasdaq econ cal request

* more html tags

* add random-user-agent to pyproject

* another edge case - 24H -> 00:00

* Feature/v4-unusual-options: Adds unusual options endpoint. (#5677)

* add unusual options fro Intrinio

* test params

* move sorting

* Change type

* update FMP fetcher to use the V4 endpoint (#5681)

* Fix price performance not parsed correctly when there is only 1 ticker (#5678)

* Add crypto search standard model and FMP data for it (#5679)

* Apply renaming to news data (#5684)

* inegration complete

* reformatted for black

* fix ruff issues

* revised integration

* changes to versions in pyproject.toml and updated lock files

* Rename models

* Update tests

* Fix typos here and there

* Update routers

* Apply rename to ultima

* Properly mock api key

* Mock ultima api key

---------

Co-authored-by: Adi Sai <adithya.sairam1@gmail.com>

* Feature/v4-market-snapshots: Adds full market snapshots (#5473)

* add stocks market snapshots from fmp

* add fmp fetcher test

* unused imports

* add polygon to market_snapshots

* update fmp market_snapshots

* black

* pydantic2 things

* updates

* change validate_obj to model_validate

* polygon cleanup

* tests

* descriptions

* black

* data description

* minor adjustments, using less local variables, improving readability and removing pandas

* better structure

* minor adjustments

* lint

* fix int tests

---------

Co-authored-by: hjoaquim <h.joaquim@campus.fct.unl.pt>

* Add NBBO Quotes, with historical, to `obb.stocks.quote()` (#5617)

* coerce string type, add default=None

* add polygon stock quote

* limit cleanup

* add greater/less than to params

* limit param

* black

* tests

* fix tests

* test_etf

* rename model and function to NBBO

* add standard model for nbbo

* improve standardization

* not redifining builtin max

* fix input params

* integration test params

* removing unused import

Co-authored-by: Pratyush Shukla <ps4534@nyu.edu>

* revamped code

* re-recorded nbbo test

* added symbol validator

* added alias in field

* Stock news -> Company news

---------

Co-authored-by: hjoaquim <h.joaquim@campus.fct.unl.pt>
Co-authored-by: Pratyush Shukla <ps4534@nyu.edu>
Co-authored-by: Theodore Aptekarev <aptekarev@gmail.com>

* hotfix/update-quandl-to-nasdaq: Merge Quandl + Nasdaq (#5682)

* merge quandl nd nasdaq

* dev_install.py

* test param name

* economy python integration test

* Add more input types to data processing commands (#5453)

* adding more supported types to data processing commands

* ruff

* removing unused import

* docstrings

* data processing types covertion

* data description

* test filters

* ruff

* fix package builder tests

* ruff

* removing print

* Add stocks/dps module (#5638)

* short volume

* pyproject missed

* add sec short interest with days to cover

* Add FINRA OTC endopoint

* some tests

* rest of unit tests

* Integratin test scripts

* dev install

* linters caught by pre commit

* lines too long

* Last test files

* maybe it needs a lock file?

* thanks pratty

* Maybe ill actually listen to pratty next time

* sec test date freeze

* I cant hit buttons

* re record tests

* tests 3

* Stockgrid test

* regen int test

* more testing

* lint

* moved these fns

* dis killing me

* one more test

* last one

* Update integration tests

* Fix tests picking up wrong dps route

---------

Co-authored-by: Theodore Aptekarev <aptekarev@gmail.com>

* Rename Forex to Currency (#5687)

* restore fields as optional (#5686)

* Rename Crypto (#5689)

* added `crypto.price.historical`

* static

* fix tests

* fix incorrect label

* Move model from sec to finra (#5692)

* Update toml

* fixed deps

* add openbb-etf lock file

* cleanup

* cleanup (#5694)

* Rename Fixed Income (#5695)

* Rename Fixed Income

* Add routers

* Rename stocks to equity (#5688)

* Rename stocks folder

* Rename openbb_stocks package

* Rename routers

* Reroute equity menu

* Update routes based on feedback

* StockNews -> EquityNews

* StockInfo -> EquityProfile

* Rename Equity calendars

* Equity peers rename

* Rename discovery

* Rename fundamental

* Rename price command models

* Rename shorts

* Update charting router

* Fix seeking alpha upcoming release date

* Fix linting errors

* Update tests

* Fix tests

* Skip econometrics api tests because it's python only

* Update integration tests and test data

* Historical analyst estimates

* Skip cboe options chain test. Unable to record vcr cassette

* re-record cboe cassettes

* Document CBOE unit test creation caveats

---------

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* Feature/files-for-igor: Adding a few standard models for Igor (#5699)

* add some standard models for Igor

* codespell

* update descriptions

* Add devtools metapackage (#5703)

* Add openbb-devtools metapackage

* adding devtools to the dev_install script

---------

Co-authored-by: hjoaquim <h.joaquim@campus.fct.unl.pt>

* Rename econometrics commands to be more verbose (#5702)

* Expose command to model map (#5706)

* Rename ta qa (#5708)

* Rename ta folder

* Rename ta to technical

* Rename qa folder

* Rename qa to quantitative

* Rename the actual routes in quantitative

* Equity cleanup (#5707)

* cleanup

* static

* renaming to equity cc @the-praxs

* Proof reading

---------

Co-authored-by: hjoaquim <h.joaquim@campus.fct.unl.pt>
Co-authored-by: Theodore Aptekarev <aptekarev@gmail.com>

* Don't enforce imperative for first word of the docstring

* Remove ultima from default extensions

* Rename `economy` and create `index` extension (#5705)

* renamed economy and re-recorded tests

* static

* market indices

* european indices

* fred indices

* index constituents

* tests

* correct package name

* oops

* static

* linting

* Move COT commands to regulators

* Rename economy.list to available_indices

* Rename GDP models and clean up

* Fix charting tests

* Drop _indices from available_indices command

* Fix obbject to_dict test

---------

Co-authored-by: Theodore Aptekarev <aptekarev@gmail.com>

* Rename etf/disc (#5715)

* Docs/v4-working-docs: PR to Work on V4 Docs (#5693)

* Revert changes related to pydantic2 made into v3 code

* Revert fred model changes in v3

* Revert pydantic2 databento model changes in v2

* Update some names and v3 lock file

* Clean up openbb/package before the beta release

* Platform publishing procedure (#5701)

* update instructions

* introducing the version script

* publish message

* fix platform path

* reflecting version changes

* getting version from installed packages instead

* removing toml from dependencies

* Changes to get `a7` out (#5697)

* changes to get a7 out

* etf menu on the pyproject.toml

* poetry lock

* bump `openbb-etf` to `a5` to match version

* updated deps

* removing ultima

* removing references to a4 cc @the-praxs

* Bump index menu version

* poetry lock

---------

Co-authored-by: Pratyush Shukla <ps4534@nyu.edu>
Co-authored-by: Theodore Aptekarev <aptekarev@gmail.com>

* Update publishing instructions

* Bump core dependencies in extensions and providers

* Add changes to the publish script following the beta release

* Update the versions in static extension map and openbb metapackage

---------

Co-authored-by: James Maslek <jmaslek11@gmail.com>
Co-authored-by: Pratyush Shukla <ps4534@nyu.edu>
Co-authored-by: hjoaquim <h.joaquim@campus.fct.unl.pt>
Co-authored-by: Diogo Sousa <montezdesousa@gmail.com>
Co-authored-by: Igor…

* Close the quote (#7309)

* relock the repository (#7310)

* relock the main openbb package lock, which was missed in the previous pr (#7313)

* more lock updates (#7314)

* fix(cli): prevent internal type annotations from leaking into help descriptions (#7302)

* fix(cli): prevent internal type annotations from leaking into help descriptions

* fix(cli): remove whitespace from blank line to satisfy linting rules

* fix(cli): resolve linting issues in argparse_translator.py

---------

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* [Security] `openbb-core`: Regenerate Lock File (#7322)

* update openbb-core lock file

* relock devtools

* update desktop/package-lock.json and fix test mocks for higher version of jsdom

* react-markdown 9 doesn't include  anymore

* lodash in frontend-components/plotly

* ensure pandas below v3

* seroval override in desktop package.json

* patch core version

* relock packages for protobuf dependabot alert (#7335)

* [BugFix] Remove Polygon - No Longer Exists and Can't Support (#7337)

* remove polygon - no longer exists and can't support

* update metadata asset

* dev_install.py

* chore: Update outdated GitHub Actions versions (#7326)

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* fix: update outdated examples in openbb-apachebeam (#7286)

- examples/openbb-apachebeam/README.md: Fix test-run command to point to example tests directory, fix typos and grammar
- examples/openbb-apachebeam/requirements.txt: Add core 'openbb' dependency required by example
- examples/openbb-apachebeam/tests/test_obb_pipeline.py: Fix code formatting, fix 'Print nes' typo to 'Print news'

Fixes #7181

Co-authored-by: Murat Aslan <murataslan1@users.noreply.github.com>
Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* [Feature] Pandas 3.0 - Update Library For Syntax (#7336)

* fix everything for pandas 3.0

* bump yfinance version

* update core dependency python-multi-part

* lint

* Add github action that validates all changes have a formal description (#7340)

* [Feature] Add Svensson Nominal Yield Curve From Federal Reserve (#7338)

* add svensson yield curve, cleanup fomc documents endpoint, make provider a standalone extension, move apps from platform-api into extension, and update some models to add prefix/suffix in widget definitions

* black

* do what codeQL says

* black again

* use get_args instead of Literal.__args__

* linting

* test items

* grammar police

* test functions need to start with test_*

* test params

* add total factor productivity from sf fed

* openpyxl required for Excel file handling

* grammar police

* inflation expectations

* Update pr-description check action to properly handle quotes

---------

Co-authored-by: deeleeramone <>
Co-authored-by: Theodore Aptekarev <aptekarev@gmail.com>

* [BugFix] Support PEP 649 & PEP 749 (Enable Python 3.14) (#7343)

* enable future support for python 3.14

* removing calling type on standard and extra

* [BugFix] `openbb_platform_api` & `openbb_mcp_server`: Fix `app_import` path handling for drive paths. (#7344)

* fix colon handling in import_app to handle drives and add cross-platform tests

* the test collection helper too

* [BugFix] ODP Desktop: Fix Dependabot Alerts, Fix Environment Update, Improve Error Message (#7345)

* improve error message on initial environment setup before continue anyway, add openbb-cookiecutter to the extra extensions lists, seperate conda env update into updating conda packages and pip packages, update glob for dependabot

* conda update -> conda install for updating the individual packages

* cargo fmt

* cargo clippy

* chore(lint): migrate from pydocstyle to ruff (#7346)

pydocstyle has been deprecated and archived (Nov 2023). The maintainers
recommend migrating to Ruff, which has full parity with pydocstyle.

Changes:
- Add "D" (pydocstyle) rules to ruff.toml select list
- Migrate ignore rules from .pydocstyle.ini to ruff.toml
- Remove pydocstyle from CI workflow
- Delete .pydocstyle.ini

This also fixes false positives like D402 being triggered incorrectly
when a method named `name` has a docstring containing "name" anywhere.

See: https://github.com/PyCQA/pydocstyle (archived)

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* feat(platform): standardize country inputs with ISO 3166 Country type (#7333)

* feat(platform): standardize country inputs using pydantic-extra-types

Add CountryParam type that accepts:
- ISO 3166-1 alpha-2 codes (e.g., 'US', 'us')
- Full country names (e.g., 'United States')
- lower_snake_case names (e.g., 'united_states')

Uses pydantic-extra-types for programmatic ISO validation instead of
hardcoded mappings. Serializes to lowercase alpha-2 code by default.

Example usage in FMP equity screener:
- country: CountryParam | None

Partially addresses #6969

* refactor: use Country class inheriting from str for type checker compatibility

- Replace Annotated CountryParam with Country(str) class per maintainer feedback
- Store full pycountry object for access to alpha_2, alpha_3, name, numeric
- Use pycountry directly instead of pydantic-extra-types
- Add support for alpha-3 codes (e.g., 'USA')
- Update tests to cover new functionality

* Replace pycountry with static JSON country data

- Add country_data.json with ISO 3166-1 data (20KB vs 6MB pycountry)
- Update Country class to use static JSON lookup
- Remove pycountry dependency from pyproject.toml
- No decompression needed, data loaded once at import

* fix(lint): D402 docstring should start with verb

* fix(lint): work around pydocstyle D402 false positive

When a method is named `name` and `.pydocstyle.ini` uses `match = .*\.py`,
pydocstyle incorrectly triggers D402 for any docstring containing the word
"name" anywhere - not just at the start as the rule intends.

Changed docstring from "Return the full country name..." to
"The country's official designation..." to avoid the false positive.

See: https://github.com/PyCQA/pydocstyle/issues/375

* fix(lint): resolve pylint warnings in country_utils

* fix(lint): use ternary for ruff SIM108 in country_utils

* fix(lint): format ternary for black line length

---------

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* chore(ci): remove redundant bandit dependency (#7347)

Ruff's S (flake8-bandit) rules already cover bandit's checks and are
enabled in ruff.toml. Bandit was also non-blocking (|| true) in CI,
making it purely informational. Remove it to reduce dependencies and
speed up the linting pipeline.

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* update some lock files for dependabot (#7358)

* feat(platform): extend Country type standardization to equity screeners (#7355)

* feat(platform): extend Country type standardization to equity screeners

Apply standardized Country type from PR #7333 to additional providers.

Changes:
- YFinance equity_screener: Accept ISO codes, country names; convert to lowercase
- Nasdaq equity_screener: Accept ISO codes, preserve multi-value functionality
- Nasdaq economic_calendar: Accept ISO codes, convert to snake_case
- Seeking Alpha calendar_earnings: Accept ISO codes for US/CA

All validators normalize inputs to provider-expected formats while maintaining
backward compatibility with existing inputs.

Part of extending #6969 coverage across providers.

---

Question for maintainers: Should this be extended to moderate (OECD, EconDB)
or full (all providers) coverage?

* refactor: apply review feedback - validate invalid country input instead of passing to API

* refactor(yfinance): validate country input against supported list

* dedupe nasdaq screener choices definitions

* fix(nasdaq): use get_args() instead of __args__ for mypy compatibility

---------

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* add missing docstring for the list_categories tool and add pytest skip markers on unix-specific path handling tests (#7348)

* [Feature] Add Support For Python 3.14 (#7349)

* add support for Python 3.14

* escape % in argparse_translator in help strings

* black

* cli python version string

* fix test param placeholder

* fix integration_tests_testers

* add 3.14 to ODP Desktop environment creation choices

* partial lock update

* update lock files

* and the rest of the locks

---------

Co-authored-by: deeleeramone <>

* fix(cli): move custom Field kwargs to json_schema_extra for mypy compatibility (#7362)

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* [Feature] Refactor `regulators.sec.schema_files` As A General Purpose XBRL Schema Explorer (#7356)

* refactor regulators.sec.schema_files as a general purpose utility tool

* 2011 is different than 2012

* mypy

* mypy

* line too long

* recapture cassettes

* missing integration test param

* try tests without cassettes

---------

Co-authored-by: deeleeramone <>

* update package.json and pakcage-lock.json and cargo.lock (#7367)

* Upgrade GitHub Actions for Node 24 compatibility (#7371)

Signed-off-by: Salman Muin Kayser Chishti <13schishti@gmail.com>

* Upgrade GitHub Actions to latest versions (#7372)

Signed-off-by: Salman Muin Kayser Chishti <13schishti@gmail.com>
Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* refactor: sanitize dynamic column name in SEC form4 utility (#7376)

* feat(platform): add membership groups to Country type (#7352)

* feat(platform): add membership groups to Country type

Add membership groups (G7, G20, EU, NATO, OECD, OPEC, BRICS) to the Country
type, as suggested by maintainer in PR #7333.

Changes:
- Updated country_data.json with 'groups' field for each country
- Added Country.groups property returning list of group memberships
- Added Country.is_member_of(group) method for membership checks
- Added comprehensive tests for group functionality

This enables filtering and analysis by economic/political groupings:
  c = Country('Germany')
  c.groups        # ['G7', 'G20', 'EU', 'NATO', 'OECD']
  c.is_member_of('EU')  # True

Follow-up to #7333

* fix(platform): add Indonesia to BRICS (joined Jan 2025)

* refactor(platform): use TypedDict for country data typing

- Replace dict[str, Any] with CountryData TypedDict for proper type inference
- Add official data sources to module docstring
- Add _sources metadata section to country_data.json with references

Sources:
- G7: g7italy.it
- G20: g20.org
- EU: european-union.europa.eu
- NATO: nato.int
- OECD: oecd.org
- OPEC: opec.org
- BRICS: brics-russia2024.ru

* fix(platform): update group membership reference URLs

Replace broken/inaccessible URLs with stable Wikipedia references:
- G7, G20, OECD, OPEC, BRICS → Wikipedia (stable, well-maintained)
- EU, NATO → Updated to current official URLs

* feat: add developer script to regenerate country_data.json from sources

* fix: resolve codespell lint errors in update_country_data.py

* fix: resolve black/ruff lint issues in update_country_data.py

* feat(core): normalize unicode country names for ASCII input matching

* fix(core): remove Angola from OPEC and Bangladesh from NATO

Angola withdrew from OPEC effective Jan 2024. The OPEC scraper falls
back to a hardcoded list which still included AO — removed it.

Bangladesh was incorrectly scraped as a NATO member due to the scraper
iterating all wikitables on the page. Now filters tables by their
nearest h2 section header to only process member state tables.

* fix(core): normalize country names to ASCII in update script

Strip diacritical marks at generation time so country_data.json
uses ASCII-only names (e.g., Turkiye, Curacao, Cote d'Ivoire).

* Update openbb_platform/core/openbb_core/provider/utils/update_country_data.py

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* fix imports

* fix: resolve mypy type narrowing for expected counts

* restore reference.json

---------

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* feat(platform): standardize exchange inputs with ISO 10383 MIC codes (#7351)

* feat(platform): standardize exchange inputs with ISO 10383 MIC codes

Add Exchange type in provider utils that standardizes exchange inputs across
providers using ISO 10383 Market Identifier Codes (MIC).

Similar to the Country type from PR #7333, this allows providers to accept:
- ISO 10383 MIC codes (e.g., 'XNAS', 'XNYS')
- Exchange acronyms (e.g., 'NASDAQ', 'NYSE')
- Full exchange names (e.g., 'New York Stock Exchange')
- lower_snake_case names (e.g., 'new_york_stock_exchange')

The Exchange class inherits from str (storing the MIC code) while providing
access to mic, acronym, name, country, and city properties.

Example usage in FMP equity screener demonstrates provider integration pattern:
- exchange: Exchange | None with validator for provider-specific validation
- Maps Exchange.acronym to FMP's expected lowercase exchange codes

Addresses part of #6969

* refactor(platform): use ExchangeData TypedDict for proper typing

- Add ExchangeData TypedDict with explicit field types (consistent with CountryData)
- Update function signatures and class attributes to use ExchangeData
- Add _last_updated metadata to exchange_data.json
- Add Wikipedia reference URL to docstring

Aligns Exchange implementation with Country pattern from PR #7352.

* fix: add missing major exchanges including Cboe (XCBO, XCBT)

* refactor: expand to full ISO 10383 MIC list (2,272 active MICs)

Address review feedback to include all active MICs, not just curated majors.

- Expand exchange_data.json from 66 to 2,272 active MICs (operating + segments)
- Remove country and city fields (out of scope for this PR)
- Add scripts/update_exchange_data.py to regenerate from official ISO 20022
  CSV source using stdlib only (no external dependencies)
- Use first-write-wins lookup with operating MICs ordered before segments
  to resolve acronym collisions (e.g. LSE -> XLON, not XPVT)
- Update tests to reflect simplified data model

* feat: add city, country, and website fields to exchange data

* refactor: move update_exchange_data.py to provider utils

* style: fix black formatting in update_exchange_data.py

---------

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* [Feature] Add FRED Release Calendar to `economy.calendar` Endpoint (#7366)

* add fred release calendar to economy.calendar endpoint

* fix deribit test fixture

* Fix various typos (#7374)

* Fix typo: commnd -> command

* Fix typo: ommissions -> omissions

* Fix typo: survery -> survey

* Fix typo: obersvation -> observation

* Fix typo: obersvation -> observation

* Fix typos: Normaliztion -> Normalization, add missing space

* Fix typos: aicraft -> aircraft, exclud -> exclude

* Fix typo: informations -> information

* Fix typos: Miscellanous, excpet, accesories, similiar

* Fix typo: ommissions -> omissions

* Fix typo: aggreated -> aggregated

* Fix typo: foward_pe -> forward_pe

* Fix typo: contiunation -> continuation

* Fix typo: deferrred -> deferred

* Fix typo: daa -> date

* Fix typo: Activites -> Activities

* Fix typo: Activites -> Activities

* Fix typo: Recieved -> Received

* Fix typo: acquistion -> acquisition

* Fix typos: developement -> development

* Fix typo: obersvation -> observation

* Fix typo: Governement -> Government

* Fix typo: anaylsts -> analysts

* Fix typo: signapore -> singapore

* Fix typo: signapore -> singapore

* Fix typo: decalaration -> declaration

* Fix typo: Phramaceutical -> Pharmaceutical

* Revert changes to ec_codes.json

* Revert changes to ppi_codes.json

* Fix exchange_date alias to use daa

* fix linter error

Signed-off-by: luojiyin <luojiyin@hotmail.com>

---------

Signed-off-by: luojiyin <luojiyin@hotmail.com>
Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* fix incorrect checkbox state logic in extension selector (#7387)

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* update lock files... (#7391)

* [BugFix] `openbb-core`: Fix Missing Examples In `reference.json` (#7383)

* fix missing examples in reference.json

* fix comment

* fix descriptions not pulled from docstring-only endpoint parameter definitions

* ensure examples wrap parameter values in single quote

* [Feature] `openbb-sec`: Custom HTML2Markdown Conversion (#7361)

* refactor management_discussion_analysis to use custom html2markdown generator

* codespell

* mypy

* mypy

* mypy

* mypy

* some regex

* unnecessary elif

* refactor and unstack recursive nested inner functions

* fix some missing headers

* no-redef

* too-many-statements

* more header edge cases

* fix the N Weeks Ended pattern recognition

* strip breakcrumb links when finding md&a section

* experimental 20/40-F and 6-K support

* mypy..

* add pass for 6/8-K interim reports

* handle 8-k infographics too

* pylint

* pylint

---------

Co-authored-by: Theodore Aptekarev <aptekarev@gmail.com>

* fix(cli): replace eval with ast.literal_eval in parse_unknown_args_to_dict (#7390)

eval() on unsanitized CLI input allowed arbitrary code execution via
crafted --key payloads. ast.literal_eval is a safe drop-in that only
parses Python literals and raises ValueError/SyntaxError on expressions.
Also narrows the except clause and fixes the fallback key stripping bug.

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* chore: fix incorrect Tailwind class name (#7392)

Co-authored-by: Danglewood <85772166+deeleeramone@users.noreply.github.com>

* start release cycle

* extension files

* those files were not actually included

* more pyproject and lock files

* freezegun

* and the rest of them

* [BugFix] Update `get_user_agent` Utility With Modern User Agent Strings (#7398)

* update get_user_agent utility with modern user agent strings and add explicit headers to svensson_yield_curve

* no need to add step in svensson_yield_curve

* remove the comment that is breaking codespell for some reason

* another round of lock updates

* pylint

* remove searchCategory field no longer supported in widgets.json (#7400)

* platform-api

* [BugFix] openbb-core: Fix Merged Choices Validation in Python Interface (#7404)

* add choices validation to filter_inputs

* provider interface

* provider interface test

---------

Co-authored-by: deeleeramone <>

* bump core

---------

Signed-off-by: Salman Muin Kayser Chishti <13schishti@gmail.com>
Signed-off-by: luojiyin <luojiyin@hotmail.com>
Co-authored-by: montezdesousa <79287829+montezdesousa@users.noreply.github.com>
Co-authored-by: Igor Radovanovic <74266147+IgorWounds@users.noreply.github.com>
Co-authored-by: James Maslek <jmaslek11@gmail.com>
Co-authored-by: Luqman <luqazino@gmail.com>
Co-authored-by: Pratyush Shukla <ps4534@nyu.edu>
Co-authored-by: Henrique Joaquim <h.joaquim@campus.fct.unl.pt>
Co-authored-by: Michael Struwig <mnicstruwig@gmail.com>
Co-authored-by: jose-donato <43375532+jose-donato@users.noreply.github.com>
Co-authored-by: capital0 <72140796+capital0@users.noreply.github.com>
Co-authored-by: Ben Van Doren <benvandoren@users.noreply.github.com>
Co-authored-by: Andrew <andrew.kenreich@gmail.com>
Co-authored-by: Adi Sai <adithya.sairam1@gmail.com>
Co-authored-by: Sandip Saha <sandipsmoto@gmail.com>
Co-authored-by: Adwait <111136306+AdwaitSalankar@users.noreply.github.com>
Co-authored-by: Abhishek Amar <66067391+abhishek-amar@users.noreply.github.com>
Co-authored-by: DidierRLopes <dro.lopes@campus.fct.unl.pt>
Co-authored-by: professssor <62937125+professssor@users.noreply.github.com>
Co-authored-by: Farookh Zaheer Siddiqui <129654632+FarukhS52@users.noreply.github.com>
Co-authored-by: MAX SHESTOV <57421753+maxxsh@users.noreply.github.com>
Co-authored-by: jose-donato <zmcdonato@gmail.com>
Co-authored-by: Mohamad Fadil <fadil.parves@gmail.com>
Co-authored-by: Ramazan <guvencramazan@outlook.com>
Co-authored-by: joshuabuildsthings <joshuabuildsthings@users.noreply.github.com>
Co-authored-by: wojda <114031148+polooner@users.noreply.github.com>
Co-authored-by: Bim Phomthong <118195204+bimbolimbo@users.noreply.github.com>
Co-authored-by: Theodore Aptekarev <aptekarev@gmail.com>
Co-authored-by: tehcoderer <me@tehcoderer.com>
Co-authored-by: Ramazan <guvencramazan@yandex.com>
Co-authored-by: omahs <73983677+omahs@users.noreply.github.com>
Co-authored-by: minhhoang1023 <40023817+minhhoang1023@users.noreply.github.com>
Co-authored-by: Ayrton Parkinson <ayrton.parkinson1@gmail.com>
Co-authored-by: northern-64bit <75195383+northern-64bit@users.noreply.github.com>
Co-authored-by: Luqman <luqman.bello@openbb.finance>
Co-authored-by: Eric Li <38477812+EricLi-Dev@users.noreply.github.com>
Co-authored-by: Colin Delahunty <72827203+colin99d@users.noreply.github.com>
Co-authored-by: Henrique Joaquim <henriquecjoaquim@gmail.com>
Co-authored-by: Disorder AA <disorrder@gmail.com>
Co-authored-by: Diogo Sousa <montezdesousa@gmail.com>
Co-authored-by: Jingyi Z <zhao.elton@gmail.com>
Co-authored-by: Elias Rad <146735585+nnsW3@users.noreply.github.com>
Co-authored-by: mmistroni <mmistroni@gmail.com>
Co-authored-by: JC (Jonathan Chen) <jc@dijonkitchen.org>
Co-authored-by: Christian Clauss <cclauss@me.com>
Co-authored-by: Rin <dangnhatrin90@gmail.com>
Co-authored-by: Pádraic Slattery <pgoslatara@gmail.com>
Co-authored-by: Murat Aslan <78961478+murataslan1@users.noreply.github.com>
Co-authored-by: Murat Aslan <murataslan1@users.noreply.github.com>
Co-authored-by: John Seong <39040639+sandole@users.noreply.github.com>
Co-authored-by: Salman Chishti <salmanmkc@GitHub.com>
Co-authored-by: luo jiyin <luojiyin@hotmail.com>
Co-authored-by: Ocheretovich <ocheretovich@gmail.com>
Co-authored-by: T Arjun <heyarjunthekkedan@gmail.com>
2026-03-09 09:34:07 -07:00

987 lines
32 KiB
Python

"""Utils."""
import argparse
import ast
import os
import random
import re
import shutil
import sys
from contextlib import contextmanager
from datetime import (
datetime,
)
from pathlib import Path
from typing import TYPE_CHECKING, Optional
import numpy as np
import pandas as pd
import requests
from openbb_cli.config.constants import AVAILABLE_FLAIRS, ENV_FILE_SETTINGS
from openbb_cli.session import Session
from openbb_core.app.model.obbject import OBBject
from pytz import all_timezones, timezone
from rich.table import Table
if TYPE_CHECKING:
from openbb_charting.core.openbb_figure import OpenBBFigure
# pylint: disable=R1702,R0912
# pylint: disable=too-many-statements,no-member,too-many-branches,C0302
session = Session()
def remove_file(path: Path) -> bool:
"""Remove path.
Parameters
----------
path : Path
The file path.
Returns
-------
bool
The status of the removal.
"""
# TODO: Check why module level import leads to circular import.
try:
if os.path.isfile(path):
os.remove(path)
elif os.path.isdir(path):
shutil.rmtree(path)
return True
except Exception:
session.console.print(
f"\n[bold red]Failed to remove {path}\nPlease delete this manually![/bold red]"
)
return False
def print_goodbye():
"""Print a goodbye message when quitting the terminal."""
text = """
[param]Thank you for using the OpenBB Platform CLI and being part of this journey.[/param]
To stay tuned, sign up for our newsletter: [cmds]https://openbb.co/newsletter.[/]
Please feel free to check out our other products:
[bold]OpenBB Workspace[/]: [cmds]https://openbb.co[/cmds]
[bold]ODP Desktop Application:[/] [cmds]https://docs.openbb.co/odp/[/cmds]
[bold]ODP Python Package:[/] [cmds]https://docs.openbb.co/platform[/cmds]
"""
session.console.print(text)
def bootup():
"""Bootup the cli."""
if sys.platform == "win32":
# Enable VT100 Escape Sequence for WINDOWS 10 Ver. 1607
os.system("") # nosec # noqa: S605,S607
try:
if os.name == "nt":
# pylint: disable=E1101
sys.stdin.reconfigure(encoding="utf-8") # type: ignore
# pylint: disable=E1101
sys.stdout.reconfigure(encoding="utf-8") # type: ignore
except Exception as e:
session.console.print(e, "\n")
def welcome_message():
"""Print the welcome message.
Prints first welcome message, help and a notification if updates are available.
"""
session.console.print(
f"\nWelcome to OpenBB Platform CLI v{session.settings.VERSION}"
)
def reset(queue: list[str] | None = None):
"""Reset the CLI.
Allows for checking code without quitting.
"""
session.console.print("resetting...")
debug = session.settings.DEBUG_MODE
dev = session.settings.DEV_BACKEND
try:
# we clear all openbb_cli modules from sys.modules
for module in list(sys.modules.keys()):
parts = module.split(".")
if parts[0] == "openbb_cli":
del sys.modules[module]
queue_list = ["/".join(queue) if len(queue) > 0 else ""] # type: ignore
# pylint: disable=import-outside-toplevel
from openbb_cli.controllers.cli_controller import main
main(debug, dev, queue_list, module="") # type: ignore
except Exception as e:
session.console.print(f"Unfortunately, resetting wasn't possible: {e}\n")
print_goodbye()
@contextmanager
def suppress_stdout():
"""Suppress the stdout."""
with open(os.devnull, "w") as devnull:
old_stdout = sys.stdout
old_stderr = sys.stderr
sys.stdout = devnull
sys.stderr = devnull
try:
yield
finally:
sys.stdout = old_stdout
sys.stderr = old_stderr
def first_time_user() -> bool:
"""Check whether a user is a first time user.
A first time user is someone with an empty .env file.
If this is true, it also adds an env variable to make sure this does not run again.
Returns
-------
bool
Whether or not the user is a first time user
"""
if ENV_FILE_SETTINGS.stat().st_size == 0:
session.settings.set_item("PREVIOUS_USE", True)
return True
return False
def parse_and_split_input(an_input: str, custom_filters: list) -> list[str]:
"""Filter and split the input queue.
Uses regex to filters command arguments that have forward slashes so that it doesn't
break the execution of the command queue.
Currently handles unix paths and sorting settings for screener menus.
Parameters
----------
an_input : str
User input as string
custom_filters : List
Additional regular expressions to match
Returns
-------
List[str]
Command queue as list
"""
# Make sure that the user can go back to the root when doing "/"
if an_input and an_input == "/":
an_input = "home"
# everything from ` -f ` to the next known extension
file_flag = r"(\ -f |\ --file )"
up_to = r".*?"
known_extensions = r"(\.(xlsx|csv|xls|tsv|json|yaml|ini|openbb|ipynb))"
unix_path_arg_exp = f"({file_flag}{up_to}{known_extensions})"
# Add custom expressions to handle edge cases of individual controllers
custom_filter = ""
for exp in custom_filters:
if exp is not None:
custom_filter += f"|{exp}"
del exp
slash_filter_exp = f"({unix_path_arg_exp}){custom_filter}"
filter_input = True
placeholders: dict[str, str] = {}
while filter_input:
match = re.search(pattern=slash_filter_exp, string=an_input)
if match is not None:
placeholder = f"{{placeholder{len(placeholders) + 1}}}"
placeholders[placeholder] = an_input[
match.span()[0] : match.span()[1]
] # noqa:E203
an_input = (
an_input[: match.span()[0]] + placeholder + an_input[match.span()[1] :]
) # noqa:E203
else:
filter_input = False
commands = an_input.split("/") if "timezone" not in an_input else [an_input]
for command_num, command in enumerate(commands):
if command == commands[-1] == "":
return list(filter(None, commands))
matching_placeholders = [tag for tag in placeholders if tag in command]
if len(matching_placeholders) > 0:
for tag in matching_placeholders:
commands[command_num] = command.replace(tag, placeholders[tag])
return commands
def return_colored_value(value: str):
"""Return the string value based on condition.
Return it with green, yellow, red or white color based on
whether the number is positive, negative, zero or other, respectively.
Parameters
----------
value: str
string to be checked
Returns
-------
value: str
string with color based on value of number if it exists
"""
values = re.findall(r"[-+]?(?:\d*\.\d+|\d+)", value)
# Finds exactly 1 number in the string
if len(values) == 1:
if float(values[0]) > 0:
return f"[green]{value}[/green]"
if float(values[0]) < 0:
return f"[red]{value}[/red]"
if float(values[0]) == 0:
return f"[yellow]{value}[/yellow]"
return f"{value}"
# pylint: disable=too-many-arguments,too-many-positional-arguments
def print_rich_table( # noqa: PLR0912
df: pd.DataFrame,
show_index: bool = False,
title: str = "",
index_name: str = "",
headers: list[str] | pd.Index | None = None,
floatfmt: str | list[str] = ".2f",
show_header: bool = True,
automatic_coloring: bool = False,
columns_to_auto_color: list[str] | None = None,
rows_to_auto_color: list[str] | None = None,
export: bool = False,
limit: int | None = 1000,
columns_keep_types: list[str] | None = None,
use_tabulate_df: bool = True,
):
"""Prepare a table from df in rich.
Parameters
----------
df: pd.DataFrame
Dataframe to turn into table
show_index: bool
Whether to include index
title: str
Title for table
index_name : str
Title for index column
headers: List[str]
Titles for columns
floatfmt: Union[str, List[str]]
Float number formatting specs as string or list of strings. Defaults to ".2f"
show_header: bool
Whether to show the header row.
automatic_coloring: bool
Automatically color a table based on positive and negative values
columns_to_auto_color: List[str]
Columns to automatically color
rows_to_auto_color: List[str]
Rows to automatically color
export: bool
Whether we are exporting the table to a file. If so, we don't want to print it.
limit: Optional[int]
Limit the number of rows to show.
columns_keep_types: Optional[List[str]]
Columns to keep their types, i.e. not convert to numeric
"""
if export:
return
MAX_COLS = session.settings.ALLOWED_NUMBER_OF_COLUMNS
MAX_ROWS = session.settings.ALLOWED_NUMBER_OF_ROWS
# Make a copy of the dataframe to avoid SettingWithCopyWarning
df = df.copy()
show_index = not isinstance(df.index, pd.RangeIndex) and show_index
# convert non-str that are not timestamp or int into str
# eg) praw.models.reddit.subreddit.Subreddit
for col in df.columns:
if columns_keep_types is not None and col in columns_keep_types:
continue
try:
if not any(
isinstance(df[col].iloc[x], pd.Timestamp)
for x in range(min(10, len(df)))
):
df[col] = df[col].apply(pd.to_numeric)
except (ValueError, TypeError):
df[col] = df[col].astype(str)
def _get_headers(_headers: list[str] | pd.Index) -> list[str]:
"""Check if headers are valid and return them."""
output = _headers
if isinstance(_headers, pd.Index):
output = list(_headers)
if len(output) != len(df.columns):
raise ValueError("Length of headers does not match length of DataFrame.")
return output # type: ignore
if session.settings.USE_INTERACTIVE_DF:
df_outgoing = df.copy()
# If headers are provided, use them
if headers is not None:
# We check if headers are valid
df_outgoing.columns = _get_headers(headers)
if show_index and index_name not in df_outgoing.columns:
# If index name is provided, we use it
df_outgoing.index.name = index_name or "Index"
df_outgoing = df_outgoing.reset_index()
for col in df_outgoing.columns:
if col == "":
df_outgoing = df_outgoing.rename(columns={col: " "})
session._backend.send_table( # type: ignore # pylint: disable=protected-access
df_table=df_outgoing,
title=title,
theme=session.user.preferences.table_style,
)
return
df = df.copy() if not limit else df.copy().iloc[:limit]
if automatic_coloring:
if columns_to_auto_color:
for col in columns_to_auto_color:
# checks whether column exists
if col in df.columns:
df[col] = df[col].apply(lambda x: return_colored_value(str(x)))
if rows_to_auto_color:
for row in rows_to_auto_color:
# checks whether row exists
if row in df.index:
df.loc[row] = df.loc[row].apply(
lambda x: return_colored_value(str(x))
)
if columns_to_auto_color is None and rows_to_auto_color is None:
df = df.map(lambda x: return_colored_value(str(x))) # type: ignore
exceeds_allowed_columns = len(df.columns) > MAX_COLS
exceeds_allowed_rows = len(df) > MAX_ROWS
if exceeds_allowed_columns:
original_columns = df.columns.tolist()
trimmed_columns = df.columns.tolist()[:MAX_COLS]
df = df[trimmed_columns]
trimmed_columns = [
col for col in original_columns if col not in trimmed_columns
]
if exceeds_allowed_rows:
n_rows = len(df.index)
max_rows = MAX_ROWS
df = df[:max_rows]
trimmed_rows_count = n_rows - max_rows
if use_tabulate_df:
table = Table(title=title, show_lines=True, show_header=show_header)
if show_index:
table.add_column(index_name)
if headers is not None:
headers = _get_headers(headers)
for header in headers:
table.add_column(str(header))
else:
for column in df.columns:
table.add_column(str(column))
if isinstance(floatfmt, list) and len(floatfmt) != len(df.columns):
raise (
ValueError(
"Length of floatfmt list does not match length of DataFrame columns."
)
)
if isinstance(floatfmt, str):
floatfmt = [floatfmt for _ in range(len(df.columns))]
for idx, values in zip(df.index.tolist(), df.values.tolist()):
# remove hour/min/sec from timestamp index - Format: YYYY-MM-DD # make better
row_idx = [str(idx)] if show_index else []
row_idx += [
(
str(x)
if not isinstance(x, float) and not isinstance(x, np.float64)
else (
f"{x:{floatfmt[idx]}}"
if isinstance(floatfmt, list)
else (
f"{x:.2e}"
if 0 < abs(float(x)) <= 0.0001
else f"{x:floatfmt}"
)
)
)
for idx, x in enumerate(values)
]
table.add_row(*row_idx)
session.console.print(table)
else:
session.console.print(df.to_string(col_space=0))
if exceeds_allowed_columns:
session.console.print(
f"[yellow]\nAllowed number of columns exceeded ({session.settings.ALLOWED_NUMBER_OF_COLUMNS}).\n"
f"The following columns were removed from the output: {', '.join(trimmed_columns)}.\n[/yellow]"
)
if exceeds_allowed_rows:
session.console.print(
f"[yellow]\nAllowed number of rows exceeded ({session.settings.ALLOWED_NUMBER_OF_ROWS}).\n"
f"{trimmed_rows_count} rows were removed from the output.\n[/yellow]"
)
if exceeds_allowed_columns or exceeds_allowed_rows:
session.console.print(
"Use the `--export` flag to analyse the full output on a file."
)
def check_non_negative(value) -> int:
"""Argparse type to check non negative int."""
new_value = int(value)
if new_value < 0:
raise argparse.ArgumentTypeError(f"{value} is negative")
return new_value
def check_positive(value) -> int:
"""Argparse type to check positive int."""
new_value = int(value)
if new_value <= 0:
raise argparse.ArgumentTypeError(f"{value} is an invalid positive int value")
return new_value
def validate_register_key(value: str) -> str:
"""Validate the register key to ensure it does not contain the reserved word 'OBB'."""
if "OBB" in value:
raise argparse.ArgumentTypeError(
"The register key cannot contain the reserved word 'OBB'."
)
return str(value)
def get_user_agent() -> str:
"""Get a not very random user agent."""
user_agent_strings = [
"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.10; rv:86.1) Gecko/20100101 Firefox/86.1",
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:86.1) Gecko/20100101 Firefox/86.1",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:82.1) Gecko/20100101 Firefox/82.1",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:86.0) Gecko/20100101 Firefox/86.0",
"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:86.0) Gecko/20100101 Firefox/86.0",
"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.10; rv:83.0) Gecko/20100101 Firefox/83.0",
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:84.0) Gecko/20100101 Firefox/84.0",
]
return random.choice(user_agent_strings) # nosec # noqa: S311
def get_flair() -> str:
"""Get a flair icon."""
current_flair = str(session.settings.FLAIR)
flair = AVAILABLE_FLAIRS.get(current_flair, current_flair)
return flair
def get_dtime() -> str:
"""Get a datetime string."""
dtime = ""
if session.settings.USE_DATETIME and get_user_timezone_or_invalid() != "INVALID":
dtime = datetime.now(timezone(get_user_timezone())).strftime("%Y %b %d, %H:%M")
return dtime
def get_flair_and_username() -> str:
"""Get a flair icon and username."""
flair = get_flair()
if dtime := get_dtime():
dtime = f"{dtime} "
return f"{dtime}{flair}"
def is_timezone_valid(user_tz: str) -> bool:
"""Check whether user timezone is valid.
Parameters
----------
user_tz: str
Timezone to check for validity
Returns
-------
bool
True if timezone provided is valid
"""
return user_tz in all_timezones
def get_user_timezone() -> str:
"""Get user timezone if it is a valid one.
Returns
-------
str
user timezone based on .env file
"""
return session.settings.TIMEZONE
def get_user_timezone_or_invalid() -> str:
"""Get user timezone if it is a valid one.
Returns
-------
str
user timezone based on timezone.openbb file or INVALID
"""
user_tz = get_user_timezone()
if is_timezone_valid(user_tz):
return f"{user_tz}"
return "INVALID"
def check_file_type_saved(valid_types: list[str] | None = None):
"""Provide valid types for the user to be able to select.
Parameters
----------
valid_types: List[str]
List of valid types to export data
Returns
-------
check_filenames: Optional[List[str]]
Function that returns list of filenames to export data
"""
def check_filenames(filenames: str = "") -> str:
"""Check if filenames are valid.
Parameters
----------
filenames: str
filenames to be saved separated with comma
Returns
----------
str
valid filenames separated with comma
"""
if not filenames or not valid_types:
return ""
valid_filenames = list()
for filename in filenames.split(","):
if filename.endswith(tuple(valid_types)):
valid_filenames.append(filename)
else:
session.console.print(
f"[red]Filename '{filename}' provided is not valid!\nPlease use one of the following file types:"
f"{','.join(valid_types)}[/red]\n"
)
return ",".join(valid_filenames)
return check_filenames
def remove_timezone_from_dataframe(df: pd.DataFrame) -> pd.DataFrame:
"""Remove timezone information from a dataframe.
Parameters
----------
df : pd.DataFrame
The dataframe to remove timezone information from
Returns
-------
pd.DataFrame
The dataframe with timezone information removed
"""
date_cols = []
index_is_date = False
# Find columns and index containing date data
if (
df.index.dtype.kind == "M"
and hasattr(df.index.dtype, "tz")
and df.index.dtype.tz is not None # type: ignore
):
index_is_date = True
for col, dtype in df.dtypes.items():
if dtype.kind == "M" and hasattr(df.index.dtype, "tz") and dtype.tz is not None:
date_cols.append(col)
# Remove the timezone information
for col in date_cols:
df[col] = df[col].dt.date
if index_is_date:
index_name = df.index.name
df.index = df.index.date # type: ignore
df.index.name = index_name
return df
def compose_export_path(func_name: str, dir_path: str) -> Path:
"""Compose export path for data from the terminal.
Creates a path to a folder and a filename based on conditions.
Parameters
----------
func_name : str
Name of the command that invokes this function
dir_path : str
Path of directory from where this function is called
Returns
-------
Path
Path variable containing the path of the exported file
"""
now = datetime.now()
# Resolving all symlinks and also normalizing path.
resolve_path = Path(dir_path).resolve()
# Getting the directory names from the path. Instead of using split/replace (Windows doesn't like that)
# check if this is done in a main context to avoid saving with openbb_cli
if resolve_path.parts[-2] == "openbb_cli":
path_cmd = f"{resolve_path.parts[-1]}"
else:
path_cmd = f"{resolve_path.parts[-2]}_{resolve_path.parts[-1]}"
default_filename = f"{now.strftime('%Y%m%d_%H%M%S')}_{path_cmd}_{func_name}"
full_path = Path(session.user.preferences.export_directory) / default_filename
return full_path
def ask_file_overwrite(file_path: Path) -> tuple[bool, bool]:
"""Provide a prompt for overwriting existing files.
Returns two values, the first is a boolean indicating if the file exists and the
second is a boolean indicating if the user wants to overwrite the file.
"""
if session.settings.FILE_OVERWRITE:
return False, True
if session.settings.TEST_MODE:
return False, True
if file_path.exists():
overwrite = input("\nFile already exists. Overwrite? [y/n]: ").lower()
if overwrite == "y":
file_path.unlink(missing_ok=True)
# File exists and user wants to overwrite
return True, True
# File exists and user does not want to overwrite
return True, False
# File does not exist
return False, True
# This is a false positive on pylint and being tracked in pylint #3060
# pylint: disable=abstract-class-instantiated,too-many-positional-arguments
def save_to_excel(df, saved_path, sheet_name, start_row=0, index=True, header=True):
"""Save a Pandas DataFrame to an Excel file.
Args:
df: A Pandas DataFrame.
saved_path: The path to the Excel file to save to.
sheet_name: The name of the sheet to save the DataFrame to.
start_row: The row number to start writing the DataFrame at.
index: Whether to write the DataFrame index to the Excel file.
header: Whether to write the DataFrame header to the Excel file.
"""
overwrite_options = {
"o": "replace",
"a": "overlay",
"n": "new",
}
if not saved_path.exists():
with pd.ExcelWriter(saved_path, engine="openpyxl") as writer:
df.to_excel(writer, sheet_name=sheet_name, index=index, header=header)
else:
with pd.ExcelFile(saved_path) as reader:
overwrite_option = "n"
if sheet_name in reader.sheet_names:
overwrite_option = input(
"\nSheet already exists. Overwrite/Append/New? [o/a/n]: "
).lower()
start_row = 0
if overwrite_option == "a":
existing_df = pd.read_excel(saved_path, sheet_name=sheet_name)
start_row = existing_df.shape[0] + 1
with pd.ExcelWriter(
saved_path,
mode="a",
if_sheet_exists=overwrite_options[overwrite_option], # type: ignore
engine="openpyxl",
) as writer:
df.to_excel(
writer,
sheet_name=sheet_name,
startrow=start_row,
index=index,
header=False if overwrite_option == "a" else header,
)
# This is a false positive on pylint and being tracked in pylint #3060
# pylint: disable=abstract-class-instantiated,too-many-positional-arguments
def export_data(
export_type: str,
dir_path: str,
func_name: str,
df: pd.DataFrame = pd.DataFrame(),
sheet_name: str | None = None,
figure: Optional["OpenBBFigure"] = None,
margin: bool = True,
) -> None:
"""Export data to a file.
Parameters
----------
export_type : str
Type of export between: csv,json,xlsx,xls
dir_path : str
Path of directory from where this function is called
func_name : str
Name of the command that invokes this function
df : pd.Dataframe
Dataframe of data to save
sheet_name : str
If provided. The name of the sheet to save in excel file
figure : Optional[OpenBBFigure]
Figure object to save as image file
margin : bool
Automatically adjust subplot parameters to give specified padding.
"""
if export_type:
saved_path = compose_export_path(func_name, dir_path).resolve()
saved_path.parent.mkdir(parents=True, exist_ok=True)
for exp_type in export_type.split(","):
# In this scenario the path was provided, e.g. --export pt.csv, pt.jpg
if "." in exp_type:
saved_path = saved_path.with_name(exp_type)
# In this scenario we use the default filename
else:
if ".OpenBB_openbb_cli" in saved_path.name:
saved_path = saved_path.with_name(
saved_path.name.replace(".OpenBB_openbb_cli", "OpenBBCLI")
)
saved_path = saved_path.with_suffix(f".{exp_type}")
exists, overwrite = False, False
is_xlsx = exp_type.endswith("xlsx")
if sheet_name is None and is_xlsx or not is_xlsx:
exists, overwrite = ask_file_overwrite(saved_path)
if exists and not overwrite:
existing = len(list(saved_path.parent.glob(saved_path.stem + "*")))
saved_path = saved_path.with_stem(f"{saved_path.stem}_{existing + 1}")
df = df.replace(
{
r"\[yellow\]": "",
r"\[/yellow\]": "",
r"\[green\]": "",
r"\[/green\]": "",
r"\[red\]": "",
r"\[/red\]": "",
r"\[magenta\]": "",
r"\[/magenta\]": "",
},
regex=True,
)
if exp_type.endswith("csv"):
df.to_csv(saved_path)
elif exp_type.endswith("json"):
df.reset_index(drop=True, inplace=True)
df.to_json(saved_path)
elif exp_type.endswith("xlsx"):
# since xlsx does not support datetimes with timezones we need to remove it
df = remove_timezone_from_dataframe(df)
if sheet_name is None: # noqa: SIM223
df.to_excel(
saved_path,
index=True,
header=True,
)
else:
save_to_excel(df, saved_path, sheet_name)
elif saved_path.suffix in [".jpg", ".png"]:
if figure is None:
session.console.print("No plot to export.")
continue
figure.show(export_image=saved_path, margin=margin)
else:
session.console.print("Wrong export file specified.")
continue
if saved_path.exists():
session.console.print(f"Saved file: {saved_path}")
else:
session.console.print(f"Failed to save file: {saved_path}")
if figure is not None:
figure._exported = True # pylint: disable=protected-access
def system_clear():
"""Clear screen."""
os.system("cls||clear") # nosec # noqa: S605,S607
# Write an abstract helper to make requests from a url with potential headers and params
def request(
url: str, method: str = "get", timeout: int = 0, **kwargs
) -> requests.Response:
"""Make requests from a url with potential headers and params.
Parameters
----------
url : str
Url to make the request to
method : str
HTTP method to use. Choose from:
delete, get, head, patch, post, put, by default "get"
timeout : int
How many seconds to wait for the server to send data
Returns
-------
requests.Response
Request response object
Raises
------
ValueError
If invalid method is passed
"""
method = method.lower()
if method not in ["delete", "get", "head", "patch", "post", "put"]:
raise ValueError(f"Invalid method: {method}")
# We want to add a user agent to the request, so check if there are any headers
# If there are headers, check if there is a user agent, if not add one.
# Some requests seem to work only with a specific user agent, so we want to be able to override it.
headers = kwargs.pop("headers", {})
timeout = timeout or session.user.preferences.request_timeout
if "User-Agent" not in headers:
headers["User-Agent"] = get_user_agent()
func = getattr(requests, method)
return func(
url,
headers=headers,
timeout=timeout,
**kwargs,
)
def parse_unknown_args_to_dict(unknown_args: list[str] | None) -> dict[str, str]:
"""Parse unknown arguments to a dictionary."""
unknown_args_dict = {}
if unknown_args:
for idx, arg in enumerate(unknown_args):
if arg.startswith("--"):
if idx + 1 < len(unknown_args):
try:
unknown_args_dict[arg.replace("--", "")] = ast.literal_eval(
unknown_args[idx + 1]
)
except (ValueError, SyntaxError):
unknown_args_dict[arg.replace("--", "")] = unknown_args[idx + 1]
else:
session.console.print(
f"Missing value for argument {arg}. Skipping this argument."
)
return unknown_args_dict
def handle_obbject_display(
obbject: OBBject,
chart: bool = False,
export: str = "",
sheet_name: str = "",
**kwargs,
):
"""Handle the display of an OBBject."""
df: pd.DataFrame = pd.DataFrame()
fig: OpenBBFigure | None = None
if chart:
try:
if obbject.chart:
obbject.show(**kwargs)
else:
obbject.charting.to_chart(**kwargs) # type: ignore
if export:
fig = obbject.chart.fig # type: ignore
df = obbject.to_dataframe()
except Exception as e:
session.console.print(f"Failed to display chart: {e}")
elif session.settings.USE_INTERACTIVE_DF:
obbject.charting.table() # type: ignore
else:
df = obbject.to_dataframe()
print_rich_table(
df=df,
show_index=True,
title=obbject.extra.get("command", ""),
export=bool(export),
)
if export and not df.empty:
if sheet_name and isinstance(sheet_name, list):
sheet_name = sheet_name[0]
func_name = (
obbject.extra.get("command", "")
.replace("/", "_")
.replace(" ", "_")
.replace("--", "_")
)
export_data(
export_type=",".join(export),
dir_path=os.path.dirname(os.path.abspath(__file__)),
func_name=func_name,
df=df,
sheet_name=sheet_name,
figure=fig,
)
elif export and df.empty:
session.console.print("[yellow]No data to export.[/yellow]")