Merge remote-tracking branch 'OpenBB-finance/develop' into feature/argparse_translator

.github/pull_request_template.md

@@ -1,31 +1,80 @@
-# Description
+# Pull Request OpenBB
 
-- [ ] Summary of the change / bug fix.
+## The OpenBBTerminal
+
+<details>
+<summary> Pull Request for the OpenBBTerminal</summary>
+
+### Description
+
+- [ ] Summary of the change/ bug fix.
 - [ ] Link # issue, if applicable.
 - [ ] Screenshot of the feature or the bug before/after fix, if applicable.
 - [ ] Relevant motivation and context.
 - [ ] List any dependencies that are required for this change.
 
+### How has this been tested?
 
-# How has this been tested?
+- Please describe the tests that you ran to verify your changes.
+- Please provide instructions so we can reproduce.
+- Please also list any relevant details for your test configuration.
 
-* Please describe the tests that you ran to verify your changes.
-* Provide instructions so we can reproduce.
-* Please also list any relevant details for your test configuration.
-- [ ] Make sure affected commands still run in terminal
-- [ ] Ensure the SDK still works
-- [ ] Check any related reports
+- [ ] Ensure the affected commands still execute in the OpenBB Terminal.
+- [ ] Ensure the Platform (previously named SDK) is working as intended.
+- [ ] Check any related reports.
 
+### Checklist
 
-# Checklist:
-
+- [ ] I ensure I have self-reviewed my code.
+- [ ] I have commented/documented my code, particularly in hard-to-understand sections.
 - [ ] I have adhered to the GitFlow naming convention and my branch name is in the format of `feature/feature-name` or `hotfix/hotfix-name`.
 - [ ] Update [our documentation](https://openbb-finance.github.io/OpenBBTerminal/) following [these guidelines](https://github.com/OpenBB-finance/OpenBBTerminal/tree/main/website).  Update any user guides that are affected by the changes.
 - [ ] Update our tests following [these guidelines](https://github.com/OpenBB-finance/OpenBBTerminal/tree/main/tests).
 - [ ] Make sure you are following our [CONTRIBUTING guidelines](https://github.com/OpenBB-finance/OpenBBTerminal/blob/main/CONTRIBUTING.md).
 - [ ] If a feature was added make sure to add it to the corresponding [integration test script](https://github.com/OpenBB-finance/OpenBBTerminal/tree/develop/openbb_terminal/miscellaneous/integration_tests_scripts).
 
+</details>
+
+## The OpenBB Platform
+
+<details>
+<summary> Pull Request for the OpenBB Platform</summary>
+
+### Description
+
+- [ ] Summary of the change/ bug fix.
+- [ ] Link # issue, if applicable.
+- [ ] Screenshot of the feature or the bug before/after fix, if applicable.
+- [ ] Relevant motivation and context.
+- [ ] List any dependencies that are required for this change.
+
+### How has this been tested?
+
+- Please describe the tests that you ran to verify your changes.
+- Please provide instructions so we can reproduce.
+- Please also list any relevant details for your test configuration.
+
+- [ ] Ensure all unit and integration tests pass.
+- If you modified/added command(s):
+  - [ ] Ensure the command(s) execute with the expected output.
+    - [ ] API.
+    - [ ] Python Interface.
+  - [ ] If applicable, please add new tests for the command (see [CONTRIBUTING.md](/openbb_platform/CONTRIBUTING.md) to leverage semi-automated testing).
+- If a new provider was introduced or a new fetcher was added to an existing provider:
+  - [ ] Ensure the existing tests pass.
+  - [ ] Ensure the new provider and/or fetcher is stable and usable.
+  - [ ] If applicable, please add new tests for the provider and/or fetcher (see [CONTRIBUTING.md](/openbb_platform/CONTRIBUTING.md) to leverage semi-automated testing).
+- If a new provider or extension was added:
+  - [ ] Update the list of [Extensions](/openbb_platform/EXTENSIONS.md).
+  - [ ] Update the list of [Providers](/openbb_platform/PROVIDERS.md).
+  - [ ] If it's a community extension or provider, update the [integration tests GitHub Action workflow](/.github/workflows/platform-api-integration-test.yml).
+
+### Checklist
 
-# Others
 - [ ] I have performed a self-review of my own code.
 - [ ] I have commented my code, particularly in hard-to-understand areas.
+- [ ] I have adhered to the GitFlow naming convention and my branch name is in the format of `feature/feature-name` or `hotfix/hotfix-name`.
+- [ ] I ensure that I am following the [CONTRIBUTING guidelines](https://github.com/OpenBB-finance/OpenBBTerminal/blob/main/CONTRIBUTING.md).
+  - [ ] (If applicable) I have updated tests following [these guidelines](/openbb_platform/CONTRIBUTING.md#qa-your-extension).
+
+</details>

.github/workflows/branch-name-check.yml

@@ -26,12 +26,12 @@ jobs:
 
         echo "Source-branch=$source_branch" >> $GITHUB_OUTPUT
         echo "target-branch=$target_branch" >> $GITHUB_OUTPUT
-    
+
     - name: Show Output result for source-branch and target-branch
       run: |
         echo "source-branch=${{ steps.branch-name-check.outputs.source-branch }}"
         echo "target-branch=${{ steps.branch-name-check.outputs.target-branch }}"
-    
+
     - name: Check branch name for develop PRs
       if: ${{ steps.branch-name-check.outputs.target-branch == 'develop' }}
       run: |
@@ -50,4 +50,4 @@ jobs:
         else
           echo "PR is not from a hotfix or release branch. Pull requests must be from a hotfix or release branch and target the main branch"
           exit 1
-        fi
+        fi

.github/workflows/build-release.yml

@@ -10,8 +10,7 @@ env:
 on:
   push:
     branches:
-      - release/*
-      - main
+      - release/v3/*
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}

.github/workflows/docker-build.yml

@@ -5,7 +5,7 @@ on: workflow_dispatch
 jobs:
   publish-image:
     runs-on: ubuntu-latest
-    env: 
+    env:
       OPENBBTERMINAL_DOCKER_GITHUB_REPOSITORY: ${{ github.repository }}
       OPENBB_LOGGING_APP_NAME: "gst_docker"
 
@@ -45,4 +45,4 @@ jobs:
       - name: Publishing the latest Docker image
         run: |
           docker push "${OPENBBTERMINAL_DOCKER_POETRY_IMAGE_LATEST}"
-        if: startsWith(github.ref, 'refs/heads/main')
+        if: startsWith(github.ref, 'refs/heads/main')

.github/workflows/gh-pages.yml

@@ -6,9 +6,10 @@ on:
       - main
       - release/*
   pull_request:
-    types: [closed]
-    branches:
-      - develop
+      types: [opened, synchronize, reopened, closed]
+      branches:
+        - develop
+        - 'docs/*'
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
@@ -64,7 +65,13 @@ jobs:
           source $VENV
           pip uninstall Brotli -y
           pytest tests/website --autodoc
-          python website/generate_sdk_markdown.py && python website/generate_terminal_markdown.py
+          python website/generate_sdk_v3_markdown.py
+          python website/generate_terminal_v3_markdown.py
+          pip install -U poetry
+          python openbb_platform/dev_install.py -e all
+          pip uninstall nbmake -y
+          python website/generate_platform_v4_markdown.py
+          python website/generate_widgets_library.py
 
       - name: Setup Node
         uses: actions/setup-node@v3
@@ -90,7 +97,7 @@ jobs:
           (github.event_name == 'pull_request' &&
           github.event.pull_request.merged == true &&
           github.event.pull_request.base.ref == 'develop' &&
-          startsWith(github.head_ref, 'docs/')) 
+          startsWith(github.head_ref, 'docs/'))
         with:
           github_token: ${{ secrets.DEPLOY_TOKEN }}
           publish_dir: ./website/build

.github/workflows/labels-PR.yml

@@ -10,7 +10,7 @@ jobs:
         with:
           mode: minimum
           count: 1
-          labels: "guides, bug, build, do not merge, docker, docs, duplicate, enhancement, feat XS, feat S, feat M, feat L, feat XL, help wanted, refactor, tests, discord, dependencies, notebookreports, dashboards, analytics, portfolio, installer, SDK, release "
+          labels: "guides, bug, build, docker, docs, feat XS, feat S, feat M, feat L, feat XL, help wanted, refactor, tests, dependencies, release"
       - uses: mheap/github-action-required-labels@v1
         with:
           mode: exactly

.github/workflows/linting.yml

@@ -30,6 +30,12 @@ jobs:
     steps:
       - name: Checkout Code
         uses: actions/checkout@2541b1294d2704b0964813337f33b291d3f8596b # actions/checkout v3.0.2
+        with:
+          ref: ${{ github.head_ref }}
+          fetch-depth: 20
+
+      - name: Fetch base branch
+        run: git fetch --no-tags --depth=20 origin ${{ github.base_ref }}
 
       - name: Setup Python 3.9
         uses: actions/setup-python@v4
@@ -37,6 +43,15 @@ jobs:
           python-version: "3.9"
           architecture: x64
 
+      - name: Get changed files in openbb_terminal and openbb_platform for PR
+        if: github.event_name == 'pull_request'
+        run: |
+          # "Checking PR diff"
+          echo "terminal_files=$(git diff --name-only origin/${{ github.base_ref }}...${{ github.head_ref }} | grep 'openbb_terminal/.*\.py$' | xargs)" >> $GITHUB_ENV
+          echo $terminal_files
+          echo "platform_files=$(git diff --name-only origin/${{ github.base_ref }}...${{ github.head_ref }} | grep 'openbb_platform/.*\.py$' | grep -v 'openbb_platform/openbb/package' | grep -v 'integration' | grep -v 'tests' | xargs)" >> $GITHUB_ENV
+          echo $platform_files
+
       - uses: actions/cache@v3
         with:
           path: ~/.cache/pip
@@ -44,14 +59,28 @@ jobs:
           restore-keys: ${{ runner.os }}-linting-${{ hashFiles('**/poetry.lock') }}
 
       - run: |
-          pip install bandit black codespell mypy==1.1.1 pylint==2.17.0 ruff==0.0.256
+          pip install bandit black codespell==2.2.5 mypy==1.5.1 pylint==2.17.0 ruff==0.0.285
           pip install types-pytz types-requests types-termcolor types-tabulate types-PyYAML types-python-dateutil types-setuptools types-six
       - run: bandit -x ./tests -r . || true
       - run: black --diff --check .
-      - run: codespell --ignore-words-list=vai,mis,gard,commun,parana,statics,ro,zar,zlot,jewl,ba,buil,coo,ether,hist,hsi,mape,navagation,operatio,pres,ser,yeld,shold,ist,varian,datas,ake,creat,vie,hel,ket,toke,certi,buidl,ot,te,buda,shs,welp --quiet-level=2 --skip=./tests,.git,*.css,*.csv,*.html,*.ini,*.ipynb,*.js,*.json,*.lock,*.scss,*.txt,*.yaml,./build/pyinstaller,./website/config.toml -x openbb_terminal/economy/fedreserve_model.py
+      - run: codespell --ignore-words=.codespell.ignore --skip="$(tr '\n' ',' < .codespell.skip | sed 's/,$//')" --quiet-level=2
       - run: ruff .
-      - run: mypy --ignore-missing-imports openbb_terminal
-      - run: pylint terminal.py openbb_terminal tests
+      - run: |
+          # Run linters for openbb_terminal
+          if [ -n "${{ env.terminal_files }}" ]; then
+            mypy --ignore-missing-imports ${{ env.terminal_files }}
+            pylint terminal.py ${{ env.terminal_files }}
+          else
+            echo "No Python files changed in openbb_terminal"
+          fi
+      - run: |
+          # Run linters for openbb_platform
+          if [ -n "${{ env.platform_files }}" ]; then
+            # Add mypy to this part of the linting workflow once we're ready
+            pylint ${{ env.platform_files }}
+          else
+            echo "No Python files changed in openbb_platform"
+          fi
 
   markdown-link-check:
     name: Markdown Linting

.github/workflows/nightly-build.yml

@@ -53,3 +53,12 @@ jobs:
         with:
           workflow: pypi-nightly.yml
           token: ${{ secrets.GITHUB_TOKEN }}
+
+  trigger-api-integration-test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Trigger Platform API Integration Test
+        uses: aurelien-baudet/workflow-dispatch@v2
+        with:
+          workflow: platform-api-integration-test.yml
+          token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/platform-api-integration-test.yml

@@ -0,0 +1,120 @@
+name: API Integration Tests
+
+on:
+  workflow_dispatch:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  api-integration-tests:
+    runs-on: ubuntu-latest
+    env:
+      SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}
+      OPENBB_API_AUTH: true
+      OPENBB_API_USERNAME: "USER"
+      OPENBB_API_PASSWORD: "PASS"
+
+    steps:
+      - name: Slack Notification - Starting
+        uses: act10ns/slack@v1
+        with:
+          status: starting
+          channel: "#workflows"
+          message: Starting API Integration Test...
+        if: always()
+
+      - name: Checkout code
+        uses: actions/checkout@v3
+
+      - name: Checkout specific ref
+        if: github.event_name == 'pull_request'
+        run: git fetch origin ${{ github.event.pull_request.head.ref }} && git checkout FETCH_HEAD
+
+      - name: Setup Python 3.9
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.9"
+          architecture: x64
+
+      - name: Load cached venv
+        id: cached-python-environment
+        uses: actions/cache@v3
+        with:
+          path: runner_env
+          key: runner_env-${{ runner.os }}-v1-${{ hashFiles('**/poetry.lock', '**/dev_install.py') }}
+
+      - name: Install Dependencies
+        if: steps.cached-python-environment.outputs.cache-hit != 'true'
+        run: |
+          python -m venv runner_env
+          source runner_env/bin/activate
+          pip install poetry toml
+          python openbb_platform/dev_install.py -e all
+
+      - name: Populate System Setting Files
+        run: |
+          mkdir -p ~/.openbb_platform
+          touch ~/.openbb_platform/system_settings.json
+          echo '{"log_collect":false}' > ~/.openbb_platform/system_settings.json
+          cat ~/.openbb_platform/system_settings.json
+
+      - name: Create and Populate user_settings.json from GitHub Secrets
+        run: |
+          touch ~/.openbb_platform/user_settings.json
+          echo '{
+            "credentials": {
+              "benzinga_api_key": "${{ secrets.BENZINGA_API_KEY }}",
+              "fmp_api_key": "${{ secrets.FMP_API_KEY }}",
+              "polygon_api_key": "${{ secrets.POLYGON_API_KEY }}",
+              "alpha_vantage_api_key": "${{ secrets.ALPHA_VANTAGE_API_KEY }}",
+              "fred_api_key": "${{ secrets.FRED_API_KEY }}",
+              "intrinio_api_key": "${{ secrets.INTRINIO_API_KEY }}",
+              "tradingeconomics_api_key": "${{ secrets.TRADINGECONOMICS_API_KEY }}",
+              "quandl_api_key": "${{ secrets.QUANDL_API_KEY }}",
+              "biztoc_api_key": "${{ secrets.BIZTOC_API_KEY }}",
+              "nasdaq_api_key": "${{ secrets.NASDAQ_API_KEY }}",
+              "tiingo_token": "${{ secrets.TIINGO_TOKEN }}"
+            }
+          }' > ~/.openbb_platform/user_settings.json
+
+      - name: Launch the Uvicorn Process
+        run: |
+          source runner_env/bin/activate
+          uvicorn openbb_core.api.rest_api:app --host 0.0.0.0 --port 8000 --reload &
+
+      - name: Wait for 42 seconds
+        run: sleep 42
+
+      - name: Run API Integration Tests
+        run: |
+          source runner_env/bin/activate
+          pip list
+          cd openbb_platform
+          pytest -m integration
+
+      - name: Cache pip packages
+        uses: actions/cache@v2
+        with:
+          path: ~/.cache/pip
+          key: runner_env-${{ runner.os }}-v1-${{ hashFiles('**/poetry.lock', '**/dev_install.py') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-
+
+      - name: Upload summary to Slack
+        uses: adrey/slack-file-upload-action@master
+        with:
+          token: ${{ secrets.SLACK_API_TOKEN }}
+          initial_comment: "API Integration test summary"
+          title: "Integration test summary"
+          path: summary.txt
+          channel: ${{ secrets.SLACK_CHANNEL_ID }}
+
+      - name: Slack Notification - Success/Failure
+        uses: act10ns/slack@v1
+        with:
+          status: ${{ job.status }}
+          steps: ${{ toJson(steps) }}
+          channel: "#workflows"
+        if: always()

.github/workflows/platform-core.yml

@@ -0,0 +1,46 @@
+name: Test Platform V4
+
+on:
+  pull_request:
+    branches:
+      - develop
+    paths:
+      - 'openbb_platform/**'
+
+concurrency:
+    group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.sha }}
+    cancel-in-progress: true
+
+jobs:
+  unit_tests:
+    runs-on: ubuntu-latest
+    strategy:
+        fail-fast: false
+
+        matrix:
+          python_version:
+            ["3.9", "3.10", "3.11"]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+        with:
+          ref: ${{ github.event.pull_request.head.sha }}
+      - name: Install Python ${{ matrix.python_version }}
+        uses: actions/setup-python@v4
+        with:
+            python-version: ${{ matrix.python_version }}
+            allow-prereleases: true
+            cache: "pip"
+
+      - name: Cache pip packages
+        uses: actions/cache@v2
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-${{ matrix.python-version }}-${{ hashFiles('openbb_platform/poetry.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-pip-
+
+      - name: Run tests
+        run: |
+          pip install nox
+          nox -s tests --python ${{ matrix.python_version }}

.github/workflows/pypi-nightly.yml

@@ -12,37 +12,52 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-    - uses: actions/checkout@v3
-    - name: Setup Python 3.9
-      uses: actions/setup-python@v4
-      with:
-        python-version: "3.9"
+      - uses: actions/checkout@v3
+      - name: Setup Python 3.9
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.9"
 
-    - name: Update version in pyproject.toml and Edit docs
-      run: |
-        sed -i 's/name = ".*"/name = "openbb-nightly"/' pyproject.toml
-        sed -i "3s/version = \"\(.*\)\"/version = \"\1.dev$(date +%Y%m%d)\"/" pyproject.toml
-        head pyproject.toml
-        sed -i 's/pip install openbb/pip install openbb-nightly/g; s/from openbb_terminal.sdk import openbb/from openbb_terminal.sdk import openbb/g' ./website/pypi.md
+      - name: Install pypa/build
+        run: >-
+          python -m
+          pip install
+          build
+          --user
 
+      - name: OpenBBTerminal - Update version in pyproject.toml and Edit docs
+        run: |
+          sed -i 's/name = ".*"/name = "openbb-terminal-nightly"/' pyproject.toml
+          sed -i "3s/version = \"\(.*\)\"/version = \"\1.dev$(date +%Y%m%d%H%M)\"/" pyproject.toml
+          sed -i 's|pip install openbb-terminal|pip install openbb-terminal-nightly|g' ./website/pypi.md
 
-    - name: Install pypa/build
-      run: >-
-        python -m
-        pip install
-        build
-        --user
+      - name: OpenBBTerminal - Build a binary wheel and a source tarball
+        run: >-
+          python -m
+          build
+          --sdist
+          --wheel
+          --outdir dist/
+          .
 
-    - name: Build a binary wheel and a source tarball
-      run: >-
-        python -m
-        build
-        --sdist
-        --wheel
-        --outdir dist/
-        .
+      - name: OpenBBTerminal - Publish distribution 📦 to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          packages-dir: dist/
+          password: ${{ secrets.NIGHTLY_PYPI_API_TOKEN_OBB_TERMINAL_NIGHTLY }}
 
-    - name: Publish distribution 📦 to PyPI
-      uses: pypa/gh-action-pypi-publish@release/v1
-      with:
-        password: ${{ secrets.NIGHTLY_PYPI_API_TOKEN }}
+      - name: OpenBB Platform - Update version in pyproject.toml and Edit docs
+        run: |
+          sed -i 's/name = ".*"/name = "openbb-nightly"/' openbb_platform/pyproject.toml
+          sed -i "3s/version = \"\(.*\)\"/version = \"\1.dev$(date +%Y%m%d%H%M)\"/" openbb_platform/pyproject.toml
+
+      - name: OpenBB Platform - Create the dinamically generated wheel
+        run: |
+          python -m pip install poetry toml
+          python build/pypi/openbb_platform/nightly.py
+
+      - name: OpenBB Platform - Publish distribution 📦 to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          packages-dir: openbb_platform/dist/
+          password: ${{ secrets.NIGHTLY_PYPI_API_TOKEN }}

.github/workflows/pypi.yml

@@ -3,8 +3,7 @@ name: Deploy to PyPI
 on:
   push:
     branches:
-      - release/*
-      - main
+      - release/v3/*
 
   workflow_dispatch:
 

.github/workflows/test_pypi_platform.yml

@@ -0,0 +1,68 @@
+name: Deploy the OpenBB Platform and the OpenBBTerminal to Test PyPI
+
+on:
+  push:
+    branches:
+      - feature/v4-pypi
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  deploy-test-pypi:
+    name: Build and publish 📦 to TestPyPI
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v3
+      - name: Setup Python 3.9
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.9"
+
+      - name: Install pypa/build
+        run: >-
+          python -m
+          pip install
+          build
+          --user
+
+      - name: OpenBBTerminal - Update version in pyproject.toml and Edit docs
+        run: |
+          sed -i 's/name = ".*"/name = "openbb-terminal-nightly"/' pyproject.toml
+          sed -i "3s/version = \"\(.*\)\"/version = \"\1.dev$(date +%Y%m%d%H%M)\"/" pyproject.toml
+          sed -i 's|pip install openbb-terminal|pip install openbb-terminal-nightly|g' ./website/pypi.md
+
+      - name: OpenBBTerminal - Build a binary wheel and a source tarball
+        run: >-
+          python -m
+          build
+          --sdist
+          --wheel
+          --outdir dist/
+          .
+
+      - name: OpenBBTerminal - Publish distribution 📦 to Test PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          packages-dir: dist/
+          password: ${{ secrets.TEST_PYPI_API_TOKEN_OBB_TERMINAL_NIGHTLY }}
+          repository-url: https://test.pypi.org/legacy/
+
+      - name: OpenBB Platform - Update version in pyproject.toml and Edit docs
+        run: |
+          sed -i 's/name = ".*"/name = "openbb-nightly"/' openbb_platform/pyproject.toml
+          sed -i "3s/version = \"\(.*\)\"/version = \"\1.dev$(date +%Y%m%d%H%M)\"/" openbb_platform/pyproject.toml
+
+      - name: OpenBB Platform - Create the dinamically generated wheel
+        run: |
+          python -m pip install poetry toml
+          python build/pypi/openbb_platform/nightly.py
+
+      - name: OpenBB Platform - Publish distribution 📦 to Test PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          packages-dir: openbb_platform/dist/
+          password: ${{ secrets.TEST_PYPI_API_TOKEN_OBB_PLATFORM_NIGHTLY }}
+          repository-url: https://test.pypi.org/legacy/