mirror of
https://github.com/OpenBB-finance/OpenBB.git
synced 2026-11-12 12:22:35 +00:00
[V5] Refactor openbb-oecd for full SDMX 2 API Exposure and V5 Packaging (#7413)
* stash some changes * add more robust testing * mypy * point PR at V5 * introduce spec file * codespell * test fix * fix workflow environment setup * fix workflow environment setup * fix workflow environment setup * add pyyaml to dependencies * split lint jobs * fix workflow environment setup * fix workflow environment setup * workflow env setup * workflow env setup * clean up code comments * add auth hook entrypoints * codespell * add codegen feature * codespell * move _unpack into dispatchers for consistency with codegen packages * surface nested models in the response * fix missing coverage in CI * socrata updates * test fix * detect plotly output * add --include and --exclude flags from generate-extension command * refactor openbb-platform-api for V5 operations and spec file handling * delete poetry.lock * add test workflow * add temp branch id for test workflow * add httpx to dev dependencies * add a conftest.py * tweak conftest * tweak conftest * python 3.10 issues * more python 3.10 * more python 3.10 * more python 3.10 * more python 3.10 * fix url template resolution * multi-spec support * fix a windows path issue * merge branch develop * table builder, integration tests, and more * linting * merge branch develop * bit of cleanup * some more updates, cleanup, and apps.json * codespell * black * integration test params and linting * mypy * pylint * black * more pylint * recapture teest cassettes * black again * more linting * table presentation choices * black * fix merge conflict * split up metadata module and reformat cache to use msgpack * mypy * regenerate lock file * use json and include dependencies to make openbb-oecd a standalone install * pylint * more linting * serve metadata as dependency instead of assign and initialize * fix integration_tests_testers to not count dependency injections as paramters * add assets to git ignore * ruff * v5 refactor and publish workflow * test coverage and ty fixes * codespell * update lock file * workflow setup * ty check * workflow setup * Scope PR to OECD changes only Restore non-OECD files in the diff against v5 to their v5 state. The branch was originally created off feature/cli-2 and inherited 186 files of unrelated changes (cli, agent_server, mcp_server, platform_api, core/, .github/workflows/* not oecd). Those have nothing to do with the openbb-oecd v5 refactor and were polluting the PR diff. * scope cleanup: include leaked v5-resident files (general-linting workflow, platform_api test fixtures) --------- Co-authored-by: deeleeramone <> Co-authored-by: Copilot <copilot@github.com>
This commit is contained in:
Danglewood
212
.github/workflows/publish-openbb-oecd.yml
vendored
Normal file
212
.github/workflows/publish-openbb-oecd.yml
vendored
Normal file
@@ -0,0 +1,212 @@
|
||||
name: 📦 publish openbb-oecd
|
||||
|
||||
# Triggers exclusively on the merge of a ``release/openbb-oecd-v*`` branch
|
||||
# into ``develop``. No push / tag triggers. Three sequential jobs —
|
||||
# build once, test the built artifact across the full cross-platform
|
||||
# matrix, and only publish to PyPI if the entire matrix is green.
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [closed]
|
||||
branches: [develop, v5]
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
version-check:
|
||||
name: Verify version bump
|
||||
if: ${{ (github.event_name == 'pull_request' && github.event.pull_request.merged == true && startsWith(github.event.pull_request.head.ref, 'release/openbb-oecd-v')) || github.event_name == 'workflow_dispatch' }}
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
version: ${{ steps.parse.outputs.version }}
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
working-directory: openbb_platform/providers/oecd
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
ref: develop
|
||||
|
||||
- name: Parse [project].version from pyproject.toml
|
||||
id: parse
|
||||
run: |
|
||||
VERSION=$(grep -m1 -E '^version[[:space:]]*=' pyproject.toml | sed -E 's/^version[[:space:]]*=[[:space:]]*"([^"]+)".*/\1/')
|
||||
if [ -z "$VERSION" ]; then
|
||||
echo "::error::Could not parse [project].version from pyproject.toml."
|
||||
exit 1
|
||||
fi
|
||||
echo "Local version: $VERSION"
|
||||
echo "version=$VERSION" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Fail if version already exists on PyPI
|
||||
env:
|
||||
VERSION: ${{ steps.parse.outputs.version }}
|
||||
run: |
|
||||
STATUS=$(curl -sS -o /dev/null -w "%{http_code}" "https://pypi.org/pypi/openbb-oecd/${VERSION}/json")
|
||||
case "$STATUS" in
|
||||
200)
|
||||
echo "::error::openbb-oecd ${VERSION} is already published on PyPI."
|
||||
echo "Bump openbb_platform/providers/oecd/pyproject.toml [project].version and re-merge a fresh release branch."
|
||||
exit 1
|
||||
;;
|
||||
404)
|
||||
echo "openbb-oecd ${VERSION} is not yet on PyPI — proceeding."
|
||||
;;
|
||||
*)
|
||||
echo "::error::Unexpected status $STATUS from PyPI JSON API. Aborting to avoid a silent skip."
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
- name: Verify release branch name matches pyproject version
|
||||
if: github.event_name == 'pull_request'
|
||||
env:
|
||||
VERSION: ${{ steps.parse.outputs.version }}
|
||||
BRANCH: ${{ github.event.pull_request.head.ref }}
|
||||
run: |
|
||||
EXPECTED="release/openbb-oecd-v${VERSION}"
|
||||
if [ "$BRANCH" != "$EXPECTED" ]; then
|
||||
echo "::error::Release branch '${BRANCH}' does not match pyproject version '${VERSION}'."
|
||||
echo "Expected branch name: '${EXPECTED}'."
|
||||
echo "Either rename the release branch or update [project].version in pyproject.toml so they agree."
|
||||
exit 1
|
||||
fi
|
||||
echo "Branch '${BRANCH}' matches pyproject version '${VERSION}'."
|
||||
|
||||
build:
|
||||
name: Build sdist + wheel
|
||||
needs: version-check
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
working-directory: openbb_platform/providers/oecd
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
ref: develop
|
||||
|
||||
- uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.11"
|
||||
|
||||
- run: python -m pip install --upgrade pip uv
|
||||
|
||||
- name: Restore OECD SDMX cache (warming pool for PR workflow)
|
||||
# Releases always force-rebuild (see env below); this step exists
|
||||
# so the post-build ``save`` writes the fresh artifact back into
|
||||
# the shared cache pool keyed by ``generate_cache.py``'s hash.
|
||||
# The PR-test workflow reads from the same pool.
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: openbb_platform/providers/oecd/openbb_oecd/assets/oecd_cache.json.xz
|
||||
key: oecd-sdmx-${{ hashFiles('openbb_platform/providers/oecd/openbb_oecd/utils/generate_cache.py') }}
|
||||
|
||||
- name: Build sdist + wheel (fresh OECD cache)
|
||||
# ``OPENBB_OECD_FORCE_CACHE_REBUILD=1`` makes the build hook
|
||||
# bypass its idempotent skip and regenerate against live OECD
|
||||
# SDMX so the published artifact is always current.
|
||||
env:
|
||||
OPENBB_OECD_FORCE_CACHE_REBUILD: "1"
|
||||
run: uv build --no-sources
|
||||
|
||||
- uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: openbb-oecd-dist
|
||||
path: openbb_platform/providers/oecd/dist/
|
||||
if-no-files-found: error
|
||||
retention-days: 7
|
||||
|
||||
test:
|
||||
name: Test (${{ matrix.os }} / Python ${{ matrix.python_version }})
|
||||
needs: build
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
# Linux: full sweep across every released Python.
|
||||
- { os: ubuntu-latest, python_version: "3.10" }
|
||||
- { os: ubuntu-latest, python_version: "3.11" }
|
||||
- { os: ubuntu-latest, python_version: "3.12" }
|
||||
- { os: ubuntu-latest, python_version: "3.13" }
|
||||
- { os: ubuntu-latest, python_version: "3.14" }
|
||||
- { os: ubuntu-latest, python_version: "3.15" }
|
||||
# macOS + Windows: min and max supported.
|
||||
- { os: macos-latest, python_version: "3.10" }
|
||||
- { os: macos-latest, python_version: "3.15" }
|
||||
- { os: windows-latest, python_version: "3.10" }
|
||||
- { os: windows-latest, python_version: "3.14" }
|
||||
runs-on: ${{ matrix.os }}
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
working-directory: openbb_platform/providers/oecd
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
ref: develop
|
||||
|
||||
- uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: ${{ matrix.python_version }}
|
||||
allow-prereleases: true
|
||||
|
||||
- uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: openbb-oecd-dist
|
||||
path: openbb_platform/providers/oecd/dist
|
||||
|
||||
- run: python -m pip install --upgrade pip uv
|
||||
|
||||
- name: Seed the cache asset from the built wheel
|
||||
# Extract the bundled cache out of the wheel into the source
|
||||
# tree so the editable install below sees it on disk and the
|
||||
# build hook's idempotent ``if _CACHE_PATH.exists(): reuse``
|
||||
# branch fires — guaranteeing no OECD fetch in this job.
|
||||
run: |
|
||||
WHEEL=$(ls dist/openbb_oecd-*.whl | head -1)
|
||||
mkdir -p openbb_oecd/assets
|
||||
python -c "
|
||||
import sys, zipfile, shutil
|
||||
wheel = sys.argv[1]
|
||||
dest = sys.argv[2]
|
||||
with zipfile.ZipFile(wheel) as z:
|
||||
with z.open('openbb_oecd/assets/oecd_cache.json.xz') as src, open(dest, 'wb') as out:
|
||||
shutil.copyfileobj(src, out)
|
||||
" "$WHEEL" openbb_oecd/assets/oecd_cache.json.xz
|
||||
|
||||
- name: Install openbb-oecd from source (+ dev group)
|
||||
# Source install via ``-e .`` reads ``[tool.uv.sources]`` and
|
||||
# resolves ``openbb-core[pandas]`` from the sibling tree at
|
||||
# ``../../core``. Installing the wheel directly would fail
|
||||
# dependency resolution until ``openbb-core>=2.0.0`` ships
|
||||
# to PyPI; the wheel is still the artifact uploaded to PyPI
|
||||
# by the ``publish`` job — this matrix just validates the
|
||||
# source the wheel was built from. ``--group dev`` installs
|
||||
# ``openbb-devtools``, which bundles pytest, pytest-asyncio,
|
||||
# pytest-cov, pytest-recorder, etc.
|
||||
run: |
|
||||
uv pip install --system -e .
|
||||
uv pip install --system --group dev
|
||||
|
||||
- name: Run unit tests
|
||||
run: pytest tests --cov=openbb_oecd --cov-report=term-missing
|
||||
|
||||
publish:
|
||||
name: Publish to PyPI
|
||||
needs: test
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
id-token: write # required for PyPI OIDC trusted publishing
|
||||
steps:
|
||||
- uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: openbb-oecd-dist
|
||||
path: dist
|
||||
|
||||
- uses: pypa/gh-action-pypi-publish@release/v1
|
||||
with:
|
||||
packages-dir: dist
|
||||
228
.github/workflows/test-unit-openbb-oecd.yml
vendored
Normal file
228
.github/workflows/test-unit-openbb-oecd.yml
vendored
Normal file
@@ -0,0 +1,228 @@
|
||||
name: 🧪 test openbb-oecd
|
||||
|
||||
# PR-test for the OECD provider. Triggers when a PR touches any file
|
||||
# under ``openbb_platform/providers/oecd/`` or this workflow itself.
|
||||
#
|
||||
# Build-once, test-many: a single ``build`` job materializes the OECD
|
||||
# SDMX cache (or restores it from the shared cache pool) and produces
|
||||
# the wheel; the ``test`` matrix downloads that exact wheel and runs
|
||||
# pytest under every supported (OS, Python) combination.
|
||||
#
|
||||
# The OECD cache asset is cached across runs via ``actions/cache``,
|
||||
# keyed solely on the SHA of ``generate_cache.py``. OECD metadata is
|
||||
# stable over weeks/months and the cassette-driven test suite never
|
||||
# hits the live API, so we accept arbitrary cache age — the only thing
|
||||
# that should invalidate is a logic change to the generator itself.
|
||||
# A manual override is exposed as a ``workflow_dispatch`` input for
|
||||
# emergency cache busting without having to touch the generator.
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [develop, main, v5, feature/v5-platform-api]
|
||||
paths:
|
||||
- "openbb_platform/providers/oecd/**"
|
||||
- ".github/workflows/test-unit-openbb-oecd.yml"
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
force_cache_rebuild:
|
||||
description: "Force the build hook to regenerate the OECD cache from live SDMX (ignores actions/cache)."
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
# Cancel in-flight matrix runs when a new commit lands on the same PR.
|
||||
group: test-openbb-oecd-${{ github.event.pull_request.number || github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
build:
|
||||
name: Build sdist + wheel
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
working-directory: openbb_platform/providers/oecd
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.11"
|
||||
|
||||
- run: python -m pip install --upgrade pip uv
|
||||
|
||||
- name: Restore OECD SDMX cache
|
||||
# The cache is keyed solely by the generator's content hash.
|
||||
# Same hash → same cache, no time-based expiry. ``restore-keys``
|
||||
# is a defensive fallback: if the exact key misses (e.g. a tiny
|
||||
# comment-only change to generate_cache.py invalidated the
|
||||
# hash), pull the most recent prior cache so the hook has a
|
||||
# baseline file on disk and the worst case is the same as the
|
||||
# current force-rebuild path.
|
||||
id: oecd-cache
|
||||
uses: actions/cache@v4
|
||||
with:
|
||||
path: openbb_platform/providers/oecd/openbb_oecd/assets/oecd_cache.json.xz
|
||||
key: oecd-sdmx-${{ hashFiles('openbb_platform/providers/oecd/openbb_oecd/utils/generate_cache.py') }}
|
||||
restore-keys: |
|
||||
oecd-sdmx-
|
||||
|
||||
- name: Build sdist + wheel
|
||||
# Honor the cache hit unless the manual override is set. On
|
||||
# exact-key cache-hit, the build hook sees the file already on
|
||||
# disk and skips the 4-minute OECD fetch.
|
||||
env:
|
||||
OPENBB_OECD_FORCE_CACHE_REBUILD: >-
|
||||
${{ (inputs.force_cache_rebuild == true || steps.oecd-cache.outputs.cache-hit != 'true') && '1' || '0' }}
|
||||
run: uv build --no-sources
|
||||
|
||||
- uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: openbb-oecd-dist
|
||||
path: openbb_platform/providers/oecd/dist/
|
||||
if-no-files-found: error
|
||||
retention-days: 7
|
||||
|
||||
lint:
|
||||
name: Lint (ruff + ty)
|
||||
# ``needs: build`` is load-bearing: ``build`` is the SOLE owner of
|
||||
# the OECD SDMX cache build. By the time this job runs, ``build``
|
||||
# has uploaded the wheel containing a fresh cache asset, and this
|
||||
# job consumes it — guaranteeing the build hook's regenerate path
|
||||
# NEVER fires in this job. There is no parallel cache-fetch path.
|
||||
needs: build
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
working-directory: openbb_platform/providers/oecd
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: "3.11"
|
||||
|
||||
- run: python -m pip install --upgrade pip uv
|
||||
|
||||
- uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: openbb-oecd-dist
|
||||
path: openbb_platform/providers/oecd/dist
|
||||
|
||||
- name: Seed the cache asset from the built wheel
|
||||
# Extract the bundled cache out of the wheel into the source
|
||||
# tree at the path the build hook checks. With the file already
|
||||
# present, the hook's idempotent ``if _CACHE_PATH.exists() and
|
||||
# not force: reuse`` branch fires on the editable install
|
||||
# below — zero chance of an OECD fetch.
|
||||
run: |
|
||||
WHEEL=$(ls dist/openbb_oecd-*.whl | head -1)
|
||||
mkdir -p openbb_oecd/assets
|
||||
python -c "
|
||||
import sys, zipfile, shutil
|
||||
wheel = sys.argv[1]
|
||||
dest = sys.argv[2]
|
||||
with zipfile.ZipFile(wheel) as z:
|
||||
with z.open('openbb_oecd/assets/oecd_cache.json.xz') as src, open(dest, 'wb') as out:
|
||||
shutil.copyfileobj(src, out)
|
||||
" "$WHEEL" openbb_oecd/assets/oecd_cache.json.xz
|
||||
ls -la openbb_oecd/assets/oecd_cache.json.xz
|
||||
|
||||
- name: Install openbb-oecd in editable mode (+ dev group)
|
||||
# ``-e .`` runs the Hatchling build hook, which reuses the
|
||||
# cache asset already on disk (seeded above). ``[tool.uv.sources]``
|
||||
# pins openbb-core / openbb-devtools to the sibling source trees
|
||||
# so ty sees the exact types about to ship together — not
|
||||
# whatever's on PyPI. ``--group dev`` pulls in ``openbb-devtools``
|
||||
# (ruff, ty, pytest-asyncio, etc.) so lint tooling versions stay
|
||||
# pinned in one place rather than hardcoded across workflows.
|
||||
run: |
|
||||
uv pip install --system -e .
|
||||
uv pip install --system --group dev
|
||||
|
||||
- name: ruff check
|
||||
run: ruff check openbb_oecd tests hatch_build.py
|
||||
|
||||
- name: ruff format --check
|
||||
run: ruff format --check openbb_oecd tests hatch_build.py
|
||||
|
||||
- name: ty check
|
||||
# ``ty`` discovers the system Python's site-packages — the
|
||||
# editable install above means all transitive types are
|
||||
# importable and ty can report real semantic errors, not just
|
||||
# "object of type X may not be iterable" warnings against
|
||||
# unresolved bare ``Sized`` / ``Unknown``.
|
||||
run: ty check openbb_oecd
|
||||
|
||||
test:
|
||||
name: Test (${{ matrix.os }} / Python ${{ matrix.python_version }})
|
||||
needs: build
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- { os: ubuntu-latest, python_version: "3.10" }
|
||||
- { os: ubuntu-latest, python_version: "3.11" }
|
||||
- { os: ubuntu-latest, python_version: "3.12" }
|
||||
- { os: ubuntu-latest, python_version: "3.13" }
|
||||
- { os: ubuntu-latest, python_version: "3.14" }
|
||||
- { os: ubuntu-latest, python_version: "3.15" }
|
||||
- { os: macos-latest, python_version: "3.10" }
|
||||
- { os: macos-latest, python_version: "3.15" }
|
||||
- { os: windows-latest, python_version: "3.10" }
|
||||
- { os: windows-latest, python_version: "3.14" }
|
||||
runs-on: ${{ matrix.os }}
|
||||
defaults:
|
||||
run:
|
||||
shell: bash
|
||||
working-directory: openbb_platform/providers/oecd
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
|
||||
- uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version: ${{ matrix.python_version }}
|
||||
allow-prereleases: true
|
||||
|
||||
- uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: openbb-oecd-dist
|
||||
path: openbb_platform/providers/oecd/dist
|
||||
|
||||
- run: python -m pip install --upgrade pip uv
|
||||
|
||||
- name: Seed the cache asset from the built wheel
|
||||
# Extract the bundled cache out of the wheel into the source
|
||||
# tree at the path the build hook checks. With the file
|
||||
# already present, the editable install below short-circuits
|
||||
# the hook's regenerate branch — zero chance of an OECD fetch.
|
||||
run: |
|
||||
WHEEL=$(ls dist/openbb_oecd-*.whl | head -1)
|
||||
mkdir -p openbb_oecd/assets
|
||||
python -c "
|
||||
import sys, zipfile, shutil
|
||||
wheel = sys.argv[1]
|
||||
dest = sys.argv[2]
|
||||
with zipfile.ZipFile(wheel) as z:
|
||||
with z.open('openbb_oecd/assets/oecd_cache.json.xz') as src, open(dest, 'wb') as out:
|
||||
shutil.copyfileobj(src, out)
|
||||
" "$WHEEL" openbb_oecd/assets/oecd_cache.json.xz
|
||||
|
||||
- name: Install openbb-oecd from source (+ dev group)
|
||||
# Source install via ``-e .`` reads ``[tool.uv.sources]`` and
|
||||
# resolves ``openbb-core`` from the sibling tree at ``../../core``,
|
||||
# which is the only place ``openbb-core[pandas]>=2.0.0`` exists
|
||||
# until that version ships to PyPI. ``--group dev`` installs
|
||||
# ``openbb-devtools``, which bundles pytest, pytest-asyncio,
|
||||
# pytest-cov, pytest-recorder, etc. — the single pinned source
|
||||
# of test-tool versions for the whole platform.
|
||||
run: |
|
||||
uv pip install --system -e .
|
||||
uv pip install --system --group dev
|
||||
|
||||
- name: Run unit tests
|
||||
run: pytest tests --cov=openbb_oecd --cov-report=term-missing --cov-fail-under=100
|
||||
Reference in New Issue
Block a user