jerry/OpenBB
1
0
Fork 0
mirror of https://github.com/OpenBB-finance/OpenBB.git synced 2026-11-03 07:51:49 +00:00
Code Issues Packages Projects Releases Wiki Activity

[Feature] Improve PyPi publishing tool (#6542)

Browse Source 
* Update PUBLISH.md

* Refactor publish.py

* fix ruff check

* final publish.py

* feat: add option to change semantic version with poetry

* replace with publish --build cmd
This commit is contained in:
montezdesousa
2024-07-01 16:42:35 +01:00
committed by GitHub
parent c63b3c8bab
commit a064524642
3 changed files with 235 additions and 134 deletions
Download Patch File Download Diff File Expand all files Collapse all files

100
build/pypi/openbb_platform/PUBLISH.md vendored
View File

@@ -1,90 +1,60 @@
# Publishing to PyPI
## Pre-release procedure
## Pre-release
> [!WARNING]
> The `release` branch is solely for the purpose of publishing the package(s)! Any last minute changes should be made in appropriate PRs and merged to the `develop` branch. Once the `release` branch is created, the `develop` branch should be frozen for further commits.
> The `release` branch is solely for the purpose of publishing the package(s)! Any last minute changes should be made in appropriate PRs and merged to the `develop` branch. Once the `release` branch is created, the `develop` branch should be frozen for further commits until this process finishes.
> A `release` branch for a particular package should only cater to the changes for that package directory. For e.g. the `release/openbb-core-2.0.0` branch should only contain changes for the `openbb-core` package i.e. in the `openbb_platform/core` directory.
### Flow to display only files changes during release cycle in the `release/...` -> `main` PR
1. Merge main into develop before release branch exists, solve any conflict
2. In the branch `release/…` (before or after publishing)
3. `git merge main -X ours` (ignore the changes, we brought them in 1.)
4. `git commit -m "Merge branch 'main' into release/4.2.2"` (empty commit)
### Pre-release checklist
0. Merge `main` into `develop` and solve any conflict. You can do this by checking out `develop`, creating a branch `feature/merge-main-into-develop` and running `git merge main`. Merge this into `develop` if there are changes to commit.
1. Open a PR with the changes to be published in the format `release/<version>` (for e.g. `release/4.0.0` ). For a particular package use the format `release/<package>-<version>` (for e.g. `release/openbb-core-1.0.1`).
2. Ensure all the CI workflows pass.
3. Ensure all unit tests pass: `pytest openbb_platform -m "not integration"`
4. Ensure all integration tests pass: `pytest openbb_platform -m integration`
2. Bump `openbb` package version in `openbb_platform/pyproject.toml`.
## Release procedure
> [!WARNING]
> The version must be incremented before running the tests in 4. and 5., since there are tests that rely of this to check for deprecated endpoints.
3. Ensure all the CI workflows pass.
4. Ensure all unit tests pass: `pytest openbb_platform -m "not integration"`
5. Ensure all integration tests pass: `pytest openbb_platform -m integration`
## Release
> Ensure you have the appropriate credentials and permissions to publish to PyPI.
1. Run the following commands for publishing the packages to PyPI:
Consider using the `--dry-run` flag to check if everything is correct before publishing.
Also, it might be a good idea to run the script in batches to ensure that the packages are published correctly and the dependencies pick the correct versions.
> For a single package release, the following steps are optional since the package can be bumped manually.
1. For the core package run: `python build/pypi/openbb_platform/publish.py --core`
2. For the extension packages run: `python build/pypi/openbb_platform/publish.py --extensions`
3. For the `openbb` package - **which requires manual publishing** - do the following
3.1. Bump the `openbb` package version and the extension versions on `openbb_platform/pyproject.toml` to the latest version.
> [!TIP]
> Consider using the poetry plugin `up` for updating the extensions to the latest version:
> 1. `poetry self add poetry-plugin-up`
> 2. `poetry up --latest`
> [!WARNING]
> Create a new environment before proceeding.
> Make sure that only required extensions are installed
3.2. Run `pip install -e .` from `openbb_platform`
3.3. Re-build the static assets that are bundled with the package: `python -c "import openbb; openbb.build()"`
- Run `python -c "import openbb"` after building the static to check that no additional static is being built.
- Run any command to smoke test if the static assets are being built correctly.
3.4. Run unit tests to validate the existence of deprecated endpoints (or watch this through GitHub Actions)
3.5. Run `poetry publish --build` from `openbb_platform`
3.6. Run `poetry lock` from `openbb_platform`
> [!TIP]
> Note that, in order for packages to pick up the latest versions of dependencies, it is advised to clear the local cache of the dependencies:
>
> We can do that with `pip cache purge` and `poetry cache clear pypi --all`
>
> Also, sometimes there might be some delay in the PyPI API, so it might be necessary to wait a few minutes before publishing the next package.
> Consider using the `--dry-run` flag to check if everything is correct before publishing.
2. Merge the `release/<package>-<version>` branch to the `main` branch.
3. Run the `Deploy to GitHub Pages` action in [openbb-docs](https://github.com/OpenBB-finance/openbb-docs/actions). Go to the [docs](https://docs.openbb.co) website to see the changes.
> For a single package release (e.g. patching `openbb-core`), the following steps are not required since the package can be published manually.
## Post-release procedure
1. To publish `openbb-core` run: `python build/pypi/openbb_platform/publish.py --core`
2. To publish **ALL** extensions run: `python build/pypi/openbb_platform/publish.py --extensions`
3. To publish `openbb` (the main package) run: `python build/pypi/openbb_platform/publish.py --openbb`
> [!TIP]
> Note that, in order for packages to pick up the latest versions of dependencies, it is advised to clear the local cache of the dependencies: we can do that with `pip cache purge` and `poetry cache clear pypi --all`. Sometimes there might be some delay in the PyPI API, so it might be necessary to wait a few minutes for pip to pick the latest versions.
2. Publish the CLI
2.1. Bump `openbb` dependency on `cli/pyproject.toml` to the latest version
2.2. Run `poetry publish --build` from `cli`
3. Regenerate assets for external use by running `python assets/scripts/generate_extension_data.py`.
4. Merge `release/<package>-<version>` to the `main` branch.
5. Run `Deploy to GitHub Pages` action in [openbb-docs](https://github.com/OpenBB-finance/openbb-docs/actions) to update the documentation website. Go to [docs.openbb.co](https://docs.openbb.co) to see the changes.
## Post-release
1. Install the packages on Google Colaboratory via PyPi and test to check if everything is working as expected.
2. Install the packages in a new environment locally via PyPi and test to check if everything is working as expected.
3. Regenerate assets for external use by running `python assets/scripts/generate_extension_data.py`
4. Open a new PR with the `release/<package>-<version>` branch pointing to the `develop` branch.
5. Merge the `release/<package>-<version>` branch to the `develop` branch.
6. If any bugs are encountered, create a new branch - `hotfix` for `main` and `bugfix` for `develop` and merge them accordingly.
3. Open a new PR with the `release/<package>-<version>` branch pointing to the `develop` branch.
4. Merge the `release/<package>-<version>` branch to the `develop` branch.
5. If any bugs are encountered, create a new branch - `hotfix` for `main` and `bugfix` for `develop` and merge them accordingly.
### Generate the changelog
1. Run the changelog automation by using the "release_drafter" GA and passing the number of the previous Release.
1. Run the changelog automation by using the "release_drafter" GitHub action and input the number of the last release PR (not this one!).
2. Edit and make the changelog live on the repository.
3. Paste it in the platform-release-changelog Slack channel.
### Publish the CLI
1. Bump `openbb` dependency on `cli/pyproject.toml` to the latest version.
2. Run `poetry publish --build` from `cli`

261
build/pypi/openbb_platform/publish.py
View File

@@ -1,19 +1,23 @@
"""Publish the OpenBB Platform to PyPi."""
import argparse
import subprocess
import logging
import sys
from functools import partial
from pathlib import Path
from subprocess import PIPE, run
from typing import Literal
PLATFORM_PATH = Path(__file__).parent.parent.parent.parent.resolve() / "openbb_platform"
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
console = logging.StreamHandler()
logger.addHandler(console)
formatter = logging.Formatter("[%(levelname)s] %(message)s")
console.setFormatter(formatter)
CORE_PACKAGES = ["core"]
EXTENSION_PACKAGES = ["extensions", "providers", "obbject_extensions"]
CMD = [sys.executable, "-m", "poetry"]
EXTENSION_DEPENDENCIES_UPDATE_CMD = ["add", "openbb-core=latest", "--lock"]
VERSION_BUMP_CMD = ["version", "patch"]
PUBLISH_CMD = ["publish", "--build"]
DIR_PLATFORM = Path(__file__).parent.parent.parent.parent.resolve() / "openbb_platform"
DIR_CORE = ["core"]
DIR_EXTENSIONS = ["extensions", "providers", "obbject_extensions"]
def parse_args():
@@ -27,9 +31,16 @@ def parse_args():
"-e",
"--extensions",
action="store_true",
help="Publish extension packages.",
help="Publish extension packages, such as openbb-equity or openbb-fmp.",
dest="extensions",
)
parser.add_argument(
"-o",
"--openbb",
action="store_true",
help="Publish openbb package.",
dest="openbb",
)
parser.add_argument(
"--dry-run",
action="store_true",
@@ -37,74 +48,192 @@ def parse_args():
default=False,
dest="dry_run",
)
parser.add_argument(
"-v",
"--verbose",
action="store_true",
help="Run the tool in verbose mode.",
default=False,
dest="verbose",
)
parser.add_argument(
"-y",
"--yes",
action="store_true",
help="Answer input questions.",
default=False,
dest="yes",
)
parser.add_argument(
"--semver",
help="Semantic version.",
default="patch",
choices=["patch", "minor", "major"],
dest="semver",
)
return parser.parse_args()
def update_extension_dependencies(path: Path):
"""Update the extension dependencies"""
subprocess.run(
CMD + EXTENSION_DEPENDENCIES_UPDATE_CMD, # noqa: S603
cwd=path.parent,
def publish(
dry_run: bool = False,
core: bool = False,
extensions: bool = False,
openbb: bool = False,
verbose: bool = False,
semver: Literal["patch", "minor", "major"] = "patch",
):
"""Publish the Platform to PyPi with optional core or extensions."""
package_directories = []
if core:
package_directories.extend(DIR_CORE)
if extensions:
package_directories.extend(DIR_EXTENSIONS)
partial_run = partial(
run,
check=True,
stdout=None if verbose else PIPE,
stderr=None if verbose else PIPE,
)
def bump_version(path: Path):
"""Bump the version of the package"""
print(f"\n> {path.parent.stem}") # noqa: T201
subprocess.run(CMD + VERSION_BUMP_CMD, cwd=path.parent, check=True) # noqa: S603
def publish(dry_run: bool = False, core: bool = False, extensions: bool = False):
"""Publish the Platform to PyPi with optional core or extensions."""
package_paths = []
if core:
print("Working with core packages...") # noqa: T201
package_paths.extend(CORE_PACKAGES)
if extensions:
print("Working with extensions...") # noqa: T201
package_paths.extend(EXTENSION_PACKAGES)
for sub_path in package_paths:
is_extension = sub_path in EXTENSION_PACKAGES
for path in sorted(PLATFORM_PATH.rglob(f"{sub_path}/**/pyproject.toml")):
for _dir in package_directories:
is_extension = _dir in DIR_EXTENSIONS
paths = [
p
for p in sorted(DIR_PLATFORM.rglob(f"{_dir}/**/pyproject.toml"))
if "devtools" not in str(p)
]
total = len(paths)
logger.info("~~~ /%s ~~~", _dir)
for i, path in enumerate(paths):
logger.info(
"🚀 (%s/%s) Publishing openbb-%s...",
i + 1,
total,
path.parent.stem.replace("_", "-"),
)
try:
# Update dependencies
if is_extension and "devtools" not in str(path):
update_extension_dependencies(path)
# Bump version
bump_version(path)
# Publish (if not dry run)
if not dry_run:
subprocess.run(
CMD + PUBLISH_CMD, # noqa: S603
# Update openbb-core to latest in each pyproject.toml
if is_extension:
partial_run(
[
sys.executable,
"-m",
"poetry",
"add",
"openbb-core=latest",
"--lock",
],
cwd=path.parent,
check=True, # noqa: S603
)
# Bump pyproject.toml version
partial_run(
[sys.executable, "-m", "poetry", "version", semver],
cwd=path.parent,
)
# Publish (if not dry running)
if not dry_run:
partial_run(
[
sys.executable,
"-m",
"poetry",
"build",
"publish",
"--build",
],
cwd=path.parent,
)
logger.info("✅ Success")
except Exception as e:
print(f"Error publishing {path.parent}:\n{e}") # noqa: T201
logger.error("❌ Failed to publish %s:\n\n%s", path.parent.stem, e)
if openbb:
STEPS = 7
logger.info("~~~ /openbb ~~~")
logger.info("🧩 (1/%s) Installing poetry-plugin-up...", STEPS)
partial_run(
["pip", "install", "poetry-plugin-up"],
cwd=DIR_PLATFORM,
)
logger.info("⏫ (2/%s) Updating openbb pyproject.toml...", STEPS)
partial_run(
[sys.executable, "-m", "poetry", "up", "--latest"],
cwd=DIR_PLATFORM,
)
logger.info("🔒 (3/%s) Writing openbb poetry.lock...", STEPS)
partial_run(
[sys.executable, "-m", "poetry", "up", "--latest"],
cwd=DIR_PLATFORM,
)
logger.info("📍 (4/%s) Installing openbb from /%s...", STEPS, DIR_PLATFORM.stem)
partial_run(
["pip", "install", "-U", "--editable", "."],
cwd=DIR_PLATFORM,
)
logger.info("🚧 (5/%s) Building python interface...", STEPS)
result = run(
[sys.executable, "-c", "import openbb; openbb.build()"], # noqa: S603
cwd=DIR_PLATFORM,
check=True,
capture_output=True,
text=True,
)
if verbose:
logger.info("Captured result -> %s", result)
if result.stderr:
logger.error("❌ stderr is not empty!")
raise Exception(result.stderr)
logger.info("🧪 (6/%s) Unit testing...", STEPS)
partial_run(
["pytest", "tests", "-m", "not integration"],
cwd=DIR_PLATFORM,
)
logger.info("🚭 (7/%s) Smoke testing...", STEPS)
# TODO: Improve smoke test coverage here
result = run( # noqa: S603
[ # noqa: S603
sys.executable,
"-c",
"from openbb import obb; obb.equity.price.historical('AAPL', provider='yfinance')",
],
cwd=DIR_PLATFORM,
check=True,
capture_output=True,
text=True,
)
if verbose:
logger.info("Captured result -> %s", result)
if result.stderr:
logger.error("❌ stderr is not empty!")
raise Exception(result.stderr)
logger.info("👍 Great success! 👍")
logger.info(
"Confirm any files changed and run `poetry publish --build` from /openbb_platform"
)
if __name__ == "__main__":
msg = """
You are about to publish a new version of OpenBB Platform to PyPI.
Please ensure you've read the "PUBLISH.md" file.
Also, please double check with `poetry config --list` if you're publishing to PyPI or TestPyPI.
"""
msg = (
"\nYou are about to publish a new version of OpenBB Platform to PyPI."
+ "\n\nPlease ensure you've read the PUBLISH.md file and double check with "
+ "`poetry config --list` if you're publishing to PyPI or TestPyPI."
)
args = parse_args()
res = input(f"{msg}\n\nDo you want to continue? [y/N] ")
if not args.dry_run:
msg += "\n\n🛑 You are NOT using the --dry-run flag!"
res = "y" if args.yes else input(f"{msg}\n\nDo you want to continue? [y/N] ")
print("") # noqa: T201
if res.lower() == "y":
publish(dry_run=args.dry_run, core=args.core, extensions=args.extensions)
openbb_package_msg = """
In order to publish the `openbb` package you need to manually update the
versions in the `pyproject.toml` file. Follow the steps below:
1. Bump version: `poetry version prerelease --next-phase`
2. Re-build the static assets that are bundled with the package
3. Publish: `poetry publish --build`
"""
print(openbb_package_msg) # noqa: T201
logger.info("Process started. Press Ctrl+C to abort.")
try:
publish(
dry_run=args.dry_run,
core=args.core,
extensions=args.extensions,
openbb=args.openbb,
verbose=args.verbose,
semver=args.semver,
)
except KeyboardInterrupt:
sys.exit(1)

8
openbb_platform/core/openbb_core/app/static/utils/linters.py
View File

@@ -36,10 +36,12 @@ class Linters:
self.console.log(f"\n* {linter}")
self.print_separator("^")
command = [linter] + list(self.directory.glob("*.py"))
command = [linter]
if flags:
command.extend(flags) # type: ignore
subprocess.run(command, check=False) # noqa: S603
subprocess.run(
command + list(self.directory.glob("*.py")), check=False # noqa: S603
)
self.print_separator("-")
else:
@@ -54,7 +56,7 @@ class Linters:
def ruff(self):
"""Run ruff."""
flags = ["--fix"]
flags = ["check", "--fix"]
if not self.verbose and not Env().DEBUG_MODE:
flags.append("--silent")
self.run(linter="ruff", flags=flags)