Finding interface changes by comparing python stubs files

Question

I am trying to determine the interfaces changes of the public parts of a python package to a previous version. This is to help with semantic versioning (MAJOR_CHANGE.MINOR_CHANGE.PATCH) The best way I can think of to do this is the following:

1. Generate stubs files for new version of package (you can do this using stubgen. This gives you the public interface for the whole package, sub packages and modules...
2. Compare stubs for new and old version of the package.
   • If there are no interface changes, then the new version will just +1 to the PATCH number
   • If the original interface remains the same but entirly new functions (or attributes) are added, this will be a minor change so +1 to MINOR
   • If any part of the original interface is changed, it will be a MAJOR breaking change so +1 to MAJOR

So basically, the question is, what is the best way to compare 2 packages of stubs files for these changes? We could compare the AST files although these do not include type info (we could try to use this to get around that https://github.com/python/typed_ast)

Example stubs file here:

# Stubs for positioning.point (Python 3.6)
#
# NOTE: This dynamically typed stub was automatically generated by stubgen.

import numpy as np
from .exceptions import UnacceptableCartesianCoordinates
from .frame import Frame
from .methods import check_frames_have_common_parent, get_coordinates_of_point_in_frame, lowest_common_parent
from typing import Any

class Point:
    def __init__(self, frame: Frame, point_coordinates: np.ndarray) -> None: ...
    @classmethod
    def from_cartesian(cls: Any, frame: Frame, x: float, y: float, z: float) -> Point: ...
    @classmethod
    def from_cylindrical(cls: Any, frame: Frame, r: float, phi: float, z: float) -> Point: ...
    @classmethod
    def from_spherical(cls: Any, frame: Frame, r: float, theta: float, phi: float) -> Point: ...
    @classmethod
    def from_old_point_in_new_frame(cls: Any, old_point: Point, new_frame: Frame) -> Point: ...
    @classmethod
    def at_origin(cls: Any, frame: Frame) -> Point: ...
    @property
    def frame(self): ...
    def __eq__(self, other: Any) -> bool: ...

EDIT:

To be extra clear (and with fancy diagrams!) I am looking to compare the public interfaces between 2 Packages, A and B. The best way to generate the public interface with type information is to use stubgen. So, The public interface for packages will be defined by a package of .pyi files.

Then we use a new magic tool (called something snappy like, interface-diff) to compare both of the interfaces for A and B. At the highest level we will then have 4 potential outcomes when comparing the interfaces.

1) MAJOR - Things removed from A AND NOT things added to B

2) MAJOR - Things removed from A AND things added to B

3) MINOR - Things added to B AND NOT removed from A

4) PATCH - A == B

Happy to write this all myself if someone could put me onto the right lines :)

Answer 1

What you need are abstract syntax trees (AST's) of each stub. No new nodes means patch, new leaf nodes probably means minor (depends on language and location of node) and new non-leaf nodes means major. But that's not the entire solution really, because not all breaking changes show up at the interface stubs. You also need to have functional verification that any new implementation of the existing interfaces, actually satisfy the documented behaviors (contracts) and that use of any new interfaces doesn't break the old interfaces.

You should be able to find good parser/AST implementations among the publicly available Python implementations. You can use AST's of the old and new implementations to find potential breaking changes behind existing interfaces!