Re-use pymethods for multiple structs

97 views Asked by At

Some context first of all. I would provide an example but this crate doesn't actually exist: what I'm working with is much, much more complicated and what I describe below is a hypothetical way to de-complicate it. Currently my actual crate has a proc macro that gets called on each struct, that generates a #[pymethods] impl block (the proc macro can accept a tokenstream to add more methods for individual structs as needed), so technically there is only one #[pymethods] impl block. This makes debugging pretty hard, as any error in the proc macro shows up in the various files that the macro is called from, with no information on where it actually comes from. I have tried to use the multiple-pymethods feature of PyO3, but can't get that my code to compile with that that breaks cargo test and I couldn't get that to work for the life of me.

The question:

Say I have a Python extension crate in Rust that has multiple structs, all with serde serialization and deserialization functions that are written exactly the same way, e.g. .to_yaml(), .from_yaml(), etc. These are exposed to Python via a #[pymethods] impl block.

As the code for all these methods are exactly the same, I want to be able to define them in one single spot, and apply them to all the structs. If there was no single #[pymethods] requirement, I would simply create a trait and impl it for all my structs. Is it doable via extending the #[pymethods] impl block of each struct with the methods with a macro or something? Any other ideas?

EDIT: More detail on the issues with multiple-pymethods:

Adding the multiple-pymethods feature causes a linking error on macOS, which is fixed by adding a .cargo/config.toml at the crate root, following these instructions: https://pyo3.rs/main/building_and_distribution#macos

Now I cargo test returns the following error: dyld[9884]: symbol not found in flat namespace (_PyBaseObject_Type)

EDIT: I've added an example for the hypothetical code:

Cargo.toml:

[package]
name = "pyo3-testing"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[lib]
name = "pyo3_testing"
crate-type = ["cdylib"]

[dependencies]
anyhow = "1.0.75"
pyo3 = { version = "0.19.0", features = ["anyhow", "extension-module"] }
# pyo3 = { version = "0.19.0", features = ["anyhow", "extension-module", "multiple-pymethods"] }
serde = { version = "1.0.189", features = ["derive"] }
serde_yaml = "0.9.25"

lib.rs:

use anyhow;
use pyo3::prelude::*;
use serde::{Deserialize, Serialize};
use serde_yaml;

/// A Python module implemented in Rust.
#[pymodule]
fn pyo3_testing(_py: Python, m: &PyModule) -> PyResult<()> {
    // m.add_function(wrap_pyfunction!(sum_as_string, m)?)?;
    m.add_class::<Point2D>()?;
    Ok(())
}

#[pyclass]
#[derive(Debug, Serialize, Deserialize, PartialEq)]
pub struct Point2D {
    pub x: f64,
    pub y: f64,
}

#[pyclass]
#[derive(Debug, Serialize, Deserialize, PartialEq)]
pub struct Point3D {
    pub x: f64,
    pub y: f64,
    pub z: f64,
}

#[pymethods]
impl Point2D {
    // This method is unique to Point2D
    #[new]
    fn new(x: f64, y: f64) -> Self {
        Self { x, y }
    }

    // DUPLICATE METHOD
    fn to_yaml(&self) -> anyhow::Result<String> {
        Ok(serde_yaml::to_string(self)?)
    }

    // DUPLICATE METHOD
    #[staticmethod]
    fn from_yaml(yaml: &str) -> anyhow::Result<Self> {
        Ok(serde_yaml::from_str(yaml)?)
    }
}

#[pymethods]
impl Point3D {
    // This method is unique to Point3D
    #[new]
    fn new(x: f64, y: f64, z: f64) -> Self {
        Self { x, y, z }
    }

    // DUPLICATE METHOD
    fn to_yaml(&self) -> anyhow::Result<String> {
        Ok(serde_yaml::to_string(self)?)
    }

    // DUPLICATE METHOD
    #[staticmethod]
    fn from_yaml(yaml: &str) -> anyhow::Result<Self> {
        Ok(serde_yaml::from_str(yaml)?)
    }
}

#[cfg(test)]
mod point2d_tests {
    use super::*;

    #[test]
    fn test_to_yaml() -> anyhow::Result<()> {
        let expected_yaml = "x: 1.0\ny: 2.0\n";

        let point = Point2D::new(1.0, 2.0);
        let actual_yaml = point.to_yaml()?;

        anyhow::ensure!(
            actual_yaml == expected_yaml,
            "{actual_yaml:?} != {expected_yaml:?}",
        );
        Ok(())
    }

    #[test]
    fn test_from_yaml() -> anyhow::Result<()> {
        let expected_point = Point2D::new(1.0, 2.0);

        let yaml = "x: 1.0\ny: 2.0\n";
        let actual_point = Point2D::from_yaml(yaml)?;

        anyhow::ensure!(
            actual_point == expected_point,
            "{actual_point:?} != {expected_point:?}",
        );
        Ok(())
    }
}

#[cfg(test)]
mod point3d_tests {
    use super::*;

    #[test]
    fn test_to_yaml() -> anyhow::Result<()> {
        let expected_yaml = "x: 1.0\ny: 2.0\nz: 3.0\n";

        let point = Point3D::new(1.0, 2.0, 3.0);
        let actual_yaml = point.to_yaml()?;

        anyhow::ensure!(
            actual_yaml == expected_yaml,
            "{actual_yaml:?} != {expected_yaml:?}",
        );
        Ok(())
    }

    #[test]
    fn test_from_yaml() -> anyhow::Result<()> {
        let expected_point = Point3D::new(1.0, 2.0, 3.0);

        let yaml = "x: 1.0\ny: 2.0\nz: 3.0\n";
        let actual_point = Point3D::from_yaml(yaml)?;

        anyhow::ensure!(
            actual_point == expected_point,
            "{actual_point:?} != {expected_point:?}",
        );
        Ok(())
    }
}

.cargo/config.toml:

[target.x86_64-apple-darwin]
rustflags = [
  "-C", "link-arg=-undefined",
  "-C", "link-arg=dynamic_lookup",
]

[target.aarch64-apple-darwin]
rustflags = [
  "-C", "link-arg=-undefined",
  "-C", "link-arg=dynamic_lookup",
]
0

There are 0 answers