Using tft.scale_to_gaussian for preprocessing a dataset without using other tensorflow operations

12 views Asked by At

I'm working on a project where I have a set of longtail data that I want to transform into a Gaussian distribution. I'm looking to achieve something similar to scikit-learn's PowerTransformer, but using TensorFlow Transform (TFT) instead. I've come across various resources mentioning the use of Beam Pipelines and other complex setups, but I'm looking for a simpler solution that just applies this specific transformer and converts the result back into a NumPy array.

Here's one simplified version of the code I've been working with:

import numpy as np
import tensorflow as tf
import tensorflow_transform as tft

tf.config.run_functions_eagerly(False)

def generate_longtail_data(size, shape, scale=1.0):
    data = (np.random.pareto(shape, size) + 1) * scale
    return data

# Generate longtail data
longtail_data = generate_longtail_data(1000, 5)

# Convert data to TensorFlow Tensors
data_tensor = tf.constant(longtail_data, dtype=tf.float32)

# Scale data to a Gaussian distribution
scaled_data = tft.scale_to_z_score(data_tensor)

# Execute TensorFlow operations to get the scaled data array
with tf.compat.v1.Session() as sess:
    scaled_data_np = sess.run(scaled_data)

print("Scaled Longtail Data:")
print(scaled_data_np)

However, I'm running into errors related to tf.placeholder() and compatibility with Eager Execution.

My questions are:

How can I correctly apply tft.scale_to_gaussian (or a similar function) to transform longtail data into a Gaussian distribution without setting up a full Beam Pipeline? How can I efficiently convert the result back into a NumPy array, especially considering TensorFlow 2.x's Eager Execution? Additionally, while my boss prefers using the TFT function, I'm also open to any alternative methods that can achieve a similar transformation if there's a simpler or more direct approach available. I've found very few Google entries when searching for this function, so any guidance or examples would be greatly appreciated.

0

There are 0 answers