[FIXED] ValueError: DataFrame constructor not properly called error on trying to call the data from pipeline

Issue

I was trying to visualise iris dataset before and after using StandardScaler Using seaborn But i got an error on trying to define the DataFrame

I found a lot of similar questions but none of them explained how to convert pipeline data to DataFrame

        X = DATA.drop(['class'], axis = 'columns')
        y = DATA['class'].values
        
        X_train, X_test, y_train, y_test=train_test_split(X,y, test_size=0.20,random_state =42)

        gbl_pl=[]
        gbl_pl.append(('standard_scaler_gb',
        StandardScaler()))
        print(gbl_pl)           
        gblpq=Pipeline((gbl_pl))
        scaled_df=gblpq.fit(X_train,y_train)
        print(scaled_df.named_steps['standard_scaler_gb'].mean_)
      
        scaled_df =pd.DataFrame(scaled_df,
        columns=['petal_length', 'petal_width',
        'sepal_length','sepal_width'])        
        
        fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(6, 5))
        ax1.set_title('Before Scaling')
        sns.kdeplot(X['petal_length'], ax=ax1)
        sns.kdeplot(X['petal_width'], ax=ax1)
        sns.kdeplot(X['sepal_length'], ax=ax1)
        sns.kdeplot(X['sepal_width'], ax=ax1)
                
        ax2.set_title('After Standard Scaler')
        sns.kdeplot(scaled_df['petal_length'], ax=ax2)
        sns.kdeplot(scaled_df['petal_width'], ax=ax2)
        sns.kdeplot(scaled_df['sepal_length'], ax=ax2)
        sns.kdeplot(scaled_df['sepal_width'], ax=ax2)
        plt.savefig("output73.png")

Error

    columns=['petal_length', 'petal_width',
   File "/data/user/0/ru.iiec.pydroid3/files/aarch64-linux-android/lib/python3.8/site-packages/pandas/core/frame.py", line 509, in __init__
     raise ValueError("DataFrame constructor not properly called!")
 ValueError: DataFrame constructor not properly called!

Solution

You are conflating the transformers (StandardScaler and Pipeline) with data. The return value of your gblpq.fit(X_train y_train) is a pipeline object, not data. To get the scaled data, you can use the pipeline's transform method (or the convenience combination fit_transform).

    ...
    scaled_df = gblpq.fit_transform(X_train, y_train)
    # gblpq is now fitted and contains learned statistics like the means:
    print(gblpq.named_steps['standard_scaler_gb'].mean_)
    # while scaled_df contains the transformed data as a numpy array
    
    scaled_df = pd.DataFrame(
        scaled_df,
        columns = [petal_length', petal_width', 'sepal_length', 'sepal_width']
    )
    ...

Answered By - Ben Reiniger