Hyper-tuning of SVM regressor using Grid search and cuML

Question

I tried tuning the SVM regressor parameters using the code below. However, during the search for the best params, the grid-search model tends to choose the first kernel of the model within the proposed kernels, every time. I tried different combinations to see if I could reach good results.

I attempted to use various cross-validations(3, 5-default, 7).

I used/removed the score function of the GridSearch model, to see if this impact.

I also tried with higher values of the penalty score (see the attached).

Please can anyone confirm this?
And is the kernels' order in the list/Grid params impact the search?

Note: I used the cuML to benefit from using the GPU and to speed up the search. I can send you the dataset if you want

from cuml.svm import SVR
from sklearn.model_selection import GridSearchCV
#from sklearn.svm import SVR
from sklearn.metrics import make_scorer, mean_squared_error
from pprint import pprint


# Hyper-tunning for SVM regressor

import numpy as np

base_svr = SVR()
# If you want to use your custom score function, specify the function and use it. 
def my_rmse_loss_func(y_true, y_pred):
    return np.sqrt(mean_squared_error((y_true, y_pred)))
     

scorer = make_scorer(mean_squared_error, greater_is_better=False)#,, squared = False



param_grid_svr = {'C': [0.0001, 0.01, 0.1, 0.9,1, 1.1, 2,3], # I used 10, 100 
              'gamma': ['scale', 'auto'], 
              'kernel': ['rbf', 'poly', 'sigmoid', 'precomputed','linear'], #, 
              'epsilon': [0.01, 0.1, 0.2 ,0.22, 0.3, 1]}
# Create a GridSearchCV object and fit it to the training data with 7 cv

svr_cv = GridSearchCV(base_svr,param_grid_svr, n_jobs = -1 , # Use 4
                         scoring='neg_mean_squared_error', cv = 7 ,verbose=3 ,return_train_score =True  ) # ,refit=True mean_squared_error, scoring='neg_mean_squared_error' ,



# Train the model in GPU

svr_cv.fit(X_train,y_train)

# Predictions
## By using the re-trained model
predictions_RF_regr_tuned= svr_cu.predict(X_test)