Running the Bayesian optimizer, and when the maximize function is executed, "one of the lower bounds is greater than an upper bound." Error occurs

Question

We are running the Bayesian Optimizer for hyper parameter tuning. By the way, I get this error. The same error occurs even if you experiment with changing all of the parameter ranges. Please answer what should be done.

def XGB_cv(max_depth,learning_rate, n_estimators, gamma
           ,min_child_weight, max_delta_step, subsample
       ,colsample_bytree, silent=True, nthread=-1):
model = xgb.XGBClassifier(max_depth=int(max_depth),
                          learning_rate=learning_rate,
                          n_estimators=int(n_estimators),
                          silent=silent,
                          nthread=nthread,
                          gamma=gamma,
                          min_child_weight=min_child_weight,
                          max_delta_step=max_delta_step,
                          subsample=subsample,
                          colsample_bytree=colsample_bytree)
RMSE = cross_val_score(model, train2, y, scoring='accuracy', cv=5).mean()
return RMSE

pbounds = {'max_depth': (5, 10),
      'learning_rate': (0, 0.5),
      'n_estimators': (50, 1000),
      'gamma': (1, 0.01),
      'min_child_weight': (0,10),
      'max_delta_step': (0, 0.1),
      'subsample': (0, 0.8),
      'colsample_bytree' :(0, 0.99),
      }

xgboostBO = BayesianOptimization(f = XGB_cv, pbounds = pbounds, verbose = 2, random_state = 1 )
xgboostBO.maximize(init_points=2, n_iter = 10, acq='ei', xi=0.01)


~\Anaconda3\lib\site-packages\scipy\optimize\lbfgsb.py in _minimize_lbfgsb(fun, x0, args, jac, bounds, disp, maxcor, ftol, gtol, eps, maxfun, maxiter, iprint, callback, maxls, finite_diff_rel_step, **unknown_options)
292     # check bounds
293     if (new_bounds[0] > new_bounds[1]).any():
--> 294         raise ValueError("LBFGSB - one of the lower bounds is greater than an upper bound.")
295 
296     # initial vector must lie within the bounds. Otherwise ScalarFunction and
ValueError: LBFGSB - one of the lower bounds is greater than an upper bound.

Answer 1

I know nothing of this Bayesian stuff, but in box bounded optimization it is a no-no to provide lower bounds greater than upper bounds:

‘gamma': (1, 0.01),

Not sure if this is your issue but it took me all of 7 seconds to see it.