make random forest with fitness function and genetic algorithm

Question

I would like to set up a random forest for my fitness function using a genetic algorithm, like they do with weights in neural networks... But it's quite difficult for me to figure out what exactly I need to change in the rf model

set.seed(123)
X <- matrix(rnorm(1000),ncol = 5)
Y <- rnorm(nrow(X))
library(randomForest)
rf <- randomForest(Y~.,X,ntree=50)

I guess I need to change something in rf$forest$... but what exactly I am not very clear about. Can someone explain to me what exactly I need to change in order to change the model and adjust it for my fitness function