(Rails) Some users getting paid twice when withdrawing money from their balance to bank account. Can't reproduce

Question

In my app a worker can build up a credit balance in their account and withdraw it to their bank account as they see fit. There is a MoneyController::withdraw action that calls a .withdraw_funds method on the current_worker which makes a call to the Balanced Payments API and credits their bank account if the amount they've requested to withdraw is <= the amount in their balance. a Transaction is created, attached to the worker's Account, that lists the amount that was subtracted from their balance and credited to their bank account.

Recently something has been happening where this controller action gets hit and the whole process is taking place twice, even though the request is supposed to be rejected if the balance is empty. The withdrawal happens twice and two transactions are generated with the same or very close timestamp. However, it only happens for some workers, and I can't reproduce the error on the development or staging servers. I was hoping someone could give me some advice as to how to proceed with debugging this. Here is the relevant code:

MoneyController
  def withdraw
    if current_worker.withdraw_funds((params[:amount].to_d*100).to_i, params[:bbank])
      redirect_to worker_money_path, notice: "Successfully withdrew $#{params[:amount]}"
    else
      redirect_to worker_money_path, alert: "Failed to withdraw funds. Please contact us for assistance."
    end
  end

/

worker.rb

  def withdraw_funds(amount, bbank_id)
    bcust = self.get_balanced
    bbank = Balanced::BankAccount.fetch("/bank_accounts/#{bbank_id}")
    if bbank and (bbank.customer.id == bcust.id)
      puts "bank belongs to worker"
      if self.account.balance >= amount
        res = bbank.credit(amount: amount)
        self.account.debit(amount)
        Transaction.create(amount: amount, tag: 'cashout', source_id: self.account.id, destination_id: nil, balanced_id: res.id)
        return true
      else
        puts "worker #{self} doesn't have #{amount} in account"
        return false
      end
    else
      puts "bank does not belong to worker"
      return false
    end
  end

If the worker's balance contains $50 and two requests are made for $50, the first should succeed and then the second should fail because the balance is now $0 (hence if self.account.balance >= amount).

I was able to look through the logs of the development server as well and find the logs for when this happened:

Started POST "/money/withdraw" for 68.119.221.188 at 2014-11-05 13:56:41 -0500
Processing by MoneyController#withdraw as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"x5olIpvJf2K37lYRJypIIHYNhAdZUm1ptill13w9Evw=", "amount"=>"48.50", "bbank"=>"BA2...", "commit"=>"Withdraw"}
Started POST "/money/withdraw" for xx.xxx.xxx.xxx at 2014-11-05 13:56:46 -0500
Processing by MoneyController#withdraw as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"x5olIpvJf2K37lYRJypIIHYNhAdZUm1ptill13w9Evw=", "amount"=>"48.50", "bbank"=>"BA2...", "commit"=>"Withdraw"}
Redirected to http://myapp.com/money
Completed 302 Found in 4467.9ms (ActiveRecord: 54.7ms)
Started GET "/worker/money" for xx.xxx.xxx.xxx at 2014-11-05 13:56:50 -0500
Processing by Worker::MoneyController#index as HTML
Redirected to http://myapp.com/money
Completed 302 Found in 9099.1ms (ActiveRecord: 69.6ms)

I notice in the logs that both requests have the same authenticity token but am not sure what else to take away from them. I thought it could be as simple as the worker clicking the "Withdraw" button multiple times but in my attempts to recreate the issue in development and staging that never caused the problem. The requests were just queued and the succeeding requests always caused the proper response that the balance was empty.

EDIT I set up a test scenario on the production server and was able to reproduce the problem by clicking the Withdraw button multiple times. Anyone have any idea as to why this would happen in Production but not in Development or Staging? Could connection speed have anything to do with it?

Answer 1

Since you are not posting client code, I could suggest you put some kind of lock to Worker so as he can only have one running transaction at a time... something like in_process:boolean and process_start:timestamp (both combined).

So, when you start a transaction, you make sure that in_process is false and set it to true, also modifying the process_start (put there for accidental locks, for example, assume a lock active if in_process && process_start>(Time.now - 10.minutes).

After the process is finished, you set the in_process flag back to false.

This way, only one process per user can be active.

Of course, if we have some html, this will probably not be necessary, but it's good practice to have business (controller) logic behind processes that involve money anyhow.

EDIT: Perhaps it would also be good idea to keep the Worker locked for some seconds after the transaction has ended.