How can Rubinius and JRuby possibly be this slow?

Question

I decided to see how long it would take to iterate through an array of hashes. Below is the code:

pairs = [{name: "firstname", value: "string"},{name: "lastname", value: "string"},{name: "country", value: "string"},{name: "city", value: "string"},{name: "state", value: "string"},{name: "company", value: "string"},{name: "year", value: "string"},{name: "political_affiliation", value: "string"},{name: "social_security_number", value: "string"}] * 1000
blank = {}

start = Time.now
pairs.each do |pair|
  blank[pair[:name]] = pair[:value]
end

p Time.now - start

Time is calculated by subtracting the current time after the loop from the current time before the loop.

This was the amount of time the computation took in YARV 2.1.1, according to the math in the code:

0.001962017

Here's how long it took in Rubinius 2.2.6:

0.022598

And jRuby 1.7.12

0.022317

Supposedly Rubinius and jRuby have performance advantages over YARV. Why do they take almost 12 times the amount of time to perform the same basic operation? Is this normal or do I have something improperly configured?

Answer 1

You are benchmarking too tiny times, which are compromised by enviroment loading. In my experience, in order to have reliable benchmarks you have to get timings of at least 10 seconds, in order to mitigate warmup times. Around 10 seconds I expect JRuby to be the most performant, followed by Ruby and Rubinius.

Let's see:

# so_24049371.rb

require 'benchmark'

# NOTE THIS: YOU SHOULD TWEAK IT IN ORDER TO HAVE BENCHMARKS OF ~ 10 SECONDS
MULTIPLIER = 5_000_000
pairs = [{name: "firstname", value: "string"},{name: "lastname", value: "string"},{name: "country", value: "string"},{name: "city", value: "string"},{name: "state", value: "string"},{name: "company", value: "string"},{name: "year", value: "string"},{name: "political_affiliation", value: "string"},{name: "social_security_number", value: "string"}] \
  * MULTIPLIER
blank = {}

puts Benchmark.measure {
  pairs.each do |pair|
    blank[pair[:name]] = pair[:value]
  end
}

---

# so_24049371.fish

source (rbenv init -|psub)

for ruby_version in 2.1.2 rbx-2.2.7 jruby-1.7.12
  rbenv shell $ruby_version
  ruby -v
  ruby so_24049371.rb
end

This is the output on my machine (I use fish shell + rbenv, you should write your own script):

> fish so_24049371.fish
ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-linux]
  8.190000   0.000000   8.190000 (  8.188726)
rubinius 2.2.7 (2.1.0 build 2014-05-20 JI) [x86_64-linux-gnu]
 14.359762   0.003525  14.363287 ( 14.193565)
jruby 1.7.12 (2.0.0p195) 2014-04-15 643e292 on Java HotSpot(TM) 64-Bit Server VM 1.7.0_55-b13 [linux-amd64]
  4.570000   0.000000   4.570000 (  4.367000)

As I supposed, JRuby is the fastest with 4.367000, than Ruby with 8.188726 and last Rubinius with 14.193565.