Why is REE faster than 1.9.3 at executing RSpec?

Question

I've always thought that 1.9.x was supposed to be faster than REE, but consuming more memory. But as it was recently pointed out to me, REE is actually faster than executing RSpec specs, and by a huge margin.

$ rvm use ree
$ rvm gemset create sandbox
$ rvm gemset use sandbox
$ gem install rspec

$ time rspec foo_spec.rb 
No examples found.

Finished in 0.07346 seconds
0 examples, 0 failures

real    0m0.104s
user    0m0.059s
sys 0m0.015s

and after doing the same thing with 1.9.3, I'm getting

$ time rspec foo_spec.rb 
No examples found.

Finished in 0.13922 seconds
0 examples, 0 failures

real    0m0.208s
user    0m0.122s
sys 0m0.022s

That's twice as much with an empty gemset, containing only rspec and executed on a empty spec file. I'm seeing even larger differences on gemsets containing multiple gems.

Why is this happening, isn't 1.9.3 supposed to be the fastest version currently available?

I'm running the latest versions installed via RVM on OS X Lion.

Answer 1

It depends on what you bench against really. If you do something as simple as that then it doesn't even count in my opinion. So, here's an example of my Backup gem which does a lot of file "requires" and runs a bunch of real world specs.

Ruby Enterprise Edition

time bundle exec rspec spec

real    0m5.579s
user    0m3.427s
sys 0m0.465s

502 examples, 0 failures

Ruby 1.9.3p0

bundle exec rspec spec

real    0m3.863s
user    0m3.552s
sys 0m0.299s

502 examples, 0 failures

The start-up time is generally faster with 1.9.3p0 since a certain patch made it in that changes the algorithm of the file require function, which on it's own I believe reduced the average load time of apps similar in size of average Ruby on Rails applications by about 30%. But, if there are barely any files initially loaded, and the process doesn't run for long, then it might be slower.

Answer 2

I would guess that you're seeing the results of GC performance tuning which are really still in REE's favour (based on real-world benchmarking we've done with a large Rails app under both REE and 1.9.3).

To eliminate GC overhead from your benchmarking, so you're really comparing apples to apples, disable it as early as possible in the process. You can do that as follows:

GC.disable

Of course, you'll only get useful results if you have sufficient memory to run all the tests without invoking GC; if your machine starts swapping then the results won't be useful.