What is the maximum throughput of Loggly?

Question

How many requests per second from a client can Loggly handle? I am only able to get around 10–20 requests processed per second and I am wondering if this is normal.

Answer 1

I just ran a bunch of tests and found that it can't really handle much via a tcp connection using syslog-ng.

Here are my test results for anyone wanting to try it. I used balabit's "loggen" program for this and sent 200 byte messages to the tcp port assigned to me by loggly. Note that although the syslog RFC (3164 at least) states that a log message should not exceed 1024 bytes, I used 200 byte packets just to be fair and because many messages are that small.

Signed up for a free account. Configured a TCP connection for testing. Tried sending various amounts, results:

Test 1: FAIL

loggen -iS -r 6000  -s 200 -I 100 logs.loggly.com 16225
Send error Broken pipe, results may be skewed.      
average rate = 1392.13 msg/sec, count=18296, time=13.142, (average) msg size=200, bandwidth=271.74 kB/sec

Test 2: FAIL

loggen -iS -r 4000  -s 200 -I 100 logs.loggly.com 16225 
Send error Broken pipe, results may be skewed.       
average rate = 2767.16 msg/sec, count=121146, time=43.779, (average) msg size=200, bandwidth=540.15 kB/sec

Test 3: FAIL

loggen -iS -r 2500  -s 200 -I 100 logs.loggly.com 16225  
Send error Broken pipe, results may be skewed.      
average rate = 1931.27 msg/sec, count=85878, time=44.467, (average) msg size=200, bandwidth=376.98 kB/sec

Test 4: FAIL

loggen -iS -r 2000  -s 200 -I 100 logs.loggly.com 16225    
Send error Broken pipe, results may be skewed.      
average rate = 1617.72 msg/sec, count=83134, time=51.389, (average) msg size=200, bandwidth=315.78 kB/sec

Test 5: FAIL

loggen -iS -r 1000  -s 200 -I 100 logs.loggly.com 16225 
Send error Broken pipe, results may be skewed.      
average rate = 936.50 msg/sec, count=63331, time=67.624, (average) msg size=200, bandwidth=182.81 kB/sec

Test 6: PASS for duration configured, FAIL for > 100 seconds - SEE TEST 7

loggen -iS -r 500  -s 200 -I 100 logs.loggly.com 16225  
average rate = 325.00 msg/sec, count=32501, time=100.001, (average) msg size=200, bandwidth=63.44 kB/sec

Test 7: FAIL - Ran a new test @500 EPS for a longer period and the pipe broke after 255 seconds:

loggen -iS -r 500  -s 200 -I 10000 logs.loggly.com 16225
Send error Broken pipe, results may be skewed.     
average rate = 323.35 msg/sec, count=82642, time=255.577, (average) msg size=200, bandwidth=63.12 kB/sec

Test 8: FAIL (ran for longer @ 200 EPS, but still failed)

loggen -iS -r 200  -s 200 -I 10000 logs.loggly.com 16225 
Send error Broken pipe, results may be skewed.      
average rate = 163.53 msg/sec, count=234090, time=1431.470, (average) msg size=200, bandwidth=31.92 kB/sec

Test 9: FAIL (again, ran longer but still failed)

loggen -iS -r 50  -s 200 -I 10000 logs.loggly.com 16225    
Send error Broken pipe, results may be skewed.    
average rate = 47.36 msg/sec, count=89325, time=1886.014, (average) msg size=200, bandwidth=9.25 kB/sec

Test 10: FAIL? (same results, but lost the connection again. Hard to believe they can’t handle 10 eps?)

loggen -iS -r 10  -s 200 -I 10000 logs.loggly.com 16225 
Send error Broken pipe, results may be skewed.  
average rate = 9.94 msg/sec, count=1568, time=157.770, (average) msg size=200, bandwidth=1.94 kB/sec

Did some web searching to see what loggly can actually do, but there’s only marketing material that says it is scalable, not how scalable it is. I did find this: http://twitter.com/jordansissel/status/5948244626509824 Which is only 22 events per second…

Full Disclosure: I am the founder of LogZilla, so I was testing out the competition because we are launching a cloud-based syslog solution. My tests show that our software is able to handle anywhere from 2,000 to 12,000 events per second depending on which servers we're using in the cloud.

Answer 2

I really don't know but I've been searching for a logging solution for node.js as well without luck.

Why?

Because all of those that I've checked (didn't check all) are using synchronous disk writing! ...... which AWFULLY degrades performance.

So if you ask me - you should re-consider your needs, and log only stuff you really need.

Answer 3

I ran tests similar to the ones in Clayton answer as his results made me worried that Loggly would drop messages if I sent too many at the same time. I wanted to see if the problems Clayton encountered in 2012 still existed today.

That said, here is what I found running loggen for 60 seconds generating 100,000 messages a second.

$ loggen -iS -r 100000 -s 200 -I 60 logs-01.loggly.com port
average rate = 34885.98 msg/sec, count=2093163, time=60.000, (average) msg size=200, bandwidth=6809.74 kB/sec

I was also curious what some competitors would return for similar tests and I found the following:

Papertrail

loggen -iS -D -r 100000 -s 200 -I 60 logs2.papertrailapp.com PORT
average rate = 24344.71 msg/sec, count=1461327, time=60.026, (average) msg size=200, bandwidth=4752.09 kB/sec

Logentries

$ loggen -iS -D -r 100000 -s 200 -I 60 api.logentries.com PORT
average rate = 14076.76 msg/sec, count=844609, time=60.000, (average) msg size=200, bandwidth=2747.78 kB/sec

Obviously these are not hard numbers that will always be the same as systems change over time. This just gives us a point in time reference of how they responded when I ran the tests. Your mileage will vary!

Update: I ran a longer (nearly 3 hour) test against Loggly and received the following:

loggen -iS -r 100000 -s 200 -I 10000 logs-01.loggly.com port 
average rate = 15869.22 msg/sec, count=158692177, time=10000.000, (average) msg size=200, bandwidth=3097.67 kB/sec