RavenDB OrderByDescending and Take - Incorrect Results

Question

This query was working for me until recently. I now have 135 InstallationSummary documents in my RavenDB. Instead of getting the most recent by start time, it's mostly working, but the last couple, most recent documents aren't showing up from this query. Am I querying incorrectly? Is there a different way to do OrderByDescending and Take with RavenDB that I should be aware of? Is there a document number limit to what I can query correctly?

Note: I have debugged this, and the query indeed returns what we see in the grid. There is no transformation done between the time the query is run and what is shown in the UI.

IEnumerable<InstallationSummary> installationSummaries =
  QueryAndCacheEtags(session => session.Advanced.LuceneQuery<InstallationSummary>()
  .Include(x => x.ApplicationServerId)
  .Include(x => x.ApplicationWithOverrideVariableGroup.ApplicationId)
  .Include(x => x.ApplicationWithOverrideVariableGroup.CustomVariableGroupId)
  .OrderByDescending(summary => summary.InstallationStart)
  .Take(numberToRetrieve)).Cast<InstallationSummary>().ToList();

This grid should show a few more rows in it with start times greater than 1/19/2012 6:33:51 PM:

Edit: I removed Take(numberToRetrieve) from the query, and I'm only getting 128 of the total 160 InstallationSummary documents. I can see all 160 in RavenDB Studio, but only 128 return from the query. 128... 128... power of 2... Did I hit some limit?

Okay, it looks like I did hit the limit of 128: http://www.blogcoward.com/archive/2010/05/21/RavenDB-and-a-brief-design-philosophy-discussion-with-Ayende.aspx http://codeofrob.com/archive/2010/05/12/ravendb-basic-usage-considerations.aspx

But why? I have a Take() method in there. How am I supposed to get the 50 most recent documents?

As a bit of a hack, the query below will at least show the most recent. It isn't exactly what I want, because I want the most recent 50, regardless of date. As long as there aren't more than 50 since the start date, this will at least show the most recent items.

using Raven.Client.Linq;

DateTime startDate = new DateTime(2012, 1, 18);

IEnumerable<InstallationSummary> installationSummaries =
QueryAndCacheEtags(session => session.Query<InstallationSummary>()
.Include(x => x.ApplicationServerId)
.Include(x => x.ApplicationWithOverrideVariableGroup.ApplicationId)
.Include(x => x.ApplicationWithOverrideVariableGroup.CustomVariableGroupId)                        
.Where(x => x.InstallationStart > startDate)
.OrderByDescending(summary => summary.InstallationStart)                        
.Take(numberToRetrieve)
).Cast<InstallationSummary>().ToList();

I had to go from a LuceneQuery to just Query and I had to add the Where clause.

Answer 1

Finally worked out the real issue.

IEnumerable<InstallationSummary> installationSummaries =
    QueryAndCacheEtags(session => session.Query<InstallationSummary>()
       .Include(x => x.ApplicationServerId)
       .Include(x => x.ApplicationWithOverrideVariableGroup.ApplicationId)
       .Include(x => x.ApplicationWithOverrideVariableGroup.CustomVariableGroupId)                        
       .Where(x => x.InstallationStart > startDate)
       .OrderByDescending(summary => summary.InstallationStart)                        
       .Take(numberToRetrieve))
       .Cast<InstallationSummary>()
       .ToList();

The signature of the QueryAndCacheEtags(..) function is Func<T>, not Expression<Func<T>>. And it returns IEnumerable<T> not IQueryable<T>

This converts the statement from IQueryable<T> to IEnumerable<T> at that point. This means that the RavenDB server only processes the first part of the query, which has no filtering or ordering.

The rest of the statements and then applied in-memory to the 128 items you get back. Hence why you aren't seeing the item ordered or filtered properly.

There's a bit more info here and here

Generally you don't have to worry about the difference between Func<T> and Expression<Func<T>>, the compiler handles it for you. But if you introduce your own function call into a LINQ statement, you do need to get it right.

Answer 2

RavenDB uses eventual consistency by default, so indexes can be stale unless you explicitly specify otherwise.

Add the line below (or one of it's variants) to your query to this:

  .Customize(x => x.WaitForNonStaleResultsAsOfNow())