Reading a large batch of objects taking way too long

Question

I’m experimenting with scripting a batch of OmniFocus tasks in JXA and running into some big speed issues. I don't think the problem is specific to OmniFocus or JXA; rather I think this is a more general misunderstanding of how getting objects works - I'm expecting it to work like a single SQL query that loads all objects in memory but instead it seems to do each operation on demand.

Here’s a simple example - let’s get the names of all uncompleted tasks (which are stored in a SQLite DB on the backend):

var tasks = Application('OmniFocus').defaultDocument.flattenedTasks.whose({completed: false})
var totalTasks = tasks.length
for (var i = 0; i < totalTasks; i++) {
    tasks[i].name()
}

[Finished in 46.68s]

Actually getting the list of 900 tasks takes ~7 seconds - already slow - but then looping and reading basic properties takes another 40 seconds, presumably because it's hitting the DB for each one. (Also, tasks doesn't behave like an array - it seems to be recomputed every time it's accessed.)

Is there any way to do this quickly - to read a batch of objects and all their properties into memory at once?

Answer 1

Try something like:

tell app "OmniFocus"
  tell default document
    get name of every flattened task whose completed is false
  end tell
end tell

Apple event IPC is not OOP, it’s RPC + simple first-class relational queries. AppleScript obfuscates this, and JXA not only obfuscates it even worse but cripples it too; but once you learn to see through the faux-OO syntactic nonsense it makes a lot more sense. This and this may give a bit more insight.

[ETA: Omni recently implemented its own embedded JavaScriptCore-based scripting support in its apps; if JS is your thing you might find that a better bet.]

Answer 2

Introduction

With AppleEvents, the IPC technology that JavaScript for Automation (JXA) is built upon, the way you request information from another application is by sending it an "object specifier," which works a little bit like dot notation for accessing object properties, and a little bit like a SQL or GraphQL query.

The receiving application evaluates the object specifier and determines which objects, if any, it refers to. It then returns a value representing the referred-to objects. The returned value may be a list of values, if the referred-to object was a collection of objects. The object specifier may also refer to properties of objects. The values returned may be strings, or numbers, or even new object specifiers.

Object specifiers

An example of a fully-qualified object specifier written in AppleScript is:

a reference to the name of the first window of application "Safari"

In JXA, that same object specifier would be expressed:

Application("Safari").windows[0].name

To send an IPC request to Safari to ask it to evaluate this object specifier and respond with a value, you can invoke the .get() function on an object specifier:

Application("Safari").windows[0].name.get()

As a shorthand for the .get() function, you can invoke the object specifier directly:

Application("Safari").windows[0].name()

A single request is sent to Safari, and a single value (a string in this case) is returned.

In this way, object specifiers work a little bit like dot notation for accessing object properties. But object specifiers are much more powerful than that.

Collections

You can effectively perform maps or comprehensions over collections. In AppleScript this looks like:

get the name of every window of Application "Safari"

In JXA it looks like:

Application("Safari").windows.name.get()

Even though this requests multiple values, it requires only a single request to be sent to Safari, which then iterates over its own windows, collecting the name of each one, and then sends back a single list value containing all of the name strings. No matter how many windows Safari has open, this statement only results in a single request/response.

For-loop anti-pattern

Contrast that approach to the for-loop anti-pattern:

var nameOfEveryWindow = []
var everyWindowSpecifier = Application("Safari").windows
var numberOfWindows = everyWindowSpecifier.length
for (var i = 0; i < numberOfWindows; i++) {
    var windowNameSpecifier = everyWindowSpecifier[i].name
    var windowName = windowNameSpecifier.get()
    nameOfEveryWindow.push(windowName)
}

This approach may take much longer, as it requires length+1 number of requests to get the collection of names.

(Note that the length property of collection object specifiers is handled specially, because collection object specifiers in JXA attempt to behave like native JavaScript Arrays. No .get() invocation is needed (or allowed) on the length property.)

Filtering, and why your code example is slow

The really interesting part of AppleEvents is the so-called "whose clause". This allows you provide criteria with which to filter the objects from which the values will be returned from.

In the code you included in your question, tasks is an object specifier that refers to a collection of objects that have been filtered to only include uncompleted tasks using a whose clause. Note that this is still just reference at this point; until you call .get() on the object specifier, it's just a pointer to something, not the thing itself.

The code you included then implements the for-loop anti-pattern, which is probably why your observed performance is so slow. You are sending length+1 requests to OmniFocus. Each invocation of .name() results in another AppleEvent.

Furthermore, you're asking OmniFocus to re-filter the collection of tasks every time, because the object specifier you're sending each time contains a whose clause.

Try this instead:

var taskNames = Application('OmniFocus').defaultDocument.flattenedTasks.whose({completed: false}).name.get()

This should send a single request to OmniFocus, and return an array of the names of each uncompleted task.

Another approach to try would be to ask OmniFocus to evaluate the "whose clause" once, and return an array of object specifiers:

var taskSpecifiers = Application('OmniFocus').defaultDocument.flattenedTasks.whose({completed: false})()

Iterating over the returned array of object specifies and invoking .name.get() on each one would likely be faster than your original approach.

Answer

While JXA can get arrays of single properties of collections of objects, it appears that due to an oversight on the part of the authors, JXA doesn't support getting all of the properties of all of the objects in a collection.

So, to answer you actual question, with JXA, there is not a way to read a batch of objects and all their properties into memory at once.

That said, AppleScript does support it:

tell app "OmniFocus" to get the properties of every flattened task of default document whose completed is false

With JXA, you have to fall back to the for-loop anti-pattern if you really want all of the properties of the objects, but we can avoid evaluating the whose clause more than once by pulling its evaluation outside of the for loop:

var tasks = []
var taskSpecifiers = Application('OmniFocus').defaultDocument.flattenedTasks.whose({completed: false})()
var totalTasks = taskSpecifiers.length
for (var i = 0; i < totalTasks; i++) {
    tasks[i] = taskSpecifiers[i].properties()
}

Finally, it should be noted that AppleScript also lets you request specific sets of properties:

get the {name, zoomable} of every window of application "Safari"

But there is no way with JXA to send a single request for multiple properties of an object, or collection of objects.