Subclassing NSTextStorage breaks list editing

Question

I have a basic Mac app with a standard NSTextView. I'm trying to implement and use a subclass of NSTextStorage, but even a very basic implementation breaks list editing behavior:

1. I add a bulleted list with two items
2. I copy & paste that list further down into the document
3. Pressing Enter in the pasted list breaks formatting for the last list item.

Here's a quick video:

Two issues:

1. The bullet points of the pasted list use a smaller font size
2. Pressing Enter after the second list item breaks the third item

This works fine when I don't replace the text storage.

Here's my code:

ViewController.swift

@IBOutlet var textView:NSTextView!

override func viewDidLoad() {
   [...]
   textView.layoutManager?.replaceTextStorage(TestTextStorage())
}

TestTextStorage.swift

class TestTextStorage: NSTextStorage {

    let backingStore = NSMutableAttributedString()

    override var string: String {
        return backingStore.string
    }

    override func attributes(at location: Int, effectiveRange range: NSRangePointer?) -> [NSAttributedString.Key:Any] {
        return backingStore.attributes(at: location, effectiveRange: range)
    }

    override func replaceCharacters(in range: NSRange, with str: String) {
        beginEditing()
        backingStore.replaceCharacters(in: range, with:str)
        edited(.editedCharacters, range: range,
               changeInLength: (str as NSString).length - range.length)
        endEditing()
    }

    override func setAttributes(_ attrs: [NSAttributedString.Key: Any]?, range: NSRange) {
        beginEditing()
        backingStore.setAttributes(attrs, range: range)
        edited(.editedAttributes, range: range, changeInLength: 0)
        endEditing()
    }
}

Answer 1

You have found a bug in Swift (and maybe not just in the Swift libraries, maybe in something a bit more fundamental).

So what is going on?

You will be able to see this a bit better if you create a numbered list rather than a bulleted one. You don't need to do any copy and paste, just:

1. Type "aa", hit return, type "bb"
2. Do select all and format as a numbered list
3. Place cursor at the end of "aa" and hit return...

What you see is a mess, but you can see the two original numbers are still there and the new middle list item you started by hitting return is where all the mess is.

When you hit return the text system has to renumber the list items, as you've just inserted a new item. First, it turns out that it performs this "renumbering" even if it is a bulleted list, which is why you see the mess in your example. Second, it does this renumbering by starting at the beginning of the list and renumbering every list item and inserting a new number for the just created item.

The Process in Objective-C

If you translate your Swift code into the equivalent Objective-C and trace through you can watch the process. Starting with:

1) aa
2) bb

the internal buffer is something like:

\t1)\taa\n\t2)\tbb

first the return is inserted:

\t1)\taa\n\n\t2)\tbb

and then an internal routine _reformListAtIndex: is called and it starts "renumbering". First it replaces \t1)\t with \t1) - the number hasn't changed. Then it inserts \t2)\t between the two new lines, as at this point we have:

\t1)\taa\n\t2)\t\n\t2)\tbb

and then it replaces the original \t2)\t with \t3)\t giving:

\t1)\taa\n\t2)\t\n\t3)\tbb

and it's job is done. All these replacements are based on specifying the range of characters to replace, the insertion uses a range of length 0, and go through:

- (void)replaceCharactersInRange:(NSRange)range withString:(NSString * _Nonnull)str

which in Swift is replaced by:

override func replaceCharacters(in range: NSRange, with str: String)

The Process in Swift

In Objective-C strings have reference semantics, change a string and all parts of the code with a reference to the string see the change. In Swift strings have value semantics and strings are copied (notionally at least) on being passed to functions etc.; if the copy is changed in called function the caller won't see that change in its copy.

The text system was written in (or for) Objective-C and it is reasonable to assume it may take advantage of the reference semantics. When you replace part of its code with Swift the Swift code has to do a little dance, during the list renumbering stage when replaceCharacters() gets called the stack will look something like:

#0  0x0000000100003470 in SwiftTextStorage.replaceCharacters(in:with:)
#1  0x0000000100003a00 in @objc SwiftTextStorage.replaceCharacters(in:with:) ()
#2  0x00007fff2cdc30c7 in -[NSMutableAttributedString replaceCharactersInRange:withAttributedString:] ()
#3  0x00007fff28998c41 in -[NSTextView(NSKeyBindingCommands) _reformListAtIndex:] ()
#4  0x00007fff284fd555 in -[NSTextView(NSKeyBindingCommands) insertNewline:] ()

Frame #4 is the Objective-C code called when return was hit, after inserting the newline it calls the internal routine _reformListAtIndex:, frame #3, to do the renumbering. This calls another Objective-C routine in frame #2, which in turn calls, frame #1, what it thinks is the Objective-C method replaceCharactersInRange:withString:, but is in fact a Swift replacement. This replacement does a little dance converting Objective-C reference semantic strings to Swift value semantics strings and then calls, frame #0, the Swift replaceCharacters().

Dancing is Hard

If you trace through your Swift code just as you did the Objective-C translation when the renumbering gets to the stage of changing the original \t2)\t to \t3)\t you will see a misstep, the range given for the original \t2)\t is what is was before the new \t2)\t was inserted in the previous step (i.e. it is off by 4 positions)... and you end up with a mess and a few more dance steps later the code crashes with a string referring error as the indices are all wrong.

This suggests that the Objective-C code is relying on reference semantics, and the choreographer of the Swift dance converting reference to value and back to reference semantics has failed to meet the Objective-C code's expectations: so either when the Objective-C code, or some Swift code which has replaced it, calculates the range of the original \t2)\t it is doing so on string which hasn't been altered by the previous insertion of the new \t2)\t.

Confused? Well dancing can make you dizzy at times ;-)

Fix?

Code your subclass of NSTextStorage in Objective-C and go to bugreport.apple.com and report the bug.

HTH (more than it makes you dizzy)