Prevent Escape key from closing an NSPanel with a close box

Question

Does anyone know the best way to prevent an escape key from closing an NSPanel when it's the key window? My panel is a child window and I want it to behave a little more like semi-permanent part of the window, more like a drawer, and for the text controls in it I want to have the Escape key cancel editing.

I recently found more about windows and the Escape key in the Cocoa documentation. In NSResponder class reference under cancelOperation: where it says "the window sends a default action message of cancelOperation: to the first responder and from there the message travels up the responder chain". It seems to be different for an NSPanel, and the window closes without the first responder getting the cancelOperation: call or NSTextView delegates getting their doCommandBySelector: call.

My knowledge of in's & out's of the responder chain is shameful considering that I've been doing OS X work for as long as I have. I was thinking that I need to make keyDown: in my NSPanel subclass behave like that of a normal window. I tried overriding the NSPanel and can catch keyDown:, forwarding the call to NSWindow's keyDown: instead of super, but there was no change, Escape still closed the window without messages to the first responder. Was that even reasonable to try?

I then tried to completely reimplement my panel subclass' keyDown:, making it do this:

[self.firstResponder cancelOperation:self]

I would think this would let my text field handle the escape how it normally expects, and maybe if no text field was first responder then the call would dead end. However, I tried it and the panel simply closes just like before. Obviously I'm not intercepting things at the right level.

Does anyone know what the sequence of methods that run inbetween the low level key press events and the panel closing, or what I'd need to override to intercept it and ensure cancelOperation: goes to my first responder?

Answer 1

Swift port of keith-knauber’s answer:

class ValueEditor : NSObject, NSControlTextEditingDelegate {
enum CommandType {
    case none
    case accept
    case next
    case prev
    case cancel
}

class func commandTypeType(for command: Selector) -> CommandType {
    let commandType: CommandType

    switch command {
    case #selector(NSStandardKeyBindingResponding.insertLineBreak(_:)) :
        fallthrough
    case #selector(NSStandardKeyBindingResponding.insertNewline(_:)) :
        fallthrough
    case #selector(NSStandardKeyBindingResponding.insertNewlineIgnoringFieldEditor(_:)) :
        fallthrough
    case #selector(NSStandardKeyBindingResponding.insertParagraphSeparator(_:)) :
        commandType = .accept

    case #selector(NSStandardKeyBindingResponding.insertTab(_:)) :
        fallthrough
    case #selector(NSWindow.selectNextKeyView(_:)) :
        fallthrough
    case #selector(NSStandardKeyBindingResponding.insertTabIgnoringFieldEditor(_:)) :
        commandType = .next

    case #selector(NSStandardKeyBindingResponding.insertBacktab(_:)) :
        fallthrough
    case #selector(NSWindow.selectPreviousKeyView(_:)) :
        commandType = .prev

    case #selector(NSStandardKeyBindingResponding.cancelOperation(_:)) :
        commandType = .cancel

    default:
        commandType = .none
    }

    return commandType
}


// MARK: - NSControl delegate

func control(_ control: NSControl,
             textView: NSTextView,
             doCommandBy commandSelector: Selector) -> Bool {
    let commandType: CommandType = ValueEditor.commandTypeType(for: commandSelector)

    switch commandType {
    case .cancel:
        control.abortEditing()

        // When the user hits 'ESC' key with a field editor active, cancel the field editor,
        // but return `true` here so that the NSPanel doesn’t close.
        // Hitting 'ESC' a second time will close the NSPanel.
        return true

    default:
        return false
    }

}

}

Don’t forget to set the ValueEditor instance as the delegate of your NSTextView objects!

Answer 2

somewhere in your nib or in code, set your NSTableView delegate to your controller.

Note that setDelegate: is not the same as setDatasource:!

In my case: @interface ValueEditor : NSObject

  + (ValueEditorCmdType)cmdTypeForSelector:(SEL)command
  {
     ValueEditorCmdType cmdType = kCmdTypeNone;
     if ( command == @selector(insertLineBreak:) || command == @selector(insertNewline:) || command == @selector(insertNewlineIgnoringFieldEditor:) || command == @selector(insertParagraphSeparator:))
        cmdType = kCmdTypeAccept;
     else if (  command == @selector(insertTab:) || command == @selector(selectNextKeyView:)  || command == @selector(insertTabIgnoringFieldEditor:))
        cmdType = kCmdTypeNext;
     else if ( command == @selector(insertBacktab:) || command == @selector(selectPreviousKeyView:))
        cmdType = kCmdTypePrev;
     else if ( command == @selector(cancelOperation:) )   
        cmdType = kCmdTypeCancel;
     return cmdType;
  }

  #pragma mark - NSControl delegate
  - (BOOL)control:(NSControl *)control textView:(NSTextView *)textView doCommandBySelector:(SEL)command
  {
     ValueEditorCmdType cmdType = [ValueEditor cmdTypeForSelector:command];
     if ( cmdType == kCmdTypeCancel )
     {
        [control abortEditing];  

        // when user hits 'ESC' key with a field editor active, cancel the field editor,
        // but return YES here so that NSPanel doesn't close.  
        // Hitting 'ESC' a 2nd time will close the NSPanel.
        return YES;
     }
     return NO;
  }