How can I remove a page from zend-navigation?

Question

I am trying to remove a page from zend-navigation but it does not appear to get removed.

if (!$task->getMilestone()) {
    $navigation = $this->getServiceLocator()->get('Navigation');
    $page = $navigation->findOneBy('id', 'milestone_edit_task');
    $navigation->removePage($page);
    $page = $navigation->findOneBy('id', 'milestone_edit_task');
    die(var_dump($page->get('id')));
}

findOneBy() seems to correctly find the page. But removePage($page) does not remove the page. I'm doing the second findOneBy() to prove the page still exists.

How can I remove the page?

(this code is within a controller action, is that too late?)

As requested here is my config for container;

// Navigation
'navigation' => array(
    'default' => array(

        // Projects
        array(
            'label' => '<i class="fa fa-cubes"></i> Projects',
            'route' => 'project/default',
            'controller' => 'project',
            'pages' => array(
                array(
                    'label' => 'Project Detail',
                    'controller' => 'project',
                    'action' => 'detail',
                    'pages' => array(
                        array(
                            'label' => 'Add milestone',
                            'controller' => 'milestone',
                            'action' => 'add'
                        ),
                        array(
                            'label' => 'Edit milestone',
                            'controller' => 'milestone',
                            'action' => 'edit'
                        ),
                        array(
                            'label' => 'Delete milestone',
                            'controller' => 'milestone',
                            'action' => 'delete'
                        ),
                        array(
                            'label' => 'Add task',
                            'controller' => 'project',
                            'action' => 'task'
                        ),
                        array(
                            'label' => 'Milestone Detail',
                            'controller' => 'milestone',
                            'action' => 'detail',
                            'pages' => array(
                                array(
                                    'label' => 'Add Task',
                                    'controller' => 'task',
                                    'action' => 'add'
                                ),
                                array(
                                    'id' => 'milestone_edit_task',   // <---
                                    'label' => 'Edit Task',
                                    'controller' => 'task',
                                    'action' => 'edit'
                                ),
                                array(
                                    'label' => 'Delete Task',
                                    'controller' => 'task',
                                    'action' => 'delete'
                                ),
                                array(
                                    'label' => 'Add comment',
                                    'controller' => 'milestone',
                                    'action' => 'comment'
                                ),
                                array(
                                    'label' => 'Task Detail',
                                    'controller' => 'task',
                                    'action' => 'detail',
                                ),
                            ),
                        ),
                        array(
                            'label' => 'Add comment',
                            'controller' => 'project',
                            'action' => 'comment'
                        ),
                    ),   
                ),
                array(
                    'label' => 'Edit comment',
                    'controller' => 'comment',
                    'action' => 'edit'
                ),
                array(
                    'label' => 'Delete comment',
                    'controller' => 'comment',
                    'action' => 'delete'
                ),
            ),
        ),
    ),
),