How to wrap non oEmbed videos with div?

Question

There is this function:

/**
 * Wrap embed iframe with div.
 */
function div_wrapper($content) {
    // match any iframes
    $pattern = '~<iframe.*</iframe>|<embed.*</embed>~';
    preg_match_all($pattern, $content, $matches);

    foreach ($matches[0] as $match) {
        // wrap matched iframe with div
        $wrappedframe = '<div class="videoWrapper">' . $match . '</div>';

        //replace original iframe with new in content
        $content = str_replace($match, $wrappedframe, $content);
    }

    return $content;    
}
add_filter('the_content', 'div_wrapper');

The problem is, it will wrap everything, and does not work well with, for example twitter post embeded. How do I check if iframe is already wrapped with oEmbed (supported by oEmbed), and if not wrap it with <div class="videoWrapper">iframe html</div>?