Get Bytebuffer from JavaFX Browser

Question

I'm currently working on a project using libgdx. As im tired of bad UI-libraries, I would like to use HTML, CSS and JS.

On my first attempt, I tried java-chromium-embedded and also managed to integrate it into my project. With a few reflections I was able to get the bytebuffer of the browser and render it on top of my game. The problem is that there is only a precompiled win 64 version, but I would also like to support other platforms.

As I wasn't able to compile chromium by myself, I took a look at the javafx Browser. It seems to be much more complicated than the java-chromium-embedded java code so that I wasn't able to get the bytebuffer. Maybe someone could help me, finding the bytebuffer.

I am currently searching in:

• com.sun.webkit.WebPage

  • public void paint(WCGraphicsContext gc, int x, int y, int w, int h)

  • private void paint2GC(WCGraphicsContext gc)

• com.sun.webkit.graphics.WCRenderQueue

But there are so many buffers and lists of buffers, that I don't now where the "real" bytebuffer is.

Answer 1

Using undocumented com.sun APIs is not recommended.

Instead you could snapshot the WebView Node, and use a PixelReader from the snapshot Image to get a pixel Buffer.

I am not sure if that is exactly what you are trying to do. There are probably other ways to achieve what you want (such as overlaying windows rather than working with buffers).

Here is an example of continuously taking snapshots of an animated WebView node (playing a video) into an ImageView.

The top of the image is the original web view. The bottom of the image is a snapshot of the original image with a frame-rate overlaid (approx 60fps when run on a Retina MacBook when the original source WebView is sized to full-screen). The W and H values in the sample code can be modified to check performance for different image capture sizes.

import javafx.animation.AnimationTimer;
import javafx.application.Application;
import javafx.concurrent.Worker;
import javafx.geometry.Pos;
import javafx.scene.Scene;
import javafx.scene.control.Label;
import javafx.scene.image.*;
import javafx.scene.layout.*;
import javafx.scene.web.WebView;
import javafx.stage.Stage;

public class VideoPlayerWithFrameByFrameSnapshot extends Application {
    private static final int W = 800;
    private static final int H = 400;

    private final long[] frameTimes = new long[100];
    private int frameTimeIndex = 0 ;
    private boolean arrayFilled = false ;

    public static void main(String[] args) {
        launch(args);
    }

    @Override
    public void start(Stage stage) throws Exception {
        // create a WebView
        WebView webview = new WebView();
        webview.setMinSize(W, H);
        webview.setPrefSize(W, H);
        webview.setMaxSize(W, H);

        // create an image viewer for a WebView snapshot
        WritableImage image = new WritableImage(W, H);
        ImageView imageview = new ImageView(image);

        Label framerateLabel = new Label();

        // continuously snapshot the webview and measure current framerate.
        AnimationTimer timer = new AnimationTimer() {
            @Override
            public void handle(long now) {
                webview.snapshot(null, image);

                long oldFrameTime = frameTimes[frameTimeIndex] ;
                frameTimes[frameTimeIndex] = now ;
                frameTimeIndex = (frameTimeIndex + 1) % frameTimes.length ;
                if (frameTimeIndex == 0) {
                    arrayFilled = true ;
                }
                if (arrayFilled) {
                    long elapsedNanos = now - oldFrameTime ;
                    long elapsedNanosPerFrame = elapsedNanos / frameTimes.length ;
                    double frameRate = 1_000_000_000.0 / elapsedNanosPerFrame ;
                    framerateLabel.setText(
                        String.format(
                            "Current frame rate: %.3f", 
                            frameRate
                        )
                    );
                }
            }
        };

        // start the snapshot process once the web view load succeeds. 
        webview.getEngine().getLoadWorker().stateProperty().addListener(
            (observable, oldValue, newValue) -> {
                System.out.println(newValue);
                if (Worker.State.SUCCEEDED == newValue) {
                    timer.start();
                }
            }
        );

        // load up the big buck bunny video. 
        webview.getEngine().load(
                "http://camendesign.com/code/video_for_everybody/test.html"
        );

        // layout the scene. 
        VBox layout = new VBox(
                webview,
                new StackPane(
                        imageview,
                        framerateLabel
                )
        );
        StackPane.setAlignment(framerateLabel, Pos.TOP_LEFT);

        stage.setScene(new Scene(layout));
        stage.show();
    }
}

Framerate measurement code comes from James_D's answer to:

• What is the preferred way of getting the frame rate of a JavaFX application?

As an aside: If you want to make a game with a pure HTML UI, then I don't understand why you would involve libgdx and wish to access low level drawing primitives for capturing the graphic output of an embedded browser (rather than just displaying the HTML UI in the user's default browser or directly in an embedded browser you ship with your game), but I guess you have your reasons for pursuing this approach. In the future I will try to refrain from attempting to answer questions where the motivation and objective is quite unclear. It's not a bad question, it's just a question for which I don't think I can be of much assistance.