I was curious of the new "turn on/off" background blur functionality of Google Meet (currently in test). I have investigated a bit and it seems it is using Tensorflow Lite models:
segm_heavy.tflite
segm_lite.tflite
via WASM
mediapipe_wasm_simd.wasm
while the model graph should be
background_blur_graph.binarypb
The model seems works at the level of the HTMLCanvasElement
as far as I can see. Anyone aware of a similar model?
[UPDATE]
Thanks to Jason Mayes and Physical Ed I was able to reproduce a very close background blur effect in the Google's BodyPix demo
The settings of the application are showed in the Controls box. There is a backgroundBlurAmount
option that let you customize the blur percentage to apply as well.
The result is almost close to the official Google Meet application.
majority of segmentation models give alpha channel as a result (some give more, but alpha is most useful) - what is masked and what is not
so if you want to blur the background, its a multi-step process:
for example using
ctx.globalCompositeOperation = 'darken'
for example using
ctx.filter =
blur(8px)`;so if you want to blur the background, simply apply apply blur simply copy canvas from 4, apply blur on it and draw if back before going to step 5
regarding models, google meet is not bad but i had better results with google selfie model
bodypix is older model, great configurability but not that great results
example code: https://github.com/vladmandic/human/blob/main/src/segmentation/segmentation.ts