[FIXED] Google Meet background Blur

Issue

I was curious of the new "turn on/off" background blur functionality of Google Meet (currently in test). I have investigated a bit and it seems it is using Tensorflow Lite models:

segm_heavy.tflite
segm_lite.tflite

via WASM

mediapipe_wasm_simd.wasm

while the model graph should be

background_blur_graph.binarypb

The model seems works at the level of the HTMLCanvasElement as far as I can see. Anyone aware of a similar model?

[UPDATE]

Thanks to Jason Mayes and Physical Ed I was able to reproduce a very close background blur effect in the Google's BodyPix demo

The settings of the application are showed in the Controls box. There is a backgroundBlurAmount option that let you customize the blur percentage to apply as well.

The result is almost close to the official Google Meet application.

Solution

majority of segmentation models give alpha channel as a result (some give more, but alpha is most useful) - what is masked and what is not

so if you want to blur the background, its a multi-step process:

1. resize input to model expected size
2. run model to get alpha channel
3. resize output back to original size
4. draw original image on canvas
5. draw alpha channel over it so only foreground stays visible
   for example using ctx.globalCompositeOperation = 'darken'
6. optionally blur it a bit since model output is never perfect
   for example using ctx.filter = blur(8px)`;

so if you want to blur the background, simply apply apply blur simply copy canvas from 4, apply blur on it and draw if back before going to step 5

regarding models, google meet is not bad but i had better results with google selfie model
bodypix is older model, great configurability but not that great results

example code: https://github.com/vladmandic/human/blob/main/src/segmentation/segmentation.ts

Answered By - Vladimir Mandic