feat: ✨ Html Component that will place DOM content on the canvas at the location of its parent babylon node.

[dennemark]

translated from https://github.com/pmndrs/drei html component.

transform property does not work correctly, since matrix might be wrong - need to have a look at difference of threejs matrix and babylonjs matrix.

currently storybook example under GUI>Html Text does not work?Furthermore components within Html Component loose context. Creating Portal does not work, since react has issues with react-babylonjs. Might open a seperate issue for this.

[dennemark]

Hi @brianzinn here is the Html component. I couldn't make it work in storybook, but should normally not have any issues, except if you want to use features, such as transform via css or keep context of parent react. This component creates a new react renderer for each Html component unfortunately...

[brianzinn]

this is great @dennemark ! did you try the Portal on the create-portal branch from #164 ? I'll be definitely be trying this out after work - very excited to give it a go. I'll see how the Matrices may differ and try to get that storybook running and especially losing the context - it may finally be time to switch away from the React context, which does not cross renderer boundaries!! There is an interesting css renderer thread on the babylon forum for rendering video on a plane that should have some interesting snippets.

[dennemark]

@brianzinn unfortunately I did not find the time for #164 today. But will have a look at it soon.
Concerning the renderer this sounds interesting!
In my personal case I am have a context wrapping my app for theming and I tried to use it within the html component, but i could not use the context state. So I just had to use plain css styling for my components.

I actually tried to use ReactDOM.createPortal to create a portal to the parent element of the canvas. Unfortunately, some errors appeared. react-babylonjs thinks a dom element is added and returns this error:

react-babylonjs/src/ReactBabylonJSHostConfig.ts

Line 336 in 4ad062e

throw new Error(`Cannot generate type '${type}/${underlyingClassName}' inside 'react-babylonjs' (ie: no DOM rendering on HTMLCanvas)`)

But i feel the problem is deeper integrated in react: facebook/react#13332

Not sure if this approach would be a solution: https://dev.to/rihdusr/render-a-react-component-in-google-map-171o
I shortly tried to use pmndrs/zustand, too, to move my components to the canvas parent, but that seemed to have triggered many rerenders, so i did not continue in that direction. Currently I use this approach though as an alternative to react-babylonjs createPortal. So I am just storing my components in the zustand and send them to a node in the scene tree. That works a bit, but it feels unstable.

I think it would be good, if the Html component could avoid the ReactDOM.render function and use createPortal or something similar instead. Lets see..

[brianzinn]

yes, you cannot have any DOM elements inside the <Scene> component - they need to go to the DOM renderer. I did play with React.createPortal in the Engine some time ago and also by using a portalCanvas prop - I don't know if you have seen that part of the code (see comment on context loss!):
https://github.com/brianzinn/react-babylonjs/blob/master/src/Engine.tsx#L89

Essentially it would use a supplied canvas instead of creating one, but a selector could be added to find the canvas perhaps for a non-canvas. If it's not about attaching to an existing canvas then I need to understand a bit more about your use-case. I added the XR feature for DOM-Overlay to babylonjs and it used a CSS selector for the overlay feature.

[brianzinn]

also zustand is an interesting option, since it fixes the pesky renderer boundary issue. need to investigate how it handles multiple scenes/etc. currently there are no dependencies outside of babylon, but that is a worthy consideration.

[dennemark]

zustand normally works with global state. there are ways to use it via context, too, which might be good for multiple scenes.

concerning the dom elements within <Scene> I was hoping the portal would just go out of it. but i am not sure if portalcanvas is helpful in my case.

I am using chakra-ui and it has some nice benefits for css props. i.e. <Button colorTheme="primary"/> colors the button in primary colors. But it only works with a <ChakraProvider><MyApp></ChakraProvider>. Within the HTML component I can still use <Button bgColor="#fff">, since chakra is close to normal html/css, but I cannot access properties like the colorTheme anymore. I would have to create a new provider within the html component. But having multiple react renderers with multiple context, does not seem to make sense in my opinion. So I was hoping the children of the HTML component could flow out of react-babylonjs into my react renderer.

<ReactRenderedApp> // app with context
<div>
  <canvas (ReactBabylonjsApp)>
    <Html>
      <SomeComponents> //move from here
    </Html>
</canvas>
<Portal>
   <SomeComponents> // to here
</Portal>
//however currently:
<NewReactRenderedApp> // app without context
   <SomeComponents>
</NewReactRenderedApp>
<div>
</ReactRenderedApp>

[dennemark]

maybe lefthanded vs righthanded is the matrix issue :D https://forum.babylonjs.com/t/html-in-3d-space-part-2-bjs-way-to-matrix-transform/3214/2

[brianzinn]

The context provider can be solved with a bridge like here:
https://github.com/brianzinn/react-babylonjs/blob/master/storybook/stories/babylonjs/Basic/contextBridge.stories.js
It's not pretty!

I see what you are on about now with rendering DOM as that is the HTML children. If you are trying to npm link to a project then you need to actually npm link the react libraries to both projects. If you want to start a test repo then I am happy to get a proper end to end working - or additionally I can publish NPMs as well on 164 as there are no public API changes.

[dennemark]

The context bridge seems to be a good approach for this issue! Thanks a lot!

[brianzinn]

ok - I've merged everything - I think it's OK for now that accessing __rb_createdInstance is in place, but if I can fix without that would be better. It's a hectic week at work, but will make time. Cheers.

[brianzinn]

I'll add also to readme and docs when I get the storybook running! Thanks 😄