• Home
  • Documentation
  • Forum
  • Blog
  • Users
    • Map
    • Userlist
    • Businesses
    • Userpages
    • Connections
  • Contributions
    • All
    • Store
  • Downloads
    • vvvv beta
    • vvvv beta previews
  • Store

    Show / Hide Table of Contents

    VVVV.Tutorials.Mapping.3D

    Author: elliotwoods
    Date: 21 May, 2012
    Category: tutorial
    Credits:

    elliotwoods, microdee (for soft shadows in runtime demo)

    Download

    64bit

    5-ProjectorN (DX11).zip
    02 Jul, 2014 - 17:42
    Fixes: intersect had no depthbuffer, table load incorrect.
    4-ProjectorN_b31.2 (DX11).zip
    13 Dec, 2013 - 13:44
    DX11 Version of the modules and fx b31.2
    3-ProjectorN_b29 (DX11).zip
    17 Apr, 2013 - 11:18
    DX11 Version of the modules and fx
    2-VVVV.Tutorials.Mapping.3D.zip
    21 May, 2012 - 16:02
    VVVV.Tutorials.Mapping.3D (May 2012)
    1-VVVV.Tutorials.Mapping2-3D.zip
    03 Jun, 2012 - 14:02
    VVVV.Tutorials.Mapping.3D (June 2012)
    0-VVVV.Tutorials.Mapping2-3D.zip
    03 Jun, 2012 - 14:38
    VVVV.Tutorials.Mapping.3D (June 2012)

    Description

    Tutorial on 3D projection mapping using the CalibrateProjector node (wrapping OpenCV's CalibrateCamera routine).

    1. Quickstart guide:

    Downloads

    VVVV 27.2 or above

    opencv-plugin-(alpha)

    Tutorial files (in download link at bottom of description)

    <iframe width="560" height="315" data-src="https://www.youtube

    1. Detailed instructions

    Scan object in Reconstructme

    Convert .3ds to .x file

    Controls to add / modify points in world points

    Use your mouse in the World renderer to move control points to features on the mesh (virtual object).

    Enter Add new point
    Backspace Delete point
    Tab Select next point
    Shift+Tab Select last point

    Controls to modify points in projector space

    Use your mouse in the world renderer to drag the control points coming out of the projector onto the corresponding features of the real object. | | | |---|---| Space | Reset point to world view camera Tab | Select next point Shift+Tab | Select last point

    Export matrices

    Use the SaveViewProjection node at the bottom of the patch to save the View and Projection matrices (i.e. the result of the calibration process). These can then be loaded into other patches with the LoadViewProjection node (packaged in the download).

    1. Introduction to 3D projection mapping

    3D projection mapping

    What I'd describe as '3D projection mapping' is the act of re-projecting a virtual 3D object onto its real world counterpart using a video projector*. Thereby all of the features of the real object which are visible from the point of view of the projector have an image projected onto them, and this image is 'extracted' from the corresponding surfaces of the virtual counterpart object.

    * thereby defining '2D projection mapping' as lining up 2D shapes in a projector image with real world features in the projector's line of sight.

    The camera model

    A projector has the same optical properties as a camera, whereby a scene in 3D is projected onto a plane in 2D (camera) or an image from a plane in 2D projected onto a 3D scene (projector). This comes from the Principle of Reversibility, whereby in optics reversing any path of light results in a valid system.

    Film example

    Imagine a film camera on a tripod in front of a table which is covered in objects. When we take a photo, light hits the objects and is scattered, rays that are scattered directly towards the camera's aperture are captured on the film. Each minute section of the film is looking through the aperture at one part of the scene, and therefore captures light from that part of the scene only.

    Presume we don't move the film at all, it stays in the camera as it was when we took the photo. If we then develop the film inside the camera, then the image of the scene will appear on the film. If we open the aperture now, each section of the film is still looking through the aperture at its corresponding section of the scene. Imagine we shine light onto the film so that the film is bright, it is scattering light from each section of the film based on the intensity pattern of the image. This light then travels back through the aperture and will hit the section of the scene corresponding to each part of the image. The image of the scene will be 'projection mapped' back onto the scene (or perhaps the negative image).

    Virtual camera

    In the film example, the 'projector' and camera conveniently share the exact same optics:

    • The position and rotation relative to the scene (pose)
    • The lens, size of film and distance between lens and film (relative focal length)
    • The planar offset of the film with respect to the lens (lens offset)
    • The barrel distortion of the lens (lens distortion)

    If we want to match a virtual camera with a real video projector, then we must create a camera that shares all these above properties with the projector.

    Conveniently, the standard model of a video projector is to have no lens distortion (i.e. if you shine the image onto a flat surface, you will always get a rectangular image). Therefore we omit this property, and can very effectively define a projector with standard computer graphics camera matrices (view and projection matrices).

    The remaining properties can be categorised into 2 sets: | | | | |---|---|---| Extrinsics | World Transform | Translation <3>, Rotation <3> Intrinsics | Projection Transform | Focal length (XY) <2>, Lens offset (XY) <2>

    This gives us 10 degrees of freedom for our virtual camera.

    Aligning a camera model

    We can either enter these 10 parameters (above) manually as in HowTo Project On 3D Geometry, or we could find a mechanism that will calculate these for us.

    The previous automatic route for this was Pade projection mapping (which required a custom vertex shader). This CalibrateProjector uses OpenCV's CalirateCamera function in order to perform this automatic solving for us, and provides standard View and Projection matrices.

    The general benefit of the automatic route is that it allows a more immediate interaction between the person calibrating and the result that they want to achieve. By manipulating control points on the mesh and in the projector, the operator can more quickly and confidently achieve an accurate mapping, and fine tune the mapping by adding extra control points in problem areas, without compromising the existing data entered.

    1. Further references

    2. Notes on 3D mapping in VVVV : HowTo Project On 3D Geometry

    3. Kyle McDonald's Mapamok : https://github.com/YCAMInterlab/ProCamToolkit/wiki/mapamok-(English)

    4. Workshop files for 'Projecting on the things' workshop at Node 10 : http://workshops.vvvv.org/-ProjectingOnThings/

    5. Watch out for:

    Note: ## World window selects points at back

    This can happen if your graphics card doesn't have the same depth buffer settings available as mine If you encounter a problem selecting points in the World view where: When you select a points on the front of an object, the patch instead selects a point on the back of the object, then try the following:

    • Open Mesh.SelectPoints (1/3 way down calibrate.v4p on left hand side)
    • Open Intersect (1/5 way down Mesh.SelectPoints.v4p on left hand side)
    • Select the Renderer (EX9) in the patch (the box in the middle)
    • Open Herr Inspektor (Ctrl+I)
    • Check that the 'Windowed Depthbuffer Format' is not set to 'None', e.g. set it to 'D16'

    Comments

    Comments are no longer accepted.
    Please create a new topic in the vvvv beta forum to discuss this contribution.
    sunep
    21 May, 2012 - 19:24

    Youtube says: "This video has been removed because it is too long" :(

    we love you voice on video Elliot...

    Noir
    21 May, 2012 - 19:28

    same problem here

    elliotwoods
    21 May, 2012 - 19:36

    yep. cheers for notice! noted and currently uploading on another account which can go > 15mins sorry!

    Urbankind
    21 May, 2012 - 21:18

    Very nice tutorial, thanks for your time :)

    microdee
    22 May, 2012 - 01:28

    where are the plugins for the tables?

    elliotwoods
    22 May, 2012 - 06:15

    @microdee - these are sneakily included in the latest OpenCV plugin pack (May 2012). Should be moved to a separate download in future. Using your soft shadow contrib in the runtime example.

    u7angel
    22 May, 2012 - 10:10

    so nice ! cheers for the reconstructme info

    tekcor
    22 May, 2012 - 10:30

    very nice thanks elliot! which scanner did you use in the example?

    Desaxismundi
    22 May, 2012 - 10:41

    NEAT!

    Noir
    22 May, 2012 - 14:14

    thank u Elliot for the great contribution and for the tutorial also!

    microdee
    22 May, 2012 - 14:21

    @elliotwoods: oooh ok. hey what an honor!:D update: @elliotwoods: there's no ValuesTableBuffer.dll in the opencv contrib, however i found the source on your github. is there a compiled version somewhere?

    elliotwoods
    23 May, 2012 - 20:54

    Wow. Will include. In asakusa restaurant. Will post corrections to opencv dll's from xd_nitro's studio tomorrow

    elliotwoods
    24 May, 2012 - 09:51

    ack. actually it's going to be much easier to wait until i'm back on my own PC has to be the weekend :(

    elliotwoods
    30 May, 2012 - 22:10

    @microdee - the filename had changed to VVVV.Nodes.SpreadTableValue.dll perhaps you were looking at the old tutorial patch?

    elliotwoods
    03 Jun, 2012 - 12:56

    @tekcor : Asus Xtion Live (equivalent to a kinect camera)

    d0this
    03 Jun, 2012 - 16:22

    PURE AVVVVESOMENESS!!!!!

    Got some Problems with Points in World Projector. Sometimes when i try moving the Points in World Projector they are kind of sticky like to a grid or something and when i create a Point they do not appear at the courser but somewhere else. Am i missing something?

    Anyway this is just great thanks so much for this contribution. Makes life much easier!!!!!

    ales9000
    22 Jun, 2012 - 15:44

    you are my personal Hero!!!

    this is great!!!

    thank you elliot!!!

    newemka
    29 Jun, 2012 - 14:03

    Hello all and thank you Elliot for this piece of technical and helpful art.

    It works pretty well.

    Does someone have any idea on how to display a picture in background of the Projection renderer?

    I've tried in many way to do it but my background get always distorted.

    It would be helpful to put a picture in background in order to be able to make experimentation without having to switch on the video projector.

    Edit: I've found a way to put a picture in background, but it's dirty I can't show this... i'm sure you guys knows how to do it the clean way

    elliotwoods
    01 Jul, 2012 - 16:04

    @newemka : paste the following for an example of putting a quad fullscreen in the background of a 3d render (note Billboard and ZWriteEnable set to false)

    https://gist.github.com/3028828

    newemka
    04 Jul, 2012 - 00:50

    @Elliotwoods Thanks for the tip, I was looking on the billboard side and it works until I add the fourth calibration point then the camera projection shift my background picture. My trick is to render the projection renderer as DX9Texture and blend it with my background FileTexture on a FullscreenQuad in a new DirectX Renderer window.

    elliotwoods
    06 Jul, 2012 - 21:35

    @newemka - did you make sure that Billboard was set to WithinNormalisedProjection ? (also i think there's a separate node for this. Either way, glad you found a solution that works for you!

    colorsound
    19 Aug, 2012 - 16:56

    Hi, thanks for the nice tut and solution Elliot.

    i found a strange behaviour. I,m using beta 28 and works very well i had to repatch some staff and change the keyboard subpatch.

    Chaging the patch to other folder you get all nodes working apart from calibrate camera. i found that if you have your project folder and a folder with the opencv in it an a folder with the patches inside seem to work again. it does not matter that you have the opencv already add in the vvvv root , not doing this it did not work for me. just in case you have similar problem. cheers

    Desaxismundi
    21 Aug, 2012 - 20:42

    confirmed. had this path troubles too with beta <28.

    circuitb
    22 Aug, 2012 - 14:14

    also if you get this msg using the CalibrateCamera node: "OpenCV: For non-planar calibration rigs the initial intrinsic matrix must be specified"

    set the Flags enum to CV_CALIB_USE_INTRINSIC_GUESS CV_CALIB_ZERO_TANGENT_DIST CV_CALIB_FIX_K1 ..K2 ..K3 ..K4 ..K5 ..K6

    m4d
    30 Aug, 2012 - 00:04

    finally tried this and the projector node 3.0 is just WOW :o

    d0this
    24 Oct, 2012 - 07:03

    I have the same issues as color.

    Keyboard subpatch only working on <28 (beside 28alpha).

    TwoBeAss
    24 Oct, 2012 - 12:01

    i got that keyboard problems too... need to replace the keyboard (window) with keyboard (global)...

    sansui
    10 Feb, 2013 - 17:27

    고맙습니다 엘리엇우드님!

    ravazquez
    05 Mar, 2013 - 23:16

    Hi, I am new to the whole vvvv scene and am trying to get this to work, but am having problems with ValuesTableBuffer.dll, I cannot seem to find it anywhere.

    Could someone please help me figure this out?

    Thanks!

    colorsound
    04 Apr, 2013 - 11:15

    @Elliot please feel free to remove the DX11 Version if not appropriate,tried to contact you but did not reach you. Works well in my side.

    metrowave
    04 Apr, 2013 - 13:57

    Thanks for keep updating this!

    ShmulikF
    10 May, 2013 - 10:07

    Great Tutorial!!! Thanks Elliot. I'm having a problem with the world projector: I can't create points, or move the model in it. Instead of regular mouse pointer, I have only blinking red square like, that allows me to set only 1 point. I'll appreciate someones help on that matter.

    uploaded a screenshot of this issue

    Thanks again

    nihilist
    07 Jun, 2013 - 16:09

    Please, update link on OpenCV plugins, it is not available now.

    Urbankind
    08 Jun, 2013 - 21:16

    @ShmulikF: Its because of the keyboard subpatch is broken. Fix the keymatch and connect them to respective IO's within the same sub patch. It should work.

    Joanie_AntiVJ
    21 Jun, 2013 - 15:40

    I desperatly need help on a simple projector calibration.. Does anyone have Elliot's patch running on his machine ? You can hit me on skype (crustea). Any help would be awesome :)

    postaldude
    24 Jun, 2013 - 21:27

    i just cant get the calibrate camera node to work. it doesnt solve and flashes red while manipulating points in the renderer.
    @colorsound, i dont really understand what your folder structure is. could you please describe what exactly you did to make it work again?

    sonostrano
    27 Jun, 2013 - 18:06

    Just a question: is there anyone with this patch working with vvvv 30.2? thanks

    vex3d
    30 Jun, 2013 - 00:05

    Hey, Thanks for this tutorial, has opened up a whole new world for me.. I attended one of your classes at MAD Lab the other year and we touched on 3d mapping at the end.. I'd never quite managed to figure it out until this so thanks! I've created a patch to hook up the iPad with touchOSC and it allows the alignments to be done remotely which is great for larger scale mapping projects, and you can also use it to control the show.. It's not really in a shareable state, but maybe I'll get it there one day..

    Just wanted to say thanks :)

    Stu

    micdido
    01 Aug, 2013 - 17:29

    I have a problem with this tutorial. I miss the file VVVV.Nodes.OpenCV.dll. I searched the addons but this file is missing. Where can I download it? thanks

    mrboni
    07 Aug, 2013 - 00:33

    Just been trying this.

    I reach the end of calibration (with a mesh from reconstructme) and the calculated view is facing in a completely different direction from the scene. I've tried severaltimes, with a few vvvversions.

    I was very precise last time ans the reprojection error was only about 1.5px, but still the view faces the wrong way.

    any idea?

    mrboni
    07 Aug, 2013 - 01:47

    ok, it seems the view is inverted in the z direction somehow!

    have 'fixed' with a scale z of -1

    odd

    another issue is vvvv freezing every couple of seconds while moving markers with the mouse. seems to be something to do with the tables

    elresidente
    15 Aug, 2013 - 15:29
    quote:username:mrboni ok, it seems the view is inverted in the z direction somehow!

    have 'fixed' with a scale z of -1

    I fixed this in the may 2012 version with 30.2 by changing the coordinates pin of the CalibrateCamera node inside of CalibrateProjector from "vvvv" to "OpenCV".

    everyoneishappy
    16 Aug, 2013 - 06:03

    @ mrboni yes I ended up doing the same (also with calibrate camera). It seemed to work for my purposes but I'm not sure if scaling the Z at -1 is 100% correct. Also from memory I think I did that on the perspective transform.

    Would love to hear from Elliot about this, I thought he mentioned in his workshop at Node13 that some of the open CV stuff might be working with an Open Framewroks style matrix.

    chriss0212
    13 Dec, 2013 - 11:06

    hi colorsound

    i try to use your directx11 version but have problems with the input element node (red) vvvv.dx11.nodes.dll is missing but it is there...all other nodes seems to be ok. maybe wrong version?

    with an older version of eliot it seems, that i have also the inverted z problem. somebody who can help WHERE i have to corrct that?

    thx and greetz

    christian

    colorsound
    13 Dec, 2013 - 13:46

    hi chriss0212

    i have just download a fresh last vvvv_45beta31.2_x86 with its addons also last version of directx11-nodes-alpha which is 31.2 version as well, then vvvv.packs_.image-b31-x86 .

    i was missing VVVV.Nodes.SpreadTableValue.dll but took it from other version and worked btw i vote for this one to be in the addons.

    Then as you said input element node was red ,you just double click and select the node again and it works, that happens sometime with some older DX11 nodes i guess there was a change on the naming but nodes are there.

    I have saved it and uploaded it ;D.

    chriss0212
    13 Dec, 2013 - 17:08

    hi colorsound

    thx for the tip...this problem is solved ;)

    but in your patch i can not select any point because i am missing the red rectangle i have in older patches from elliot

    any idear?

    thx and greetz

    christian

    chriss0212
    13 Dec, 2013 - 17:26

    hi colorsound

    me again ;)

    everything seems to work! thx for your help and great patch!

    but one last question: how can i show the wireframe of the geometry in the projection output...in the mesg view i have found it

    thx and greetz

    christian

    chriss0212
    13 Dec, 2013 - 18:53

    hi colorsound

    one big problem is left. if i close the patch and reopen it the x and y values for the projection are deleted!

    any idear?

    greetz and thx

    christian

    nrsz
    30 Jan, 2014 - 14:37

    hi everybody,

    thanks for the great work! i have a question: i`ll map a room with 2 projectors that are facing each other (each one in the opposite corner of the room). The 2 projectors are connected to two laptops (there is no dualhead possibility - greetings from india ;) - i plan to animate the light position and send it via osc to the 2nd laptop/beamer. Any idea how i can align the 3d coordinates of the 2 patches/projections?

    thank you very much!

    robe
    30 Jan, 2014 - 19:48

    Just do the calibration procedure from both projector and export both the matrices to plug them in view and projection of renderer..

    nrsz
    31 Jan, 2014 - 19:47

    thanks! unfortunately i'm already stuck loading the 3d file. actually any 3d file - not even the v.x is showing up. also no .dea, .ply or .obj files. the only files loading so far are 2-subset-cube.x and 6-subset-cube.x any idea?

    i'm using the latest v4 + dx11- & image packs on win8.1, no red boxes. thank you in advance!

    newemka
    01 Feb, 2014 - 15:08

    A 3D object always needs UV mapping coordinates.

    nrsz
    19 Mar, 2014 - 00:02

    alright, blame win8. i remembered that i had it all working last year so i started to do downgrade. 29.2 and VVVV.Nodes.OpenCV works - besides keyboard issues - fine on win8. thanks for the help and greetings from http://www.maakali.org/ResidenceSEA/residency/

    HelenaAndreina
    15 Apr, 2014 - 14:36

    Hi, I'm having some trouble with this tutorial, because when I'm opening the mesh and put the scale my mesh appears far away, and I can´t not even rotate or translate it in the renderer !!! Anyone can help me!!! Thanks!!

    HelenaAndreina
    18 Apr, 2014 - 13:37

    Hi guys I'm having problems, because when I select 5 point the projector renderer doesn´t change the colour to silver, not even with more points! Anyone with the same problem??

    HelenaAndreina
    30 Apr, 2014 - 11:39

    My calibrate projector node isn't working correctly. I have all nodes with everything well ( they aren't red) When I select the points in world renderer the calibrate projector blinks red and never appears sucess. Anyone with the right version of open CV pluggins to this Tutorial?

    antokhio
    03 Jul, 2014 - 17:15

    do the forum thread with screenshots of problem

    colorsound
    22 Jul, 2014 - 10:28

    hello guys does someone have a working version for beta32.1 ? cheers

    colorsound
    22 Jul, 2014 - 17:34

    hi ok got it working with 32.1,will put how soon. cheers

    manolito
    23 Sep, 2014 - 11:57

    @colorsound do you have an update fix for 32.1 or even better 33.1? Thnx.

    colorsound
    27 Sep, 2014 - 10:11

    @manolito yes it works with 33.1 too, basically the problem is the VVVV.Nodes.OpenCV plugins version , it seems last imagepack has diferent behaviour.

    i could clean and upload the patch with the plugins that work in my side,but not now if you are in a hurry contact me on skype. cheers ;D

    manolito
    30 Sep, 2014 - 12:40

    @colorsound will get it touch via Skype :)

    Mooney
    09 Jan, 2015 - 22:40

    Hi. I'm trying to follow this tutorial. But the mouse and keyboard controls don't work for me. I'm not really sure how to fix the keyboard node as others have suggested. Using Window 8.1 with vvvv 3.3 64bit. Can't move around the world view, I can move the projector though. But I can't set points.

    Erandu
    12 Feb, 2015 - 06:55

    @Mooney I am not sure about the keyboard since it works correctly on my side, but I found that the issue about setting points is related to missing links in the "ProjectorN (DX11)\Modules\dx11\Pipet (DX11)\Pipet (DX11).v4p" patch. The texture2d and DynamicBuffer nodes needs to be correctly connected to the CS_Pipet shader.

    azercay
    09 Dec, 2015 - 12:18

    can someone help me to follow this tutorial. looks like many steps are outdated and impossible to perform. i have many red nodes, but all packs are on the right places, i've intalled Images.Pack instead of OpenCV, but i cant figure out how to get things work.

    nrsz
    26 Jun, 2016 - 15:52

    everthing seems to load fine in the vvvv_50alpha34.103_x86 version, the only problem is that i get the status message "Insufficient points" no matter what.

    Latest working version is for me 31.2 with image pack.

    ndrv
    08 Mar, 2020 - 19:02

    Nice work! I'm just wondering if this tutorial is still supposed to be the "tool to use" for 3d projection mapping in vvvv_beta_39_x64? I was following the tutorial and made the same observation as nrsz. In addition there is an exception thrown saying:

    • 00:10:13 ERR : Exception occured in TMPluginWrapperNode.Evaluate
    • 00:10:13 ERR : The type initializer for 'Emgu.CV.CvInvoke' threw an exception.

    But there are no permanent red nodes (only temporary).

    So, basically my question is, if there is an up-to-date tool resemblig the functionality or a work-around getting the libraries running? I found the **Step01_CalibrateProjector** patch in the VL.OpenCV package, which looks quite great but maybe there are other tool or workflows.

    Thank you for your help!

    joreg
    08 Mar, 2020 - 23:53

    @ndrv this tutorial does 2 things:

    1. use reconstructme to create a 3d mesh from a scene
    2. use CalibrateProjector regarding 2) VL.OpenCV today is the way to go. best start a forum thread if you have any questions regarding it.
    ndrv
    09 Mar, 2020 - 11:12

    Thanks joreg! Yes, it was only about the calibration. Will try to combine best of both worlds now.

    • Improve this Doc

    © 2020 by vvvv group. Imprint | Privacy Policy.
    Thanks to DocFX for building this website. Analytics by Plausible.

    Back to top