VVVV.Nodes.DeckLink
| Author: | elliotwoods |
| Date: | 05 Jul, 2012 |
| Category: | plugin |
| Credits: | Elliot Woods Developed with support from Lumacoustics, UK |
Download
64bit
Description
 help.png)
- Introduction
BlackMagic Design produce some pretty interesting products in the video market. Of particular interest to VVVV users is the Intensity range of products http://www.blackmagic-design.com/products/intensity/ which allows a computer to capture video live from another video source (e.g. a laptop, a video camera) over a range of video connectors (HDMI, composite, component), and the Intensity product is available in a number of packages (USB3.0, PCIe, thunderbolt). They start at $200.
An example usage scenario would be to generate visuals on computer A (e.g. this could be a Mac running Modul8), output them on DVI and connect this to the HDMI input of the BlackMagic Intensity, and then capture that content into VVVV as a texture.
Traditionally this has been possible using DirectShow, but this route has always caused noticeable latency (which increases over time). Using this plugin introduces near-zero latency (as fast as the card will allow).
- Latency
bjeorn below states 2-3 frames latency between PC A's monitor when capturing and presenting on PC B's monitor by videoing a counter on both screens.
- Usage
Use the ListDevices (DeckLink) node to give you IDeckLink devices, which you then can plug into the VideoIn (DeckLink EX9.Texture) node. Make sure to select the correct video mode on the VideoIn node to correspond with the signal being sent to the capture device (this is not automatic).
Note that the texture output from the plugin is in a strange YUV format, and to decode it, you will need to use the YUV2RGB TextureFX (supplied in this package).
If your computer is very slow (or very busy), then it may not be able to keep up with the capture device, in which case it will begin to buffer frames to make sure that you don't miss anything. In this case, if you do want to improve the latency of the stream, then bang the Flush input of VideoIn to flush the cache (feel free to attach a toggle to flush every frame).
- Notes on YUV2RGB
Why YUVtoRGB?
I found the data which comes out from the BlackMagic Intensity to be in YUV format (UYVY). On my system (ATI 6770, Intel Core i7 * 4/4), I couldn't get any of the native DirectX YUV texture formats to work. Furthermore, I found decoding the data on the CPU to be a serious issue in C# as it doesn't allow for native 8-bit math (it casts to 32-bit integers whenever you perform an operation which may result in overflow). There are 2 formats offered by SlimDX (YUY2 and Uyvy) which would be ideal, but trying to create one of these textures threw an error on my system (often these texture types can be GPU specific). In the end, the cleanest/quickest solution seemed to be to ship the data to the GPU and decode it there.
What does YUVtoRGB do?
So YUV is a colour encoding which reformats the RGB information in order of visual importance:
- Y = lightness
- U = distance from blue
- V = distance from red
Often we transmit images with full resolution Y, but compressed resolution U and V, which is the case for the BlackMagic Intensity driver. A 4:2:2 pulldown takes every 2 pixels (RGBRGB=6 bytes) and stores it as (UYVY=4 bytes).
This means that 2 pixels are stored in 4 bytes. Since 4 bytes can be stored in a single RGBA pixel, this is what I did coming out of the VideoIn node. This is then decoded in a shader back into RGB.
Since each 'fake RGBA' pixel coming out of VideoIn is turned into 2 real RGB pixels, the resolution coming out of VideoIn is actually half the width of the final image.
- Troubleshooting
As always, plugins might show some bugs when released in the wild. The first thing to do is check that your capture is working correctly in BlackMagic Media Express application. If you can see a live image in that application when in 'Log and Capture' mode (and having the right format set in Preferences) but you can't get it working in VVVV, then please comment below to help debug. If you can't get it working in Media Express, then please look into that first.
- Changelog
v1.1
Features
- Added 'Wait For Frame' options. See help patch for details]
- Added 'Frame Received' output message
Bug fixes
- Fixed exception that sometimes occurred when changing video format.
Comments
Comments are no longer accepted.Please create a new topic in the vvvv beta forum to discuss this contribution.
Which model did you use?
Intensity Pro PCIe I've got an Intensity Shuttle here as well, but that wouldn't work with my USB 3.0 in Media Express, so I didn't try in VVVV.
superr
Great!
Hey Elliot, thank you for this contribution! Are you using a GPUdirect function that copies the captured frames directly to the grphics card, or is it some sort of WDM driver that transfers the data over the CPU to the Graphics card? thx Eno
@eno - i'm going via cpu but the load is very low in testing (partly because of the YUV scenario). I'm using the BlackMagic SDK directly (not via WDM).
I don't know of any way of shipping the pixels direct across the PCIe bus using this framegrabber. Afaik, that is only possible with the Qaudro SDI capture solution*, but i'd certainly love to be proven wrong about that.
(* http://www.nvidia.co.uk/object/product_quadro_sdi_capture_uk.html )
There's ways with other 3rd party sdks and boards to transfer the textures peer-to-peer on the pci bus. Just working on that issue with a DVS board. I also have a Blackmagic SDI card here, so I will check if there's a remarkable difference to your aproach and report here.
super interesting! any references at all?
I can't see any info on DVS site about this specific feature, which product are you using? Is there mention in the manual about this?
interesting: http://visionexperts.blogspot.co.uk/2010/05/faster-memory-transfers.html
NVidia forum post on this topic: http://forums.nvidia.com/index.php?showtopic=155456 but their forums are down right now so can't read it
ok. here it is: http://developer.nvidia.com/nvidia-gpudirect%E2%84%A2-video
BlackMagic are a partner on it, however...
Which basically ties it's usage to proprietary, funded projects (which, since you're using SDI, probably is suitable for your usage).
Furthermore, it appears to only marginally affect latency and CPU usage (whilst using the same 2-copy PCIe buffer usage) since the data has to be shipped via CPU RAM anyway. So if you've got enough cores, then just dispatching the management of this process to another thread will give you almost equivalent performance (minus the development cost and expensive hardware).
I presume you'll gain something on the order of 0.1->1ms latency. However! Since DX9 is single threaded, this latency will also affect your scene render time in VVVV.
I think it's also worth noting that there's irrecoverable latency involved in vertical sync, causing you to always have 1 frame latency from the output device.
The ideal solution is where you can move a surface from GPU in PC A to a surface in GPU in PC B. Almost like an SLI bridge between 2 computers
There's 3 orders of magnitude therefore being presented:
To improve 2, we could use 120Hz from the source (not compatible with BlackMagic Intensity, but more expensive solutions offer this).
edit NB these latencies are all from framegrabber input to VVVV app. Further latency is encountered when you render this data, and when you present this data on a screen or projector
Hi Elliot I've tried to run the Blackmagic SDI card with your plugin with no success. The SDI card is not an intensity model, so it was likely that it wouldnt work ... Anyways here's the error message I get for it:
There's various SDKs from Datapath, DVS, Blackmagic ... that offer these GPUDirect functions in comjunction with the Quadro Cards. We have a sample setup from DVS running here and are looking into the details what it means to port that into a v4 plugin right now. So it would of course be interesting to see the performance gain in direct comparison to your solution ...
Interesting with the GPUDirect is that you can also write back to the card and output SDI again. DVS says they have a minimal latency of 2 frames for in-process-out. (To get access to their SDK you have to regsiter on the site.)
I'll try reach you on skype the next days, maybe we can work something out together.
Do you have an idea how much work it would be to add support for the Blackmagic SDI card? Or to port the approach to more professional cards like datapath or dvs?
@eno - ah ok I presumed that the SDI version would work also with this plugin
So it's complaining that a COM class isn't registered. This might be because the dll's the SDK needs aren't registered.
anyway yes, let's Skype, i'm elliotwoods on that
Hi, I have an intensity shuttle. I am pretty new to vvvv.... The shuttle is recognised and works with media express. The "list devices" node recognises the shuttle. Bust the status says "no device conected". The shuttle works with the classic videoin node, but not with the decklink node. I am very interested in the "low latency" point as I work for theatre and try to be as close as possible to lipsync. Some help would be great, I'm sorry for not being able to give more constructive details Thanx Mehdi
@Mehdi : are you working with the help patch? is ListDevices connected to VideoIn ?
posting screenshots can help if you want to demonstrate errors but don't know which bits to describe
Hi there, got an Intensity Pro PCIe today. Works out of the box (only had to set the right video mode). Nice.
On thing though. How did you determine the latency being 1 Frame?
I used the capture card as second monitor and put one renderer displaying a framecounter on that "screen" and another renderer displaying the captured image and the same counter on the first screen. The difference between the two counters is mostly 2 sometimes 3 frames. The Video mode is set to 1080p30 and vvvv is running at 30FPS. What am I doing wrong?
Unfortunately it´s not possible to attach the patch.
edit: here it is.
The 1 frame latency is from a BlackMagic technician I spoke to about 3 years ago regarding the internal Intensity Pro product. When doing the PC A to PC B test then there's 3 more stages the image must go through before being presented on monitor B:
#Capture from card into app (this is the 1 frame latency quoted by BlackMagic)
We negate step 4, presuming monitor A has the same latency
Step 2 could be marginally improved by forcing each render to wait until a new frame arrives from the capture device. This would lower framerate of course, but improve latency by approximately 50% of a render frame.
I've perhaps been overstating the latency ability of this capture route and haven't done as much serious testing as reading this contribution might imply. I'm mostly comparing it to a DirectShow approach. I've added your notes to the description.
Hi Elliot, here is a screenshot:
I putted a screenshot of the inspektor on videoin because I feel strange that the device input doesn't get filled.
Thank you.
Mehdi
hello Elliot thanks a lot for this contribution i'm running the patch on a quad core win 7 pro and the latest drivers from blackmagic
@keftaparty - i tried to think today about why you might be having troubles, and i can't think of anything yet.
To double check, it definitely works in BlackMagic Media Express with the same settings?
Also please try this:
Just for the record in this thread: This plugin also works with the Decklink SDI Card, but MediaExpresse / The latest Drivers have to be installed.
Has anyone tried to cards yet? That would also be an interesting test...
This plugin is also working with the Decklink Extreme 3D card. It would be awesome to be able to switch between the video inputs without having to do it in the blackmagic settings panel. Great contribution Elliotwoods.
@newemka - if you have the 2 cards, can you try with GetSlice on output of the ListDevices node onto the input of the VideoIn node. This should work in theory but I haven't been able to test so far.
Ah, you mean the actual physical inputs. Hmm..interesting. This would be in the API. I'll have a look when I'm next inside that plugin.
@elliotwoods, we are successfully running two decklink video inputs. One Blackmagic Intensity Shuttle and one Decklink Extreme 3D card. Getslice node on the Listdevice output to select which card to use.
He Elliot,
thanks for that great contribution. did a test today with alpha28. so if i start vvvv normally than the plugin seems to work, but the fx doesnt. and if i start vvvv with its amazing dx9ex extension than even the plugin stops to work.
best,
milo
I don't manage to make it work with vvvv_45beta28.1 the renderer window stay white. Too bad I stay on vvvv_45beta27.2 for now.
any experiences with the intensity shuttle or extreme via thunderbolt on a macbook pro? i'm thinking about buying one of these but first wanted to check out compatibility issues. (or are there usb3 adapters for thunderbolt available already?)
thanks
Hi I am also looking into this, the USB3 version, I ve got a Clevo P150 which sports the NEC usb 3 hardware so I should be able to get it working right, anybody can confirm this? Simone
oh i forgot to mention in /dx9ex mode the texture isn't created. info and anything else behaves like if i didn't connect anything to texture inputs same patch works in normal mode
Driver version : 2.1.39.0 Firmware : 4020
Following http://www.blackmagicdesign.com/media/1388093/USB3_Readme_Before_You_Install.pdf that should be it but I get the damn black screen, HDMI is recognized ..
Help!
Ok seems like it now started to work !! Not sure if just enabling outputs on the control panel did the trick. Beware that you need to set the Media Express window to active to get the refreshing of the preview.
So Clevo P150HM ok!
I can confirm the problem with 28.1 (white screen) the plugin is ok, it is just only the shader giving troubles. Simone
ASUS P8P67 REV 3.0 ---> not working (asmedia chip) ASUS P8P67 REV 3.1 ---> should work (nec chip)
@Whitescreen
try setting the texture format (inside the texture-fx-module) to ARGB – it's RGB right now (or vice versa?) fixed it for me.
got the intensity pro today and it works nearly perfect with my canon eos 7d. thank you elliot!
first i got a white screen out of the texture-fx, so i switched the texture format to 'No Specific' at it works.
if i turn the camera on foto-mode, the quality (iso-noise, resolution and interlacing (in fullscreen)) is much better than in video mode! but i think it even increase the latency by a few frames.
any idea to avoid the interlaced-issue at all?
also nice would be a info about the framerate.
next quest: get magic lantern to run. http://tiny.cc/kyv5mw
+1 for those two issues: interlacing and get it working with /dx9ex
One more issue: it seems like it can t write to 2 heads at the same time, like one head is fine, the other the frame rate is extremely low. DEBUG INFO : setting the /dda to the head which gives problems seems to fix it, still on the offending render I can see a huge spurge of rendering time (in debug mode) at consistent intervals. S.
Nop really, the problem comes back. The system is a hd6970 with a matrox display port and a matrox digital edition with the spatial edge blend node, 4x XGA projectors. help! S.
@ io Try removing Info (EX9.Texture) from YUY2RGB.
Not using the /dx9ex argument at startup seems to make it much better, still it is very jittery, I think I should try to create a Group in Catalyst ATI software, dunno if that may help or not.. Has anybody tried that? Simone
Maybe try setting presentation interval on one of your renders?
Hi tomorrow I ll get to the machine, what does this parameter do exactly? tx Simone
Ive just got it working with the intensity shuttle usb3 on windows8. Thank you for this great plugin!
BTW: bought a first (cheap) USB3 NEC PCIe card, didn t work, then I got a better one (Startek) and now it does work flawlessly, end of story, don t buy cheap stuff. S.
Hi, Any chance to get this node working in dx9ex mode ? That would be great for multiple screens.
Is blackmagic intensity and this node a good idea when you have an installation that will display video input for 6 hours? Can I rely the hardware/software will still run fluid & smooth after 6h? Are there any riscs I should take care of? THANK YOU!
hey all!
i can't really do any development on this right now as i don't have a blackmagic device in a working machine (thanks to Lumacoustics, i have an Intensity Pro PCIe, but my desktop is broken. My laptop is USB3 but i don't have a USB BlackMagic to test with).
@keftaparty - i tried to get it running in dx9ex but it failed. i'll have a chat to vux/others next time i try and implement this to see what i'm doing wrong.
@io - Mehdi Toutain-Lope has been emailing me about this issue (2 heads off 1 node). I can see where the issue is in the code, it's just something stupid/simple to fix. But I can't test right now :(!
Hi Elliot, Keftaparty & Mehdi Toutain-Lopez is the same... sorry for not mentionning my forum name in my emails. Thanks Mehdi
@Elliot - you can probably borrow our USB intensity if you like
Hey guys
Im testing 1 blackmagic pci with 4 outputs (matrox th2go and an extra output) the captures great, but the framerate drops a lot if im in fullscreen with these 2 renderers (800x600+2400x600)
Anyone tried a similar setup ?
If I only use 1 output (for example the matrox) with the decklink, no problem, but if I add it to the other renderer, not so well...
tried wait for frame, mainloop and all i can think of but fps keeps going down when using 2 or more devices to show the decklink stream.
Have anyone tested an Ultrastudio 3D with this plugin?
Apparently it offers 2 streams of 1080p @60fps capture
Is a thunderbolt device
Hi, @Elliot - Any chance you have time to try to fix multiple heads and/or dx9ex issues ? Mehdi
Any chance for a 64-bit Version?
+1 for dx9ex. Would be very handy if one could use it together with ffgl bridge to resolume.
My source video its a sony cx12 1080i connected by HDMI
bmdMode HD1080p2997 works fine, videoimputflagdefault, no interlace, but i have 2 images 1 over the second one like a 3d stream... so im cropping the top one (the top one works 1/2 frame latter than the buton one ),,, this is with mainloop at 60fps so im cropping the button one to get my 1080p video working fine,,, but it will be half resolution on Y axys
the other mode that works its bmdModeHD1080i5994 but with a heavy heavy interlaced effect funny thing,,, Media Express works just fine under the same setting 1080i59.94... no delay, no interlaced.
this is my camera, but i will be using it with 50i cameras,,, 50p works just fine, but with the interlace i have thar issue
on the other hand, i connected the hdmi out of my gx670 to the intensity pro... and i was able to get the image on this mode Hd1080i6000 but with fliks and interlaced.. on 1080p30 i get the same as before, 2 streams one over eachother, and if i crop them, it works fine... but it doest look rigth.... all this is on vvvv
On media Express 1080i60 it seems to works fine... but on 4v this doest work fine,,,
workarround on this ?
im plaing to convert the signal from the video mixer (sdi) to hdmi, put it in a kramer video unit, and reconvert it to any format that vvvv handles rigth
what are the setting that you guys used ?
Pd i made a simple test... my camera pointing at mi screen1, there i have lfo running and a mainloop countinf frames, with a mainloop on 30fps, on the screen2 its the output of the blackmagic,,, i recoreded everything with a go pro hero3 at 240fps to get this frame http://postimg.org/image/wyrh0lqnb/
it seems 4 frames delay, 166ms,,, this is the minimun delay possible ?
Ok after several tests this i what i found... 50i works with a interlaced effect,,, i tested 2 cameras, and its the same i even use a tvone c2-7000 to send specific resolution and test the all... so... workarround 1: set up the blackmagic as hd1080p 25 and stretch the frame. This give me no delay at all, only 3 frames, same as BlackMagic Media Express
workaround 2: manuel gave me the idea of using BOBdeinterlace shader. so I set up the blackmagic in hd1080i50 and tested again... this seems to work, sometimes it add 1 frame delay, but works... but,,, it didt look good
so i took a screenshot of each aproach, the image its croped, but its 1:1 pixel. The camera its a Sony NX5N set up in 1080i/576i... this are the results
3rd and chepest aproach will be to screenshot the BlackMagic media express preview window, i dont have interlaced there.
Hi sorry to bump this up but ... anybody can get the multi head support working? We may crowdfund the job here... Simone
+1 crowdfunding
+1 for dx9ex. it would be fantastic with ffgl bridge to resolume.
hey all
Recent developments on BlackMagic nodes have been sponsored by a particular group. We're still discussing making these improvements public. More news next week when we meet up next.
Expect 64bit, DX9ex, more colour format options, and VideoOut
Elliot
GRANDE GRANDE +1 for VideoOut and DX9ex
since some minutes I need this feature and now I read your post, Elliot.
would be awesome
any chance of dx11 as well ;)
looking forward to this, thank you!
+1 for dx9ex
+1 for videoout :)
is there a sollution for yuv2torgb shader outputting white ?
i'd be really up for doing a crowd-lanced* stage of development on this, where people pitch in requests. (crowd-lanced = freelance + crowd funded. yeah..., i don't think this name is going to take off)
So far I'm thinking:
There's a few features I can't release right now as their development is ongoing with private funding.
If anybody is interested, then please email me directly (my first name)@kimchiandchips.com
I'm in :)
@ggml - did you read julian's comment above about changing the texture format? another reason for white-out might be if you have the /dx9ex flag turned on
@elliot: thought i had. now it works. thank you
julian, elliotwoods I finalllly did it after it changd to texture format 'No specific' of dx9texture node in YUYtoRGB. I had fought with this patch long time to solve this. what different texture format 'RGB88' and 'No specific'. when occurs this?
GTX 660 win7 x64 vvvv 31.2 x86 Intensity pro driver(Desktop Video 9.9.6 for Windows)
I also very appriciate to elliot!
r8g8b8 is not a good choice to specify in a patch thats ought to be portable as it seems only very few graphic cards support this format thats why many fail with this setting.
Hello! Does anybody managed to make it work with b32? 'Frame Received' output message stays on 0
I'm getting a green output on my laptop, (optimus) I've turned off clip device, and set the dx texture to all available options, but it never gives the input source, just various colours depending on the texture format. I have tested in Media Express. Anyone have a solution keftaparty maybe?
@catweasel: I've tried on my optimus laptop, and it works here.
Are you using beta32 ?
If yes, from beta32 dx9ex is the default, and I never managed to use Elliot's Blackmagic contrib in dx9ex mode ( see posts before ).
To have it working with beta32 I had to start with dx9 flag at startup. Also in the YUV2RGB I had to change format in the DX9texture to A8R8G8B8 .
I hope this helps.
Mehdi
Ah! It was beta31.2 but I had to start with /dx9 flag, I thought that was beta32 and later... Brilliant, thanks for your help :)
For anyone reading this, just thought I would mention this plugin is now part of the VVVV.Pack.Image, with working dx9ex.
Hi there,
I am currently testing it with a BlackMagic Decklink 4k Extreme. But I can't get it running with 4k input. Any thoughts/ideas about this? Thanks!
Hello! I don't know why, but I can't get video from blackmagic shuttle USB 3.0. It works well in Express, but in vvvv there is only black screen.
I tried to use Win 7 and 10 and had similar results.
Screenshots: https://yadi.sk/d/ZwcIv6KDrXqD8
Please help me! The project is on fire
@Krementsov If you don't manage to get a picture inside vvvv you could use BlackSpout utility for Blackmagic capture
https://magicmusicvisuals.com/forums/viewtopic.php?f=6&t=201