I've implemented some support for AMD VCE. Probably all kinds of broken and memory leaky, but atleast something to build on
So...test it and break it.
There are currently 2 3 implementations: with the older OpenVideo encode API (OVE) and with AMD Media SDK (Windows' Media Foundation transform aka MFT). Now also with AMD's AMF(ramework).
If encoder freezes, encoding quality setting is too high for given fps/resolution. Lower video fps or enable custom settings and set QualityVsSpeed at or lower than 50. Also 60 fps capture has some weird memory usage issues.
I have it currently set up with OVE API so that the quality setting (when using "CRF") 10 is best and 0 is "block party".
It isn't exactly CRF, but sets fixed quantizer values. Current formula is 40 - (quality * 5) / 2
So to reiterate: checking 'Use CFR' and 'Use CBR' enables Fixed Quantizers mode (uses Quality Balance), unchecking 'Use CFR' and unchecking 'Use CBR' enables VBR.
(Edit: because I can't read, you should also read 'Use CFR' (as in framerate) as 'Use CRF' (as in bitrate)
)
OVE version can use OpenCL for colorspace conversion (YUV444 to NV12), but it is currently done before passing the frame to be encoded. It adds to encode time and can make OBS drop frames so it is better suited for 30..50 fps recording and low powered CPUs, for now.
For clarification:
D3D10 interop doesn't have anything to do with games, only with OBS as it uses D3D10 to do its magic. Ticking make OBS use CPU for NV12 conversion instead and on Win8+ DX11 engine will use NV12 textures. Dx9 and Win7 resort to using host memory for NV12 buffers.
Device TopoID is the 'physical' location on motherboard, not device vendor ID etc. as you can have more than 1 device with same ID. Check log (window). It only affects OVE because if I'm not missing something, you can't specify device for AMF (yet). But it could be using the device that AMF context got init'ed with. If that's the case then it uses the selected device from Video > Video Adapter.
GPU heavy games can cause frame freezing. You can try disabling D3D10 interop (and with AMF, select "Host" engine type) so that encoder preparation work is done on CPU instead.
Supported hardware from AMD's blog.
Binaries / change log
You may also need to install MS VC++ 2013 runtimes. (Choose by OBS' bitness, not your operating system's.)
AMF: OpenCL interop assumes that the video adapter OBS uses also has VCE support.
Binaries:
2014-11-23 [32bit] [64bit]: Test with threading (AMF only for now).
2014-11-16 N/A. Random reference counting error causing crashing.
2014-11-04 [32bit] [64bit]: Without (most) profiling code. Seemed to help reduce some encoder lag.
2014-10-28 [32bit] [64bit]: Added MinQP/MaxQP setting.
2014-10-25 [32bit] [64bit]: v0.637b+ Optional D3D10+OpenCL interoperability.
2014-09-10 [32bit] [64bit]: Using host memory for surfaces (and looks like dx9 interop). MFT is (temporarily?) replaced with AMF (AMD's media framework).
Too old:
2014-08-31 Use lower quality settings with 1080p@60. OCL "warm-up" routine.
2014-08-10 OVE: manually request keyframes, it always generated every 30 frames. MFT: implement keyframe requesting (AVEncVideoForceKeyFrame)
2014-07-20 When CBR padding is enabled, adds filler NALs. OBS GUI responsiveness test with MFT.
2014-07-03: Don't move scenes' config while vanilla OBS doesn't have scene collections.
2014-07-02: Updated OBS to v0.625b
2014-06-17 #2: Some MFT settings.
2014-06-17 #1: OVE settings dialog. MFT should be less crashy, but may have sync issues :/
2014-05-26
2014-05-19 : without sync fix, 'manually' setting keyframes (VCE seemed to put IDR every 30 frames no matter what)
2014-05-17 : hacky fix for hang when ending streaming
2014-05-12 : OBS streaming fix, automatically selects better quality settings when using lower resolution or fps <= 30
64bit build (2014-08-31)
64bit build (2014-08-10)
64bit build (2014-07-20)
64bit build (2014-07-03)
64bit build (2014-07-02)
Experimental 64bit build (2014-06-17 #2)
Experimental 64bit build (2014-06-17 #1)
Experimental 64bit build (2014-05-12 #1)
