Corrupt video file makes server 2.4 crash

[Sidonai-1]

I have a corrupt video file encoded in UHD ProRes 4444. The download from Dropbox probably failed and the last seconds of the file are corrupted.

On 2.3.2 with a UHD channel ffmpeg complains, throws an error but server stays alive.

Starting CasparCG Video and Graphics Playout Server 2.3.2 4de6d18f Dev

[2023-11-17 11:40:04.415] [error]   [ffmpeg] [prores @ 000001C698940100] invalid frame header
[2023-11-17 11:40:04.415] [error]   
[2023-11-17 11:40:04.442] [error]   Exception: C:\Program Files (x86)\Jenkins\workspace\casparcg-server-dep\master\src\modules\ffmpeg\producer\av_producer.cpp(153): Throw in function auto __cdecl caspar::ffmpeg::Decoder::()::<lambda_cf087faeeb9dabf84379bbd46f77baec>::operator ()(void) const
[2023-11-17 11:40:04.442] [error]   Dynamic exception type: class boost::exception_detail::clone_impl<struct caspar::ffmpeg::ffmpeg_error_t>
[2023-11-17 11:40:04.442] [error]   [struct boost::errinfo_api_function_ * __ptr64] = avcodec_send_packet(ctx.get(), input.front().get())
[2023-11-17 11:40:04.442] [error]   [struct boost::errinfo_errno_ * __ptr64] = 1094995529, "Unknown error"
[2023-11-17 11:40:04.442] [error]   [struct caspar::ffmpeg::tag_ffmpeg_errn_info * __ptr64] = -1094995529
[2023-11-17 11:40:04.442] [error]   [struct caspar::tag_stacktrace_info * __ptr64] =  0# 0x00007FF6BDEBA37E in casparcg
[2023-11-17 11:40:04.442] [error]    1# 0x00007FF6BDEE9690 in casparcg
[2023-11-17 11:40:04.442] [error]    2# 0x00007FF6BE0F12E0 in casparcg
[2023-11-17 11:40:04.442] [error]    3# 0x00007FF6BE0F4602 in casparcg
[2023-11-17 11:40:04.442] [error]    4# tbb::detail::r1::acquire_writer in tbb12
[2023-11-17 11:40:04.442] [error]    5# tbb::detail::r1::acquire_writer in tbb12
[2023-11-17 11:40:04.442] [error]    6# 0x00007FF6BE0F49E2 in casparcg
[2023-11-17 11:40:04.442] [error]    7# tbb::detail::r1::acquire_writer in tbb12
[2023-11-17 11:40:04.442] [error]    8# tbb::detail::r1::acquire_writer in tbb12
[2023-11-17 11:40:04.442] [error]    9# 0x00007FF6BE0E8CB3 in casparcg
[2023-11-17 11:40:04.442] [error]   10# 0x00007FF6BE0E8A4B in casparcg
[2023-11-17 11:40:04.442] [error]   11# 0x00007FF6BE0F4B5A in casparcg
[2023-11-17 11:40:04.442] [error]   12# tbb::detail::r1::acquire_writer in tbb12
[2023-11-17 11:40:04.442] [error]   13# tbb::detail::r1::acquire_writer in tbb12
[2023-11-17 11:40:04.442] [error]   14# tbb::detail::r1::acquire_writer in tbb12
[2023-11-17 11:40:04.442] [error]   15# tbb::detail::r1::acquire_writer in tbb12
[2023-11-17 11:40:04.442] [error]   16# tbb::detail::r1::acquire_writer in tbb12
[2023-11-17 11:40:04.442] [error]   17# configthreadlocale in ucrtbase
[2023-11-17 11:40:04.442] [error]   18# BaseThreadInitThunk in KERNEL32
[2023-11-17 11:40:04.442] [error]   19# RtlUserThreadStart in ntdll
[2023-11-17 11:40:04.442] [error]   
[2023-11-17 11:40:04.442] [error]   

[2023-11-17 11:40:04.442] [error]    0# 0x00007FF6BDEBA37E in casparcg
[2023-11-17 11:40:04.442] [error]    1# 0x00007FF6BDEB9CAF in casparcg
[2023-11-17 11:40:04.442] [error]    2# 0x00007FF6BE438ED7 in casparcg
[2023-11-17 11:40:04.442] [error]    3# 0x00007FF9C61A1030 in VCRUNTIME140
[2023-11-17 11:40:04.442] [error]    4# is_exception_typeof in VCRUNTIME140
[2023-11-17 11:40:04.442] [error]    5# RtlCaptureContext2 in ntdll
[2023-11-17 11:40:04.442] [error]    6# 0x00007FF6BE0F0825 in casparcg
[2023-11-17 11:40:04.442] [error]    7# 0x00007FF6BE2838E3 in casparcg
[2023-11-17 11:40:04.442] [error]    8# configthreadlocale in ucrtbase
[2023-11-17 11:40:04.442] [error]    9# BaseThreadInitThunk in KERNEL32
[2023-11-17 11:40:04.442] [error]   10# RtlUserThreadStart in ntdll
[2023-11-17 11:40:04.442] [error]   
[2023-11-17 11:40:39.238] [info]    Received message from 127.0.0.1: STOP 1-20\r\n
[2023-11-17 11:40:39.238] [info]    Sent message to 127.0.0.1:202 STOP OK\r\n

On the 2.4 builds that I tested the error makes Caspar crash and it closes.

Starting CasparCG Video and Graphics Playout Server 2.4.0 80f8ee7 Dev

[2023-11-17 11:31:46.937] [error]   [ffmpeg] [prores @ 0000020B673FE500] invalid frame header
[2023-11-17 11:31:46.937] [error]   
[2023-11-17 11:31:46.939] [fatal]   #######################
[2023-11-17 11:31:46.939] [fatal]    UNHANDLED EXCEPTION: 
[2023-11-17 11:31:46.939] [fatal]   Adress:00007FF9CE02CF19
[2023-11-17 11:31:46.939] [fatal]   Code:3765269347
[2023-11-17 11:31:46.939] [fatal]   Flag:1
[2023-11-17 11:31:46.939] [fatal]   Info:0000000FC21FF730
[2023-11-17 11:31:46.939] [fatal]   Continuing execution. 
[2023-11-17 11:31:46.939] [fatal]   #######################

I tried both UHD and a custom video mode in the 2.4 build and the result is the same.

I cannot share the video file publicly at the moment, but I can send it to @Julusian via the forum if needed. Here are the details:

General
Complete name                            : D:\2_TRAILER_RO.mov
Format                                   : MPEG-4
Format profile                           : QuickTime
Codec ID                                 : qt   0000.00.01 (qt  )
File size                                : 15.9 GiB
Duration                                 : 2 min 17 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 993 Mb/s
Frame rate                               : 25.000 FPS
Encoded date                             : 2023-09-26 06:52:16 UTC
Tagged date                              : 2023-09-26 06:52:16 UTC

Video
ID                                       : 1
Format                                   : ProRes
Format version                           : Version 1
Format profile                           : 4444
Codec ID                                 : ap4h
Duration                                 : 2 min 17 s
Bit rate mode                            : Variable
Bit rate                                 : 990 Mb/s
Width                                    : 3 840 pixels
Height                                   : 2 160 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 25.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:4:4
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 4.777
Stream size                              : 15.9 GiB (100%)
Writing library                          : avi1
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 2
Format                                   : PCM
Format settings                          : Little / Signed
Codec ID                                 : in24
Duration                                 : 2 min 17 s
Bit rate mode                            : Constant
Bit rate                                 : 2 304 kb/s
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Bit depth                                : 24 bits
Stream size                              : 37.8 MiB (0%)

Other
ID                                       : 3
Type                                     : Time code
Format                                   : QuickTime TC
Duration                                 : 2 min 17 s
Frame rate                               : 25.000 FPS
Time code of first frame                 : 00:00:00:00
Time code of last frame                  : 00:02:17:12
Time code, stripped                      : Yes

Environment

• Server version: 2.3 vs 2.4
• Operating system: Windows 10

[PeterAkakpo]

i have confirmed that too

[Sidonai-1]

We keep having this problem with some XDCam files, when they reach the end the server crashes.

Also some ProRes 422 files that have been incorrectly downloaded or are corrupt in some other way.

Do we know if this is something related to the ffmpeg version or could this be fixed on the CasparCG side?

This is becoming more of an issue recently.

Thanks.

[Bohoaush]

It is a CasparCG bug introduced when the ffmpeg producer has been rewritten to use multiple threads b657428. Previously any error from ffmpeg would be catched, so the playback would stop, but the server would keep running, now this would crash the server. I currently use a fix like this in production (got the code from #1469, but not the latest iteration). You might get audio sync issues after playing broken part of a file, but that's obviously better than the whole server crashing.

[Sidonai-1]

It is a CasparCG bug introduced when the ffmpeg producer has been rewritten to use multiple threads b657428. Previously any error from ffmpeg would be catched, so the playback would stop, but the server would keep running, now this would crash the server. I currently use a fix like this in production (got the code from #1469, but not the latest iteration). You might get audio sync issues after playing broken part of a file, but that's obviously better than the whole server crashing.

Yeah that'd be much more desireable!

If you think the code is compatible with the current master you could make a pull request with the fix and I can test the next build when it gets approved?

Thanks.

[Bohoaush]

I already tried creating PR all the way in 2022, but current repo maintainer Julusian is hesitant to merge because he lacks deep understanding of ffmpeg part of Caspar (per his words) and is not feeling comfortable about potential impact for other formats.

If you are running Linux, I can give you my build, however I am unable to build for Windows, but you can of course build it yourself.