MrBesen
/
ffmpeg
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

swscale/input: Use unsigned intermediates in rgb64ToUV_c_template 

Large rgb2yuv tables and high pixel values cause the intermediate
int32_t of ru*r + gu*g + bu*b to exceed INT_MAX, which is undefined
behavior. This causes libswscale built with LLVM -fsanitize=undefined to
assert. Using unsigned integers instead has defined behavior and
produces identical results, and makes rgb64ToUV_c_template match
rgb64ToY_c_template.

Fixes: signed integer overflow
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
This commit is contained in:
Jeremy Dorfman 2022-11-15 14:30:23 -05:00 committed by Michael Niedermayer
parent 0871cb9499
commit ce566281f9
No known key found for this signature in database
GPG Key ID: B18E8928B3948D64
1 changed files with 3 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
libswscale/input.c
View File

@ -65,9 +65,9 @@ rgb64ToUV_c_template(uint16_t *dstU, uint16_t *dstV,
int32_t rv = rgb2yuv[RV_IDX], gv = rgb2yuv[GV_IDX], bv = rgb2yuv[BV_IDX];
av_assert1(src1==src2);
for (i = 0; i < width; i++) {
int r_b = input_pixel(&src1[i*4+0]);
int g = input_pixel(&src1[i*4+1]);
int b_r = input_pixel(&src1[i*4+2]);
unsigned int r_b = input_pixel(&src1[i*4+0]);
unsigned int g = input_pixel(&src1[i*4+1]);
unsigned int b_r = input_pixel(&src1[i*4+2]);
dstU[i] = (ru*r + gu*g + bu*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT;
dstV[i] = (rv*r + gv*g + bv*b + (0x10001<<(RGB2YUV_SHIFT-1))) >> RGB2YUV_SHIFT;