this particular problem took a lot of tracking down, due to the large numbers of macros in the source. line 393 reads:
DEFINE_IMDCT
and looking for that macro, we find it at line 179. but it has numerous instructions in it, and conditionals, and any one of those could be the culprit. so first what we do is make V=1 to turn on verbose (normal) GNU make output. then we see:
jcomeau@aspire:~/rentacoder/jcomeau/floureon/ffmpeg$ nasm -f elf -DPIC -g -F dwarf -I./ -I.// -Pconfig.asm -I libavcodec/x86/ -o libavcodec/x86/imdct36.o libavcodec/x86/imdct36.asm
we change that to:
jcomeau@aspire:~/rentacoder/jcomeau/floureon/ffmpeg$ nasm -l /tmp/imdct36.lst -f elf -DPIC -g -F dwarf -I./ -I.// -Pconfig.asm -I libavcodec/x86/ -o libavcodec/x86/imdct36.o libavcodec/x86/imdct36.asm
to get a listing file. looking into the listing for the operation size not specified error, we find it at line 102741, at level <4> of the macro expansion. scrolling up, we find the level 2 instruction at line 102668 as extractps [%3 + %4], %1, 1 and the level 1 instruction at line 102583, STORE m6, m7, outq + 16*SBLIMIT, 4*SBLIMIT.
so we go back to libavcodec/x86/imdct36.asm, search for STORE, and find it at line 145. sure enough, we find 3 extractps instructions under it:
extractps [%3 + %4], %1, 1
extractps [%3 + 2*%4], %1, 2
extractps [%3 + 3*%4], %1, 3
we change them to:
extractps dword [%3 + %4], %1, 1
extractps dword [%3 + 2*%4], %1, 2
extractps dword [%3 + 3*%4], %1, 3
assuming 32-bit operands. and sure enough, it finishes the build after that, without having to install yasm.
how do I know it shouldn't be qword instead? I don't, but it doesn't make sense, since extractps only uses 32 bit destinations: "Extract a single-precision floating-point value from xmm2 at the source offset specified by imm8 and store the result to reg or m32. The upper 32 bits of r64 is zeroed if reg is r64." (http://www.felixcloutier.com/x86/EXTRACTPS.html).