I got it.
For some reason the operator & has higher precedence than + or -. So the
solution is to put proper parentheses:
function { (y-x) & (-(y+.1)+x) }
Instead of deleting my question, I'll just leave it here for others to
learn from my mistake.
--
main(i,_){for(_?--i,main(i+2,"FhhQHFIJD|FQTITFN]zRFHhhTBFHhhTBFysdB"[i]
):_;i&&_>1;printf("%s",_-70?_&1?"[]":" ":(_=0,"\n")),_/=2);} /*- Warp -*/
