Before I try my hand at this, is there some distribution that can be expected of the input values? Because if you really want this micro-optimized, any input tendencies should be known so they can possibly be taken advantage of.

So here's what I came up with. My experience with multiplication routines told me that a LUT in this case might be more trouble than it's worth, so I didn't try that. Instead, I mirrored a lot of the logic in Xeda's routine, but added even more case-specific optimizations; namely, handling each of A=64 and A=-64 specially to make them faster and to implement the clamping of 64*512 and -64*-512 to 32767.

It's probably pretty unfair to compare my routine to Xeda's original, because I'm guessing Xeda whipped up her routine in like 10 minutes and didn't optimize for speed that crazily, whereas I took more like 100 minutes, was optimizing for speed crazily, and had her code as a reference. But I'll compare them anyways.

A	Runer	Xeda
A==-64	7*(DE<0)+133	248
-64<A<0	6*bitcount(A)+(9*(A%2))+201	6*bitcount(A)+(9*(A%2))+236
A==0	52	25
A==1	96	136
A==2	112	143
A==3	127	158
0<A<64	16*lg(A)+(6*bitcount(A))+(9*(A%2))+97	16*lg(A)+(6*bitcount(A))+(9*(A%2))+121
A==64	7*(DE<0)+95	223

So about 25-40 cycles faster for most cases and about 100 cycles faster for A=-64 and A=64, but about 25 cycles slower for A=0. And don't forget that this routine also clamps to 32767, whereas the previous did not. So overall, I'd say the speed optimization was quite a success. And with all the speed optimization and the added clamping, I only increased the size from 65 to 89 92 bytes, so not too bad. I just hope it actually works, because I haven't tested it one bit! I've debugged it a fair amount now and it might fully work!

Mul_A_DE_Custom:
;
;Multiplies the signed values A and DE, where A=[-64,64] and DE=[-512,512], and
;returns the result in HL. If A*DE==32768, then the result is clamped to 32767.
;
; I: A=[-64,64], DE=[-512,512]
;
;If A*DE!=32768 && abs(A)!=64
; O: A=A<<7, DE=DE, HL=A*DE
;FO: S=A%2, Z=!A%2, H=?, P/V=?, N=0, C=?
;CC: A<0:  6*bitcount(A)+(9*(A%2))+201
;    A==0: 52
;    A==1: 96
;    A==2: 112
;    A==3: 127
;    A>3:  16*lg(A)+(6*bitcount(A))+(9*(A%2))+97
;
;If A*DE!=32768 && abs(A)==64
; O: A=(A*DE)%256, DE=DE/4, HL=A*DE
;FO: S=S(A*DE), Z=Z(A*DE>>8), H=0, P/V=P(A*DE>>8), N=0, C=0
;CC: A==64:  7*(DE<0)+95
;    A==-64: 7*(DE<0)+133
;
;If A*DE==32768
; O: A=0, DE=256, HL=32767
;FO: S=0, Z=0, H=0, P/V=0, N=0, C=0
;CC: A==64:  77
;    A==-64: 115
;
add a,a
jr c,Mul_A_DE_Custom_ANeg
add a,a
jr c,Mul_A_DE_Custom_A64
jr z,Mul_A_DE_Custom_A0
ld h,d
ld l,e
add a,a
jr c,Mul_A_DE_Custom_A5High
add a,a
jr c,Mul_A_DE_Custom_A4High
add a,a
jr c,Mul_A_DE_Custom_A3High
add a,a
jr c,Mul_A_DE_Custom_A2High
add a,a
ret nc
add hl,hl
ret z
add hl,de
ret

Mul_A_DE_Custom_ANeg64:
ex de,hl
Mul_A_DE_Custom_A64:
sra d
jp m,Mul_A_DE_Custom_A64_DENot512
jr nz,Mul_A_DE_Custom_A64_DE512
Mul_A_DE_Custom_A64_DENot512:
rr e
rra
sra d
rr e
rra
ld h,e
ld l,a
ret

Mul_A_DE_Custom_A64_DE512:
ld hl,32767
ret

Mul_A_DE_Custom_A0:
ld h,a
ld l,a
ret

Mul_A_DE_Custom_ANeg:
ld hl,0
or a
sbc hl,de
add a,a
jr z,Mul_A_DE_Custom_ANeg64
add hl,hl
add a,a
jr nc,$+3
add hl,de
Mul_A_DE_Custom_A5High:
add hl,hl
add a,a
jr nc,$+3
add hl,de
Mul_A_DE_Custom_A4High:
add hl,hl
add a,a
jr nc,$+3
add hl,de
Mul_A_DE_Custom_A3High:
add hl,hl
add a,a
jr nc,$+3
add hl,de
Mul_A_DE_Custom_A2High:
add hl,hl
add a,a
jr nc,$+3
add hl,de
add hl,hl
ret z
add hl,de
ret


---

[9AM EST 6/12] EDIT: Attempted to fix behavior when -64<A<0 by changing jr c,Mul_A_DE_Custom_ANeg64 to jr z,Mul_A_DE_Custom_ANeg64.

[10AM EST 6/12] EDIT 2: Attempted to fix behavior when A=-64.

[11AM EST 6/12] EDIT 3: Last fix failed. Attempted to fix behavior when A=-64 or A=64.

Whoops, I think I see where I went wrong. Try changing jr c,Mul_A_DE_Custom_ANeg64 (third instruction after Mul_A_DE_Custom_ANeg) to jr z,Mul_A_DE_Custom_ANeg64 and see if that fixes it, at least partially.

For all negative A? Or just for A=-64?

Okay, I think I've implemented a fix for the A=-64 behavior. Grab the new code (I made a number of changes, so I'd recommend just re-grabbing the whole thing) and try again?