Hello, great question.

In general, the less stable a leaving group is once booted from the derivative, the less likely it is to actually be booted out! The more stable a leaving group is in solution (once it's been booted) the more likely that substitution will occur.

For example, acid halides have an excellent leaving group, an electronegative chloride, iodide or bromide can bear a negative charge really well. Since these leaving groups are happier than an anhydrides or esters, the halides are more reactive.

Hope this helps!