Get the loader. I was where you are when I bought my JD 2320. The difference in that payment will not be a factor in the long run. My loader never comes off. May drop the bucket to mow.
Find a set of forks that use the same quick attach (dealer happened to have a used set) & you will never have to "heavy lift" again. I probably use my forks more than the bucket since many of the projects I bought it for are done.
I also have the hoe which does not get as much use now as it did in the beginning, same as above but it stays on except in mowing season.
Also have a box blade which again was used heavily at first, not so much now. I hate the 3 point hitch. In my opinion the worst invention ever!