I pulled the carb off and was checking it out again and the high speed needle tip isn't as sharp as I think it should be and it is bent a little bit. If you adjust the needle to the stock settings and look at it from the float side of the carb through the main jet it is against one side. I think that may be part of the problem I was experiencing with it being rich in the middle and sputtering sometimes. I don't think it is the cause of the backfiring and not wanting to get to full rpm.
I did do the valve clearances before I had put it all back together and I checked that again and they are still within spec. I am going to order a new main jet needle, along with a new head gasket and pull the valves and lap them. Hopefully this will get it running great again. It ran great after I had fixed the governor, for some reason though it just decided to act goofy now so maybe it is carbon keeping one of the valves open.