You don't necessarily have to reboot altogether. First things first - get a bigger sump so you can run a skimmer. That should help with nutrient export.
What kind of flow do you have? What's your gal/hr turnover in the tank? What/how many powerhead a do u have?
What's the total water volume? I have a wet dry on my 40 frag tank that holds 5 gal of water with a skimmer in it.. It was under $100 bucks. It depends on what you are ultimately trying to do
Agree. Even a 40g breeder under the tank with a filter sock mod and a skimmer and you'll see almost immediate results. Get the biggest sump you can fit under the tank.
Another help would be a (small even) 50 dollar media reactor with phosphate remover as well. Kent phosphate sponge. Even add some carbon with it. Again you could get a hang on the back unit for space. Big water change, no nitrates, no phosphate, no light for 3 days would do wonders. But you have to prevent the root cause. Which is why I asked parimeters. If your water source contains phosphate that's a losing battle. Also raising magnesium to around 1400 temporarily helps fights algae
One other thing I noticed is that your rock work is laying on top of eachother. Especially if you don't have a refugium or big sump, your rocks are going to play a major roll in growing good bacteria and need good water flow through them. If you add a power head and stack your rocks so water can easily flow around or through them that would really help. I'd also add a 4" block of marine pure to that 5 gallon sump. It's so porus one block would be like doubling your live rock. But all these changes collectively would take a few weeks to start working.
I'm running an aquatic life 115 skimmer in my bio cube 29 and my JBJ 20 frag tank. They need a slight mod to the air intake hose to lengthen it so it doesn't suck water and then they work great ($50). Something like that will fit about anywhere. I also run the marine pure spheres in a mesh bag in both tanks.