WARNING: Avoid latest AOS updates!

OP
OP
H

hunterallen40

Active Member
View Badges
Joined
Mar 30, 2021
Messages
475
Reaction score
574
Location
Philadelphia
Rating - 0%
0   0   0
Yes it does, I’ve used the statement before and verified it.

@herozero you are correct -- the statement will work under typical circumstances. However, this is an atypical situation: the doser doesn't think it's dosing, so it doesn't turn off when you tell it to.

If the doser was on, and fusion registered it as on, it would turn off with the rule you suggested.
 

herozero

Active Member
View Badges
Joined
Feb 8, 2010
Messages
352
Reaction score
479
Location
USA
Rating - 0%
0   0   0
@herozero you are correct -- the statement will work under typical circumstances. However, this is an atypical situation: the doser doesn't think it's dosing, so it doesn't turn off when you tell it to.

If the doser was on, and fusion registered it as on, it would turn off with the rule you suggested.
I understand the bug might be wrecking the behavior, but the statement works. Sorry this nailed you. So so disappointed in Neptune these days,
 
OP
OP
H

hunterallen40

Active Member
View Badges
Joined
Mar 30, 2021
Messages
475
Reaction score
574
Location
Philadelphia
Rating - 0%
0   0   0
I understand the bug might be wrecking the behavior, but the statement works. Sorry this nailed you. So so disappointed in Neptune these days,

Ah, ok. I think we just misunderstood you. I thought you were suggesting the if statement could be used as a safeguard against the bug.

And thanks, I appreciate the sentiment! These critters we take care of sure are delicate...
 

Sisterlimonpot

Effortless Perfection
View Badges
Joined
Jul 15, 2009
Messages
4,260
Reaction score
8,615
Location
Litchfield Park
Rating - 0%
0   0   0
This is not a request for help, so much as it is a warning for others that may encounter this issue. This is also not an attack on neptune systems. I love(d) my dosers, apex, and their whole kit. I have not had any issues for the three years I have been using their products, and, until recently, would have died on the "I would not run a system without one" hill.

At present, I am 100% convinced there is a devastating software issue that arrived with the update adding support for the new dos QD and trident np, and it has not been fixed as of writing. I would not bother posting this for a minor issue: this has almost crashed my tank TWO TIMES, and I know for a fact that I'm not the only person affected by this issue.

I will explain as best I can the issue, but the short version is this: your doser or other gear may show up in fusion as "off", but continue running, never turning off.
I wouldn't believe me either, but I have plenty of evidence. So, I'll start from the top.

A couple months ago, I got some water into my doser somehow (100% my fault), and one of the dosing heads stopped working. Salt water + electronics don't mix, this is 100% on me. As such, I decided eh, there's a new model coming out in a month. I'll just replace this one with the new one, and put the calcium dosing on my jebao for now. No issues, and just waited for the new doser to arrive.

Once it did, I plugged in the doser and realized my Apex needed an update to be able to operate the new doser (makes sense). So I updated my apex firmware and popped the new doser on. Plug and play, very easy upgrade. No form factor changes either, so the old mounting solution worked fine. Only difference was more dark theme (I am a dark theme enjoyer, so I consider this an upgrade) and silent operation. Other than that, it was the same doser I've known and loved for several years.

Several days after the upgrade, I went down to the basement to check on things, and noticed one of the metal hallide lights was on. They are supposed to turn off at 3 pm, and it was past 3, so they should have been off... Opened fusion to turn off the light, and it showed as off, despite the fact it was on. I shrugged it off, toggled it to on in fusion, and back to auto. It promptly turned the light off, so I just went on with my day, thinking nothing more than that was weird.

A couple of days go by, and business as usual. Then, around 2 pm, I notice my pH has spiked up to ~9! I head down to see what's happening, and, to my shock and horror, the doser is dumping my NaOH based alkalinity solution into my sump, despite the fact that it did not appear to be on in fusion. I could not toggle the doser off, and I was forced to unplug it.

I immediately started recording video of the doser, and took some screenshots to capture the incident as best I could.

Video of the doser running: https://photos.app.goo.gl/CX6SX6vvA3PNTF6g7

and some screenshots proving it thinks it's off and the pH spike from the fact that it isn't:


1717955730752.png
1717955748315.png


In the doser log above, the important thing to note is that there are no registered doses during this duration, including the scheduled ones that would typically happen.

Very angry about the near disaster I averted here, I went to the BRS website and wrote a very negative review. This was an emotional decision, and I think I was justifiably angry here. They responded to this by creating a support ticket for me, and I was largely okay with this.

So I hear from Neptune support, and I sent them the video above, as well as the screenshots above. I wanted to give them as much information as possible to fix this, as it absolutely has the potential to kill an aquarium.

A few days later, I see a post on a facebook group where this happened to a very prominent reefer with a very large aquarium (he can identify himself if he so chooses, but I will not drag him into this without his permission). We'll just say it's someone you would probably know. He posted about some gear failing, and I ranted about this issue there... A few days later, it happens to him, and he has substantially more at risk than I. In a very justifiably angry post, he posted that he got bit by this bug, and there were several other apex users in the comments that also experienced this same issue.

In the interim, I had neptune support look into things, and they didn't see anything wrong. They said one of my alarm statements was incorrect (not sure how i did that, but they were correct about this... It has no relevance here, though). They also noticed the firmware to my old doser was out of date (which, yes... it certainly was. It was also unplugged, and on the floor next to the aquarium, so I don't see any relevance here). The last thing was there was another update to their operating system, so I performed that update as instructed by support. I also removed the module for my old doser, as it was not connected anymore.

I suspected that they fixed the issue hastily, pushed an update, and hoped nobody ran into it. So I just kept on, business as usual...

Then it happened last night. In the most unbelievably dumb-luck type of situation (miraculously so), my dosing line was not completely submerged into my dosing reservoir for the alkalinity. I was especially tired this morning (had a frag swap yesterday... I'm sure some of you know how exhausting that can be), and I slept in. I went down to check on the corals (as I typically do), and saw the doser had the blue light to indicate it was dosing. I've always stared at it when this was happening since the malfunction just to make sure it turned off. Several seconds go by, and my heart sinks. It was happening again, despite everything being up to date and every small, irrelevant configuration issue being "fixed."

This time, though, the doser had been dosing since 2:50 AM, and it was currently 11:40 am. Again, the dumbest of luck here... It fortunately was dosing air. This only damaged the doser itself (it's not supposed to be continuously operating for that long), and it now has a click that happens as it doses.

Again, I took a video (adding a very manual timestamp at the end by showing the time on my watch).


Here's the log, where you can see the exact same thing happenenig (just a longer duration).

1717957071432.png


I am unbelievably lucky that the doser was only able to dose for ~30 minutes before it was in the air. While I personally don't worry that much if there's an isolated alkalinity spike, if I had dosed continuously for 9 hours, I would have lost everything in my aquarium.

I have reached out to neptune support again, to let them know this has happened, but I believe it's time to make other apex users aware of this issue. Please chime in if this has happened to you! The more people that report this, the more likely they will respond.

There are a few of very important takeaways from this that are critical to understand.

1. This is not user error. Everything is programmed correctly, as veryified by neptune support.

2. This is not an isolated incident. It has happened to several individuals, and three times now to me.

3. This affects more than just the new doser. This has happened to my lights as well, so I have no reason to believe it won't effect everything else as well.

4. There is no programming around this, and there are no workarounds. You cannot have a rule that turns off your doser if the pH / alk spikes. I have had to unplug the doser to stop it. I tried as best I could the second time to turn it off in fusion (I knew it was dosing air, so I wasn't too worried about taking my time here), but was not able to do so.

I sincerely hope nobody reads this as "screw you neptune" or anything of the like. I'm a robotics engineer by trade, and I understand things happen. I just hope this makes the issue more visible to others, and I hope anyone who hasn't updated their apex since the release will continue to run their old firmware.

I also sincerely hope this gets fixed quickly, and Neptune stops pushing the latest updates out until it is fixed.

But, until then, I have switched my doser to a more reliable option (my trusty jebao dosers). I got entirely too lucky the first two times, and the law of averages says I'm due for a total whipeout next time this bug happens.
Very well written and fair account of your experience.

I'm curious based on your having the old DOS, did you happen to copy and paste the old TDATA code to the new DOS QD?
 
OP
OP
H

hunterallen40

Active Member
View Badges
Joined
Mar 30, 2021
Messages
475
Reaction score
574
Location
Philadelphia
Rating - 0%
0   0   0
Very well written and fair account of your experience.

Thank you. It was a bit longer than expected, but definitely wanted to get all the details I could there so we can all get this fixed.

I'm curious based on your having the old DOS, did you happen to copy and paste the old TDATA code to the new DOS QD?

Very good question! No, I actually did a fresh configuration. I don't do any tweaking on the dosing times, so the defaults were fine.
 

rtparty

5000 Club Member
View Badges
Joined
Jan 19, 2010
Messages
5,388
Reaction score
9,137
Location
Utah
Rating - 100%
4   0   0
Just to clarify, this started with the original DOS QD update and not the fix they released a couple days later?

Or it is 100% happening on the latest update as well?
 
OP
OP
H

hunterallen40

Active Member
View Badges
Joined
Mar 30, 2021
Messages
475
Reaction score
574
Location
Philadelphia
Rating - 0%
0   0   0
Just to clarify, this started with the original DOS QD update and not the fix they released a couple days later?

Or it is 100% happening on the latest update as well?

It happened on the original update for the first time, and the second time was with the latest firmware / os updates to date. I updated the doser firmware on Saturday (hoping they pushed on update with a fix / safeguard), and the second occurrence of the bug happened at 3 am the following morning.
 

Sisterlimonpot

Effortless Perfection
View Badges
Joined
Jul 15, 2009
Messages
4,260
Reaction score
8,615
Location
Litchfield Park
Rating - 0%
0   0   0
Thank you. It was a bit longer than expected, but definitely wanted to get all the details I could there so we can all get this fixed.



Very good question! No, I actually did a fresh configuration. I don't do any tweaking on the dosing times, so the defaults were fine.
Good to know, with the continuous option for QD make it so you can't copy and paste.

Another question, What version of software is your QD running?

I have one on version 4:


Screenshot_20240610_065622_APEXFusion.jpg


And one on version 5:

Screenshot_20240610_065728_APEXFusion.jpg


But don't know why the other doesn't update.

I don't have these doing anything and probably won't for a while so I can't tell if they function differently based on the version.
 
OP
OP
H

hunterallen40

Active Member
View Badges
Joined
Mar 30, 2021
Messages
475
Reaction score
574
Location
Philadelphia
Rating - 0%
0   0   0

A_Blind_Reefer

Valuable Member
View Badges
Joined
Aug 13, 2019
Messages
2,088
Reaction score
2,772
Rating - 0%
0   0   0
Major bummer! Exactly why I wait for the dust to settle on fw updates on anything. I know you needed to in order to use the new unit, so that’s not a jab at you for doing so (one would think updates were fully vetted prior to release but things happen). Just an fyi, (as I’m super anal about this kind of thing…..although you can’t prevent everything) I personally don’t use the 1Link ports for anything. I use 24v power adaptors and aquabus with controlled outlets. For instance I have three dos units, each on its own 24v power supply, and each plugged into an eb4 outlet. For things like awc where salinity is monitored, alkalinity where ph is monitored, you can program the outlet to kill the dos upon an issue. I do have a peeve with Neptune (not so much on the actual issue, things happen) about communication and transparency. While ive always experienced stellar customer service from Neptune personally, I have noticed a few instances of playing the ostrich with its head in the sand on matters. I know no company like to admit when issues arise, but I think it does a bit of disservice to the customers when issues are cover in a veil of secrecy. This issue is still pretty fresh, but even if a small percentage had issues pulling the update alert/sending out emails to registered accounts (everyone using fusion) to not update or at least not update if you don’t have the new qd or np, would be nice. I’m not holding my breath, as I turned blue waiting for the notice for people that bought head units between date x and date y to go buy a $10 sd card (better yet send out free cards) before your systems crashes.

Edit. I guess I shouldn’t say that I don’t use 1Link ports for anything. I do have wav pumps, but those are all on external 1Links as I’ve had them forever, going back to classic.
 
Last edited:
OP
OP
H

hunterallen40

Active Member
View Badges
Joined
Mar 30, 2021
Messages
475
Reaction score
574
Location
Philadelphia
Rating - 0%
0   0   0
Just an fyi, (as I’m super anal about this kind of thing…..although you can’t prevent everything) I personally don’t use the 1Link ports for anything. I use 24v power adaptors and aquabus with controlled outlets.

You know, I actually have it hooked up with a 24v power adapter, but was saving a port by not sticking it into the EB8 :face-without-mouth:. edit: actually, my doser is plugged in via 24 V adapter. I honestly completely forgot about that. This may actually be a decent workaround for those that have upgraded, but, personally, I'm not plugging that thing in until I know this is fixed.

I know no company like to admit when issues arise, but I think it does a bit of disservice to the customers when issues are cover in a veil of secrecy.

I 100% agree. Customer support did not warn me on the call / give me any reason to believe this issue would happen again. If I was the first call, I 100% understand this. If I was not the first call... Then my entire system was put at risk. I was even told that this should not happen again, and was not given any constructive advice for how to ensure the safety of my aquarium.

It would be really great if someone from Aperture / Neptune could chime in here, as there are many apexes out there, and not everyone avoids updates when not strictly necessary.
 
Last edited:

Charlie the Reefer

Well-Known Member
View Badges
Joined
Mar 29, 2022
Messages
709
Reaction score
856
Location
Chicago
Rating - 100%
1   0   0
Thank you very much to everyone reporting on this issue and thread, to help other reefers out. Really sorry and my condolences for the issues/losses. I have a ton of apex equipment and it's good to know to keep an extra eye on things / avoid the recent update.
 
OP
OP
H

hunterallen40

Active Member
View Badges
Joined
Mar 30, 2021
Messages
475
Reaction score
574
Location
Philadelphia
Rating - 0%
0   0   0
Thank you very much to everyone reporting on this issue and thread, to help other reefers out. Really sorry and my condolences for the issues/losses. I have a ton of apex equipment and it's good to know to keep an extra eye on things / avoid the recent update.

Absolutely! I love neptune products, and still trust them to work on my other aquarium. I just don't want anyone upgrading to this and having something like this happen until we know it's safe.

I'm sure they're all over it.
 

GlassMunky

2500 Club Member
View Badges
Joined
Jan 30, 2014
Messages
3,326
Reaction score
4,409
Location
NJ-Philly Burbs
Rating - 0%
0   0   0
The unstoppable dosing? I haven't seen that in a different thread, but would appreciate it if you linked it!

I have definitely heard several users report getting bricked completely from this one.
no, not the dosing part yet, but a few other reports of bricked head units as you said.

Makes me glad i havnt updated yet, im always hesitant to update anything when what i have is working fine and theres nothing wrong with it as is for me.

Appreciate these posts tho as they help keep people informed
 

LobsterOfJustice

Valuable Member
View Badges
Joined
Dec 3, 2013
Messages
1,324
Reaction score
1,369
Location
Cary, NC
Rating - 0%
0   0   0
Just to clarify, is my system only at risk if I update the Apex and/or DOS firmware? Has anyone reported this failure without making any changes to their system?
 
OP
OP
H

hunterallen40

Active Member
View Badges
Joined
Mar 30, 2021
Messages
475
Reaction score
574
Location
Philadelphia
Rating - 0%
0   0   0
Just to clarify, is my system only at risk if I update the Apex and/or DOS firmware?

Very important clarification! It seems to only effect systems that have updated to the release that enables the new doser to run.

My other aquarium has not been updated, and runs the older model doser. It has been business as usual there.
 

A_Blind_Reefer

Valuable Member
View Badges
Joined
Aug 13, 2019
Messages
2,088
Reaction score
2,772
Rating - 0%
0   0   0
Good to know, with the continuous option for QD make it so you can't copy and paste.

Another question, What version of software is your QD running?

I have one on version 4:


Screenshot_20240610_065622_APEXFusion.jpg


And one on version 5:

Screenshot_20240610_065728_APEXFusion.jpg


But don't know why the other doesn't update.

I don't have these doing anything and probably won't for a while so I can't tell if they function differently based on the version.
Since you’re the inquisitive techie person that you are……..have you opened up (um, never mind….warranty void if removed, I’ll rephrase that), have you seen the insides of the new qd (definitely not one that you just purchased, ha)? I’m very curious to know if one could replace the steppers on an older unit with qd versions. I mean the firmware is the same now. I don’t know how much different the hardware is. I know the water resistant buttons are an improvement, but curious as to if the rest is the same. I haven’t tried programming an old dos for continuous….something for later on today I guess. Didn’t mean to derail here…..mine dos units are loud enough to bother and I’m not willing to upgrade just for noise….yet.
 
OP
OP
H

hunterallen40

Active Member
View Badges
Joined
Mar 30, 2021
Messages
475
Reaction score
574
Location
Philadelphia
Rating - 0%
0   0   0
Since you’re the inquisitive techie person that you are……..have you opened up (um, never mind….warranty void if removed, I’ll rephrase that), have you seen the insides of the new qd (definitely not one that you just purchased, ha)? I’m very curious to know if one could replace the steppers on an older unit with qd versions. I mean the firmware is the same now. I don’t know how much different the hardware is. I know the water resistant buttons are an improvement, but curious as to if the rest is the same. I haven’t tried programming an old dos for continuous….something for later on today I guess. Didn’t mean to derail here…..mine dos units are loud enough to bother and I’m not willing to upgrade just for noise….yet.

Hahaha. No worries on derailing. Probably not wise for we to open it up right now, though...

It's pretty similar in weight (maybe slightly lighter even).
 

dangles

Valuable Member
View Badges
Joined
Sep 22, 2022
Messages
1,226
Reaction score
1,458
Location
Ohio
Rating - 100%
2   0   0
Major bummer! Exactly why I wait for the dust to settle on fw updates on anything. I know you needed to in order to use the new unit, so that’s not a jab at you for doing so (one would think updates were fully vetted prior to release but things happen). Just an fyi, (as I’m super anal about this kind of thing…..although you can’t prevent everything) I personally don’t use the 1Link ports for anything. I use 24v power adaptors and aquabus with controlled outlets. For instance I have three dos units, each on its own 24v power supply, and each plugged into an eb4 outlet. For things like awc where salinity is monitored, alkalinity where ph is monitored, you can program the outlet to kill the dos upon an issue. I do have a peeve with Neptune (not so much on the actual issue, things happen) about communication and transparency. While ive always experienced stellar customer service from Neptune personally, I have noticed a few instances of playing the ostrich with its head in the sand on matters. I know no company like to admit when issues arise, but I think it does a bit of disservice to the customers when issues are cover in a veil of secrecy. This issue is still pretty fresh, but even if a small percentage had issues pulling the update alert/sending out emails to registered accounts (everyone using fusion) to not update or at least not update if you don’t have the new qd or np, would be nice. I’m not holding my breath, as I turned blue waiting for the notice for people that bought head units between date x and date y to go buy a $10 sd card (better yet send out free cards) before your systems crashes.

Edit. I guess I shouldn’t say that I don’t use 1Link ports for anything. I do have wav pumps, but those are all on external 1Links as I’ve had them forever, going back to classic.


Thank you for posting this workaround. $12 on Amazon - 1/3 the price of the one Neptune sells :face-with-rolling-eyes:

Hopefully they resolve this quickly!

And to the OP - Thank you for such a thorough and level-headed post. It's easy for people to lose their minds over things like this.
 

HAVE YOU EVER KEPT A RARE/UNCOMMON FISH, CORAL, OR INVERT? SHOW IT OFF IN THE THREAD!

  • Yes!

    Votes: 32 45.7%
  • Not yet, but I have one that I want to buy in mind!

    Votes: 9 12.9%
  • No.

    Votes: 26 37.1%
  • Other (please explain).

    Votes: 3 4.3%
Back
Top