reef-pi :: An opensource reef tank controller based on Raspberry Pi.

[Des Westcott]

Just noticed I'm getting these errors. Yet everything seems to be working fine. Any idea what could be causing them?

 

[Ranjib]

I got my robotank installed and started playing over the last few days.

so far just using 3 temp sensors and couple DC outputs for led work light in sump, fan (in temporary while I wait for a hole saw from Amazon) and RO solenoid.

Everything else connects via 2 HS300’s.

Rest is in my build log, but couldn’t resist posting my orange controller to contrast @Ranjib’s green one (green looks slick btw)

Super cool

 

[Ranjib]

Thought I would post my progress on controlling viparspectra lights with moon lights. Big thanks to @Michael Lane for making and providing a board to make things plug and play, more detailed information can be found in the controlling viparspectra thread - https://www.reef2reef.com/threads/controlling-viparspectra-lights.698216/

Here are some pics (vipar lights, they are bright, white @20 and Blues @60-

Moon lights @10%, the appear brighter in the picture than in real life, you can go all the way down to 1% and they are barely on, will use these to ramp up to the vipar and ramp down after the vipar...woot!

This looks awesome

 

[Ranjib]

Just noticed I'm getting these errors. Yet everything seems to be working fine. Any idea what could be causing them?

How’s the cpu utilization chart ?

 

[Schreiber]

At the moment, reef-pi only has a driver for ADS1119, through the ph-board implementation. I don't think the ADS1119 is readily available on breakout boards, so that's unlikely the be helpful.

On the other hand, I plan to have a driver for ADS1015/ADS1115 ready next month. Those ADC chips are pretty common on breakout boards and should be able to support the TDS module you linked.

I actually got one of these TDS sensors on a whim in an order from DFRobot a few weeks ago. Haven't had the time to play with it yet, but let me know if there's anything you'd like me to do or test!

 

[bishoptf]

Starting to get a little frustrated, grumble, grumble....Looks like my existing reef-pi unit that has been running for a year plus corrupted the SD card, I really don't like SD cards. I took this as an opportunity to rebuild with the latest version, easy enough and I got it back up an working, using 3.4. Either I have run into a bug or I am doing something wrong, but I get a portion of the way through configuring and if I reboot the service will not start up, here are the journalctl lines:

Started raspberry pi based reef tank controller.
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 driver-subsystem: registering driver id: rpi Name: Raspberry Pi
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 INFO: equipment subsystem: Finished syncing all equipment
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Successfully started subsystem: equipment
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Successfully started subsystem: temperature
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Successfully started subsystem: lightings
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Successfully started subsystem: macro
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Successfully started subsystem: timers
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Successfully started subsystem: system
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 reef-pi is up and running
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Starting https server at: 0.0.0.0:443
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Starting health checker
May 28 22:04:36 tankpi reef-pi[1147]: Methods: [GET]
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 API Doc path: /api/settings

May 28 22:05:36 tankpi reef-pi[1147]: 2020/05/28 22:05:36 Reading temperature from device: 28-011938443c3c
May 28 22:05:36 tankpi reef-pi[1147]: 2020/05/28 22:05:36 Reading temperature from device: 28-00000a9aeb40
May 28 22:05:36 tankpi reef-pi[1147]: 2020/05/28 22:05:36 health check: Used memory: 11.42  Load5: 0.24
May 28 22:05:37 tankpi reef-pi[1147]: 2020/05/28 22:05:37 temperature sub-system:  sensor 10g-green value: 78.24
May 28 22:05:37 tankpi reef-pi[1147]: panic: descriptor Desc{fqName: "10g_green_reading", help: "Module:10g-green Item:reading", constLabels: {}, variableLabels: []} is invalid: "10g_green_reading" is not a valid metric name
May 28 22:05:37 tankpi reef-pi[1147]: goroutine 44 [running]:
May 28 22:05:37 tankpi reef-pi[1147]: github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0x2176380, 0x200fd18, 0x1, 0x1)
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:400 +0x80
May 28 22:05:37 tankpi reef-pi[1147]: github.com/prometheus/client_golang/prometheus/promauto.Factory.NewGauge(0x767ce0, 0x2176380, 0x0, 0x0, 0x0, 0x0, 0x202bd20, 0x11, 0x202bd40, 0x1d, ...)
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/pkg/mod/github.com/prometheus/[email protected]/prometheus/promauto/auto.go:295 +0xc0
May 28 22:05:37 tankpi reef-pi[1147]: github.com/prometheus/client_golang/prometheus/promauto.NewGauge(...)
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/pkg/mod/github.com/prometheus/[email protected]/prometheus/promauto/auto.go:190
May 28 22:05:37 tankpi reef-pi[1147]: github.com/reef-pi/reef-pi/controller/telemetry.(*telemetry).EmitMetric(0x21b8180, 0x21a19c0, 0x9, 0x6742ce, 0x7, 0x28f5c28f, 0x40538f5c)
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/src/github.com/reef-pi/reef-pi/controller/telemetry/stats.go:175 +0x464
May 28 22:05:37 tankpi reef-pi[1147]: github.com/reef-pi/reef-pi/controller/modules/temperature.(*Controller).Check(0x2198840, 0x2197200)
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/src/github.com/reef-pi/reef-pi/controller/modules/temperature/control.go:30 +0x3dc
May 28 22:05:37 tankpi reef-pi[1147]: github.com/reef-pi/reef-pi/controller/modules/temperature.(*Controller).Run(0x2198840, 0x2197200, 0x21acd80)
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/src/github.com/reef-pi/reef-pi/controller/modules/temperature/tc.go:191 +0xc8
May 28 22:05:37 tankpi reef-pi[1147]: created by github.com/reef-pi/reef-pi/controller/modules/temperature.(*Controller).Start
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/src/github.com/reef-pi/reef-pi/controller/modules/temperature/controller.go:71 +0x1c4
May 28 22:05:37 tankpi systemd[1]: reef-pi.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 28 22:05:37 tankpi systemd[1]: reef-pi.service: Failed with result 'exit-code'.
May 28 22:07:07 tankpi systemd[1]: reef-pi.service: Service RestartSec=1min 30s expired, scheduling restart.
May 28 22:07:07 tankpi systemd[1]: reef-pi.service: Scheduled restart job, restart counter is at 9.

I have enable prometheus and haven't tried turning that off but something is amiss here, I can always drop back to 3.3 but I see a panic and an invalid argument error messages. It will not start up, I thought it was my naming convention or something but I think something is amiss in 3.4 that's not very nice.

Let me know if you need additional information, I will delete the database once again and see if I can figure out what it doesn't like.



UPDATE: I've narrowed it down to the temp sensor readings, it crashes right after trying to read the temp value, doesn't make a difference which temp senor is enable it crashes after trying to read the sensor:

2020/05/28 23:59:45 Reading temperature from device: 28-011938443c3c
May 28 23:59:46 tankpi reef-pi[695]: 2020/05/28 23:59:46 temperature sub-system:  sensor 10g value: 78.24
May 28 23:59:46 tankpi reef-pi[695]: panic: descriptor Desc{fqName: "10g_reading", help: "Module:10g Item:reading", constLabels: {}, variableLabels: []} is invalid: "10g_reading" is not a valid metric name

If I disable the temp sensor it stays up...what am I missing? I've enabled 1-wire and I2C in raspi-config, and it sees the sensors but is crashing non the less. If anyone has any idea's let me know.

Thanks

 

[peejay]

That’s not a deliberate move. We updated the charting library to address some deprecation warning, I’ll cross check .

Hi,
Did you manage to find anything related to my issue? I have re-installed Reef-Pi twice now, the last time on a new card with fresh install of Raspbian but I still get the same effect, no red bar for "Off" but green bar for "On" as normal. Also, I have a temperature graph on my dashboard, it no longer updates until I hit Reload so data is there but no updating on the graph.
Thanks.

 

[robsworld78]

That's the weird thing. When you leave them bare metal (contact with water) is when the issue happens. The one covered in silicone seems to be working so far.


Might try grounding probe. Would a simple wire from ground in the outlet to the sump work for test purposes?

Yeah that would be ok.

Also check if the shielding is connected to the metal end and if ground and shield are connected.

 

[Des Westcott]

How’s the cpu utilization chart ?

To be perfectly honest, I turned them off. I've enabled them again and will monitor.

 

[oreo54]

Starting to get a little frustrated, grumble, grumble....Looks like my existing reef-pi unit that has been running for a year plus corrupted the SD card, I really don't like SD cards.

Thanks

SD cards aren't hard drives.. huge weakness in the system..
Too many read/write cycles..

 

[Yov]

So i have another question... In a Paludarium i would need the humidity to be 70-80 during the day and 100 during the night...

So during the day i would set the humidity control with the PH settings : Upper treshold would activate a fan when i reach 80. Lower treshold wil deactivate the fan or wil activate a humidifier when it reach 70. hysteresis 2-3.. (i have to experiment with this)

But when the night fals i need to get other settings. to reach 100% humidity. How can i approach this? do i look at Macro's ?

 

[Badfish2too]

SD cards aren't hard drives.. huge weakness in the system..
Too many read/write cycles..

Well really it only experiences "wear" when it is turned off and on really. When it is left running, it is all in the RAM. A lot of major companies rely on flash storage for the OS of virtual machine hypervisors. Myself included. You could always set up a SSD or external drive and use that instead

 

[Ranjib]

Starting to get a little frustrated, grumble, grumble....Looks like my existing reef-pi unit that has been running for a year plus corrupted the SD card, I really don't like SD cards. I took this as an opportunity to rebuild with the latest version, easy enough and I got it back up an working, using 3.4. Either I have run into a bug or I am doing something wrong, but I get a portion of the way through configuring and if I reboot the service will not start up, here are the journalctl lines:

Started raspberry pi based reef tank controller.
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 driver-subsystem: registering driver id: rpi Name: Raspberry Pi
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 INFO: equipment subsystem: Finished syncing all equipment
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Successfully started subsystem: equipment
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Successfully started subsystem: temperature
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Successfully started subsystem: lightings
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Successfully started subsystem: macro
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Successfully started subsystem: timers
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Successfully started subsystem: system
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 reef-pi is up and running
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Starting https server at: 0.0.0.0:443
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 Starting health checker
May 28 22:04:36 tankpi reef-pi[1147]: Methods: [GET]
May 28 22:04:36 tankpi reef-pi[1147]: 2020/05/28 22:04:36 API Doc path: /api/settings

May 28 22:05:36 tankpi reef-pi[1147]: 2020/05/28 22:05:36 Reading temperature from device: 28-011938443c3c
May 28 22:05:36 tankpi reef-pi[1147]: 2020/05/28 22:05:36 Reading temperature from device: 28-00000a9aeb40
May 28 22:05:36 tankpi reef-pi[1147]: 2020/05/28 22:05:36 health check: Used memory: 11.42  Load5: 0.24
May 28 22:05:37 tankpi reef-pi[1147]: 2020/05/28 22:05:37 temperature sub-system:  sensor 10g-green value: 78.24
May 28 22:05:37 tankpi reef-pi[1147]: panic: descriptor Desc{fqName: "10g_green_reading", help: "Module:10g-green Item:reading", constLabels: {}, variableLabels: []} is invalid: "10g_green_reading" is not a valid metric name
May 28 22:05:37 tankpi reef-pi[1147]: goroutine 44 [running]:
May 28 22:05:37 tankpi reef-pi[1147]: github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0x2176380, 0x200fd18, 0x1, 0x1)
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/pkg/mod/github.com/prometheus/[email protected]/prometheus/registry.go:400 +0x80
May 28 22:05:37 tankpi reef-pi[1147]: github.com/prometheus/client_golang/prometheus/promauto.Factory.NewGauge(0x767ce0, 0x2176380, 0x0, 0x0, 0x0, 0x0, 0x202bd20, 0x11, 0x202bd40, 0x1d, ...)
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/pkg/mod/github.com/prometheus/[email protected]/prometheus/promauto/auto.go:295 +0xc0
May 28 22:05:37 tankpi reef-pi[1147]: github.com/prometheus/client_golang/prometheus/promauto.NewGauge(...)
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/pkg/mod/github.com/prometheus/[email protected]/prometheus/promauto/auto.go:190
May 28 22:05:37 tankpi reef-pi[1147]: github.com/reef-pi/reef-pi/controller/telemetry.(*telemetry).EmitMetric(0x21b8180, 0x21a19c0, 0x9, 0x6742ce, 0x7, 0x28f5c28f, 0x40538f5c)
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/src/github.com/reef-pi/reef-pi/controller/telemetry/stats.go:175 +0x464
May 28 22:05:37 tankpi reef-pi[1147]: github.com/reef-pi/reef-pi/controller/modules/temperature.(*Controller).Check(0x2198840, 0x2197200)
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/src/github.com/reef-pi/reef-pi/controller/modules/temperature/control.go:30 +0x3dc
May 28 22:05:37 tankpi reef-pi[1147]: github.com/reef-pi/reef-pi/controller/modules/temperature.(*Controller).Run(0x2198840, 0x2197200, 0x21acd80)
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/src/github.com/reef-pi/reef-pi/controller/modules/temperature/tc.go:191 +0xc8
May 28 22:05:37 tankpi reef-pi[1147]: created by github.com/reef-pi/reef-pi/controller/modules/temperature.(*Controller).Start
May 28 22:05:37 tankpi reef-pi[1147]:         /home/ranjib/gocode/src/github.com/reef-pi/reef-pi/controller/modules/temperature/controller.go:71 +0x1c4
May 28 22:05:37 tankpi systemd[1]: reef-pi.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 28 22:05:37 tankpi systemd[1]: reef-pi.service: Failed with result 'exit-code'.
May 28 22:07:07 tankpi systemd[1]: reef-pi.service: Service RestartSec=1min 30s expired, scheduling restart.
May 28 22:07:07 tankpi systemd[1]: reef-pi.service: Scheduled restart job, restart counter is at 9.

I have enable prometheus and haven't tried turning that off but something is amiss here, I can always drop back to 3.3 but I see a panic and an invalid argument error messages. It will not start up, I thought it was my naming convention or something but I think something is amiss in 3.4 that's not very nice.

Let me know if you need additional information, I will delete the database once again and see if I can figure out what it doesn't like.



UPDATE: I've narrowed it down to the temp sensor readings, it crashes right after trying to read the temp value, doesn't make a difference which temp senor is enable it crashes after trying to read the sensor:

2020/05/28 23:59:45 Reading temperature from device: 28-011938443c3c
May 28 23:59:46 tankpi reef-pi[695]: 2020/05/28 23:59:46 temperature sub-system:  sensor 10g value: 78.24
May 28 23:59:46 tankpi reef-pi[695]: panic: descriptor Desc{fqName: "10g_reading", help: "Module:10g Item:reading", constLabels: {}, variableLabels: []} is invalid: "10g_reading" is not a valid metric name

If I disable the temp sensor it stays up...what am I missing? I've enabled 1-wire and I2C in raspi-config, and it sees the sensors but is crashing non the less. If anyone has any idea's let me know.

Thanks

We updated the prometheus library in 3.4. It looks like the new update does not allow metric name starting with number and thus "10g" is causing it to panic (i am assuming you have a temperature controller with name 10g). For the time being changing it to something else that does not starts with a number might fix it (e,g. TenG or IOg) or just disable prometheus from the preferences. Meanwhile I'll check if we can prevent this in reef-pi side , i.e check if the metric name is valid or not, and error instead of panic if its invalid.

 

[Ranjib]

Hi,
Did you manage to find anything related to my issue? I have re-installed Reef-Pi twice now, the last time on a new card with fresh install of Raspbian but I still get the same effect, no red bar for "Off" but green bar for "On" as normal. Also, I have a temperature graph on my dashboard, it no longer updates until I hit Reload so data is there but no updating on the graph.
Thanks.

I tried to reproduce this but was not able to. My equipment chart looks correct. Can you share you equipment tab and dashboard screenshot? If you think this is always happening, then please file a github bug with all the details.

 

[oreo54]

Well really it only experiences "wear" when it is turned off and on really. When it is left running, it is all in the RAM. A lot of major companies rely on flash storage for the OS of virtual machine hypervisors. Myself included. You could always set up a SSD or external drive and use that instead

you may be right but some people don't feel quite that secure w/ sd..

"When I was scrounging in the guts of Swift itself, I was noticing how many folks were burning out SD cards doing these big expensive builds (with swap files) instead of using a USB drive for the kind for stuff that thrashes the SD card. Faster too.

My dev board has a pocket USB drive hooked up to it for that reason. "

The fragility of SD cards is the weak link in the Raspberry Pi ecosystem. Most of us seem to have at least one Pi tucked away somewhere, running a Magic Mirror, driving security cameras, or even taking care of a media library. But chances are, that Pi is writing lots and lots of log files. Logging is good — it helps when tracking down issues — but uncontrolled logging can lead to problems down the road with the Pi’s SD card.

Give Your Raspberry Pi SD Card A Break: Log To RAM

The fragility of SD cards is the weak link in the Raspberry Pi ecosystem. Most of us seem to have at least one Pi tucked away somewhere, running a Magic Mirror, driving security cameras, or even ta…

hackaday.com

Guess it depends...

Consider it just an FYI atm..

Log2Ram: Extending SD Card Lifetime for Raspberry Pi LoRaWAN Gateway

My LoRaWAN gateway (“Contributing an IoT LoRaWAN Raspberry Pi RAK831 Gateway to The Things Network” is running and working great now for more than a month and it already has transmitted…

mcuoneclipse.com

 

[Ranjib]

Well really it only experiences "wear" when it is turned off and on really. When it is left running, it is all in the RAM. A lot of major companies rely on flash storage for the OS of virtual machine hypervisors. Myself included. You could always set up a SSD or external drive and use that instead

All write operations in reef-pi is written on disk. This include all types of creation, deletion, update (equipment, connector, ato , ph etc). Everytime an equipment is switched on or off, that too is written on disk (so that reef-pi can restore the equipment in expected state upon startup). All usage data (temperature readings, heater/cooler usage etc) are written on disk once every hour.
We did some optimizations (the usage data is stored in-memory ring buffer) to reduce the disk write scenario.. but still we have some disk write operations.

 

[Ranjib]

So i have another question... In a Paludarium i would need the humidity to be 70-80 during the day and 100 during the night...

So during the day i would set the humidity control with the PH settings : Upper treshold would activate a fan when i reach 80. Lower treshold wil deactivate the fan or wil activate a humidifier when it reach 70. hysteresis 2-3.. (i have to experiment with this)

But when the night fals i need to get other settings. to reach 100% humidity. How can i approach this? do i look at Macro's ?

Yes, create two separate ph controller with specific setting and use a macro to turn one off, wait for few second and turn the other on. Make the two macros to turn them on/off at specific time. Then use two timers with those two target macros.
Let me know how it goes,,,

 

[Badfish2too]

All write operations in reef-pi is written on disk. This include all types of creation, deletion, update (equipment, connector, ato , ph etc). Everytime an equipment is switched on or off, that too is written on disk (so that reef-pi can restore the equipment in expected state upon startup). All usage data (temperature readings, heater/cooler usage etc) are written on disk once every hour.
We did some optimizations (the usage data is stored in-memory ring buffer) to reduce the disk write scenario.. but still we have some disk write operations.

But like you said minimal writes. I have seen cheap SD cards fail. But in my experience, its pretty rare for one to fail unless it is constantly reading and writing large files. I have never had one fail on me, but they are mostly relegated to cell phones, reefpi, and my servers for the hypervisor. Which really never writes

 

[Ranjib]

To be perfectly honest, I turned them off. I've enabled them again and will monitor.

that interrupt error is rare but a possible scenario that can happen, when linux kernel intervenes the sensor file reading operations. The kernel does so due to the reading process taking longer than expected. Its rare, and we can add logic to check this and retry (which is recommended). But this is the first time im learning about this error. We never really write such failure handling logic (even in profession coding work), because most often it indicates something is not working correct, i.e. the kernel is getting busy in something else, or the sensor /device management system itself is encountering some issues. Keep us posted, im curious,

 

[Ranjib]

But like you said minimal writes. I have seen cheap SD cards fail. But in my experience, its pretty rare for one to fail unless it is constantly reading and writing large files. I have never had one fail on me, but they are mostly relegated to cell phones, reefpi, and my servers for the hypervisor. Which really never writes

I would agree to that. Im using samsung evo, and most of them running fine for more than two years. Although almost all my cheaper sandisk cards did failed soon after first year of their usage. After that i aggressively updated all builds with samsung cards. The whole in-memroy ring buffer optimization came out of that experience. I also keep the reef-pi db copy in google drive for backup.