Victron MPPT data to NodeRed via external SSL connection using Arduino (NodeMcu)

09.11.2021

So typically I'm fairly good in R&D (RipOff and Duplicate) but this project got me stumped for weeks. And the fun part is that basics were working fairly quick, thanks to the gentlemen sailor Âld Skânser, whom already cracked the Victron MPPT secrets. Transporting the provided code to NodeMcu was fairly simple. Only snag was that NodeMcu has only single serial to do the reading from MPPT controller, so needed to bring into picture also software serial to be able at the same time to see the output in Arduino IDE serial monitor for debugging. Simple so far.

The good documentation for the MPPT also provide fairly easily how to flush & read the serial output and put it in common sense format. Fetching data and connecting it to the local network MQTT server data was breeze as NodeMcu has it own ESP8266 embedded into it.

So my Solar Panels will be in our Summer Cottage which is on a island. So to be able to read the data I would need to transport it via Internet to my home server where I have Mosquitto and NodeRed. I selected to most cheapest 4G modem/Wifi ZTE mf93d which I found from used gadget sales place (20 euros), equipped it with a SIM card made for IoT device (4 euros / month). Again configuring this, not a problem. Idea is to send data every 10 minutes (accurate enough) to preserve solar panel batteries.

Challenge starts

As I'm mid level paranoid on exposing my Node Red server external I really wanted to make the connection work on TSL/SSL with my Mosquitto. This is the challenge. Creating the certificates (You find tens of dozens instructions on it) is not complicated follow the instruction and bam...your 90% sure to get it working in your own LAN... but transporting it to Arduino and get it working via External connection...Mission Impossible..almost.

So copy paste, start with simple....argh... not working ...

Things that I learned.

1. Create a test server with Oracle VM VirtualBox as you reaaaaaly do not want to be rebooting / changing your live server settings frequently if you have anything "critical" on it, like I hade lights, air condition, Oil heating follow up, Robot vacuum, dozens of temp & humidity sensor etc. At one point as things were failing all together and reading my wife's body language I needed not to be rocket scientist and agree... I needed to STOP. (Setup of test server will not be explained here)

2. The outside world needs correctly working Port Forwarding testing tool / webpage so make sure you get that working so you are fixing the correct thing. The part I learned that testing a port (even though open , in my case via PFsense) is not enough for the port checker. You need to have properly configured Mosquitto server running at the time you are testing, otherwise it will report closed even the actual Port Forwarding has been done correctly. (Setup of Port forwarding will not be explained here)

3. Arduino is not really well equipped to handle TSL/SSL required certificates. Its seems whole internet audience is following this one guy Debashish Sahu and everything you see people posting & showing their working code is based on this guys work ( me including :). But again the code he is showing is fairly complex so digging out what really you only need is the trick. And as He points out he is Busy, I did not dare to ask help... hundreds of other were /are not so discreet. And hey I had all winter time to get my project working before next summer and usage of the setup.

4. Mosquitto does not by default log all your problems. You need to activate this in you mosquitto.conf (log_type all). This will start to give you more understanding why a connection might be failing.

5. Arduino IDE is not debugging and showing all data by default. You find under the actual Board information two things you need to activate the debugging to get even more information why the connection is failing

So at this part I was stuck for three weeks and getting fairly desperate. What was failing and why? Without certificates everything was working, but activating the certificates nothing was working and errors were all around. Chasing the white rabbit I googled almost all errors that MQTT connection can throw up and the one thing similar on these chases were that actually NO FIXES were given so assuming a lot of people are having the same issue.

So desperate I was that I even started reading the actual instruction and understanding how the BearSSL was working and what Ciphers (RSA or elliptical) and TLS (1.2) It can only use. Bit by Bit I started to come to conclusion that my problem was about the Certificate. As the famous debsahu was using AWS with real external certificates and not the OpenSSL X509 self signed certificates I could not really learn from him. So my focus went to trying to figure out why the same certificate that worked in internal LAN was not working on the external. Many sites were referring that the CN (or FQDN) needs to be correct for the connection to work. But no one (hundreds of samples I read) told that the FQDN needs to be the name that the outside world sees&connects when using external connections when you are using the certificate on external connections.

6. When creating The Fuc=&%#¤¤"ing Certificates, The CN (or FQDN) needs to be the external address that you use.

And tadaa.... by miracle (no shit...hard work I say) it started to work!

Sorry for the long post and no potatoes this time here are the instructions.

How to create a working SSL certificate (RSA cipher)

Use sudo su and work in the /etc/mosquitto/certs folder
openssl genrsa -out ca.key 2048
openssl req -new -x509 -days 3600 -key ca.key -out ca.crt
openssl genrsa -out server.key 2048
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 3600
openssl req -new -out server.csr -key server.key

When asked on the certificate details use the computer/server name if you are using it in internal network (though I can not think any reason why you would use certificates in own trusted LAN...but hey...).
IMPORTANT !!! If using it on external connections use the DDNS address for CN detail part.
Some instruction say that the CA.CRT and SERVER.CRT details need to be slight different to be working...I did not test this just followed their suggestion.

How to read the certificate info in ubuntu for the arduino code (and the copy past it to arduino code)

cat ca.crt (for the actual certificate. I would really only use this option)
openssl x509 -pubkey -noout -in ca.crt (if using the public key)
openssl x509 -fingerprint -in server.crt (if using the fingerprint of the server which is the most unsecure of these options)

How to setup Mosquitto so it can read certificates on your Ubuntu server (18.04 in my case)

as you work as sudo su to create the certificates created in the cert folder, they can be only access by root. You need to change the access rights to mosquitto to be the owner. The cert folder should have 0700 right and the actual cert files 0600 rights. I used Webmin tool to change them as I'm in my hart windows guy and not Linux...sorry

If rights are not properly set the mosquitto will not start.

The mosquitto.conf needs to have minimum these setting to be able to read two ports 1883 for internal MQTT messages without certificates and 8883 for external port for messages using certificates. log_type all can be removed once all debugging is done and system works as should.

# Place your local configuration in /etc/mosquitto/conf.d/

#

# A full description of the configuration file is at

# /usr/share/doc/mosquitto/examples/mosquitto.conf.example

persistence true

persistence_location /var/lib/mosquitto/

log_dest file /var/log/mosquitto/mosquitto.log

include_dir /etc/mosquitto/conf.d

per_listener_settings true

log_type all

listener 1883

allow_anonymous true

# jotain tahan valiin

listener 8883

cafile /etc/mosquitto/certs/ca.crt

keyfile /etc/mosquitto/certs/server.key

certfile /etc/mosquitto/certs/server.crt

tls_version tlsv1.2

allow_anonymous true

The working NodeMcu code

As using certificates and BearSSL is heavy on the hardware it is recommended to switch the default 80 Mhz to 160 Mhz clock speed for NodeMcu. The code has two pages (tabs); the main code and secrets.h (where certificate data is copied...not really sure why this setup and too afraid to ask debsahu :)

APOLOGIES HOW THE WEBPAGEGENERATOR FORMATS THE CODE !!

// MAIN CODE

#include <ESP8266WiFi.h>

#include <WiFiClientSecure.h>

#include <time.h>

#include <PubSubClient.h>

#include <SoftwareSerial.h>

SoftwareSerial mySerial(D6, D5); // RX, TX

#define WIFI_AP "XXX"

#define WIFI_PASSWORD "XXXX"

#define SLEEP_LENGTH 600

int status = WL_IDLE_STATUS;

int aika = 0;

unsigned long lastSend;

String label, val;

float B_VOL = 0.0, B_VOL2 = 0.0;

float B_CUR = 0.0, B_CUR2 = 0.0;

float P_VOL = 0.0, P_VOL2 = 0.0;

float P_POW = 0.0, P_POW2 = 0.0;

float Y_YD = 0.0, Y_YD2 = 0.0;

float Y_MPOW = 0.0, Y_MPOW2 = 0.0;

float VIC_ERR = 0, VIC_ERR2 = 0;

float VIC_CS = 0, VIC_CS2 = 0;

char buf[44];

int i[9];

int iset = 8;

//enable only one of these below,. Keep it simple use only the CHECK_CA_ROOT option.

#define CHECK_CA_ROOT

//#define CHECK_PUB_KEY

//#define CHECK_FINGERPRINT

////--------------------------////

#include "secrets.h"

#ifndef SECRET

const char MQTT_HOST[] = "XXXX"; // This adress name should be the same as the one you put in CN/FQDN in certificates

const int MQTT_PORT = 8883;

const char MQTT_USER[] = ""; // leave blank if no credentials used

const char MQTT_PASS[] = ""; // leave blank if no credentials used

#ifdef CHECK_CA_ROOT

static const char digicert[] PROGMEM = R"EOF(

-----BEGIN CERTIFICATE----- // Puth here your CA.CRT info !! (do not worry I removed enough lines to keep my key secure !)

MIIEATCCAumgAwIBAgIUEW0vuEMnLk1DJpVf6GCoh3bRLoQwDQYJKoZIhvcNAQEL

BQAwgY8xCzAJBgNVBAYTAkZJMRMwEQYDVQQIDApTb21lLVN0YXRlMREwDwYDVQQH

DAhOQUFOVEFMSTEPMA0GA1UECgwGU0FNT05FMQowCAYDVQQLDAExMRgwFgYDVQQD

DA9zYW1vbmUuZHludS5jb20xITAfBgkqhkiG9w0BCQEWEnNvaXNraTcxQGdtYWls

BgNVBAoMBlNBTU9ORTEKMAgGA1UECwwBMTEYMBYGA1UEAwwPc2Ftb25lLmR5bnUu

Y29tMSEwHwYJKoZIhvcNAQkBFhJzb2lza2k3MUBnbWFpbC5jb20wggEiMA0GCSqG

SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDODCTA2yShw//yECJMjMveUaHffWzdE3G8

R+LyjBzLBKBd5sRWx9NZIZdPza/P8FrtLEqJ14lYNBbQ0D4HAvXQhzXBvqXVBdDP

0rCsToWhj+JIcji4JaNGyjAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUA

A4IBAQC8j8qLS8PEWfjgU0rDervoThosueyzRasVEjzOhtz0jstoLWAqYHBKp5ye

j4V3eXM1EJKBk7/PMNF9irLxclLsX7SsmxjFKBjmJIxR4tFkKwAsVjOCDqN5fxdy

2Cq/O99V/UoRmxp5/vkEdUjNWaTaLIr9t+TVoO3ujnaEO3iftqAt8Gcs0kZStbcE

0bqazrIq9xr8ONT6yQu3kV86bA8XvAiGVfxU2MMwVvmBWEWBW2Exk12WDdKdq5v6

j3opUSTK3NllUKMHM7Vr0nSpRWtxYQzbkv/EB+9dqi+GrSwW1kKpyTxKqBSMji87

h3rrSfWPhZdjojJQee2dBLodrU+9

-----END CERTIFICATE-----

)EOF";

#endif

#ifdef CHECK_PUB_KEY

// Extracted by: openssl x509 -pubkey -noout -in ca.crt

static const char pubkey[] PROGMEM = R"KEY(

-----BEGIN PUBLIC KEY-----

MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzgwkwNskocP/8hAiTIzL

0Ic1wb6l1QXQz6GLlvNeNj4XBeP23W/sxSX6CpSy5f82QhtQpspt2GL3olzWo80K

IslGNXlINMQ96Ji/bSoIM75FzzmVzYNi06dzMPGEZLmaQkFDhJ6PeSBBUGMUszUR

qHBz52XEscPY+GrNZczRUXge1wFJkJDWxpryo/7oJeh7vMOUAyhFIZ8fcdqXxSv6

-----END PUBLIC KEY-----

)KEY";

#endif

#ifdef CHECK_FINGERPRINT

// Extracted by: openssl x509 -fingerprint -in server.crt

static const char fp[] PROGMEM = "BD:D6:B7:F7:0F:A5:49:0C:A1:D8:0F:0C:D3:AC:6F:98:B9:95:11:9B";

#endif

//////////////////////////////////////////////////////

#if (defined(CHECK_PUB_KEY) and defined(CHECK_CA_ROOT)) or (defined(CHECK_PUB_KEY) and defined(CHECK_FINGERPRINT)) or (defined(CHECK_FINGERPRINT) and defined(CHECK_CA_ROOT)) or (defined(CHECK_PUB_KEY) and defined(CHECK_CA_ROOT) and defined(CHECK_FINGERPRINT))

#error "cant have both CHECK_CA_ROOT and CHECK_PUB_KEY enabled"

#endif

BearSSL::WiFiClientSecure net;

PubSubClient client(net);

time_t now;

unsigned long lastMillis = 0;

void reconnect() {

// Loop until we're reconnected

while (!client.connected()) {

status = WiFi.status();

if ( status != WL_CONNECTED) {

WiFi.begin(WIFI_AP, WIFI_PASSWORD);

while (WiFi.status() != WL_CONNECTED) {

delay(500);

Serial.print(".");

//Uutta -->

aika++;

Serial.print(aika);

if (aika > 30) {

aika = 0;

reconnect();

}

aika = 0;

Serial.println("Connected to Wifi");

Serial.print("IP address: ");

Serial.println(WiFi.localIP());

Serial.print("MAC: ");

Serial.println(WiFi.macAddress());

Serial.print("Setting time using SNTP");

configTime(+2 * 3600, 0, "0.fi.pool.ntp.org", "3.fi.pool.ntp.org");

now = time(nullptr);

while (now < 1510592825) {

delay(500);

Serial.print(".");

now = time(nullptr);

}

Serial.println(" done!");

struct tm timeinfo;

gmtime_r(&now, &timeinfo);

Serial.print("GMT time: ");

Serial.print(asctime(&timeinfo));

Serial.print("Local time: ");

Serial.print(ctime(&now));

}

Serial.print("Connecting to Mosquitto node...");

// Create a random client ID

String clientId = "ESP8266_SOLAR_";

clientId += String(random(0xffff), HEX);

// Attempt to connect

delay(1000);

if (client.connect(clientId.c_str())) {

// if (client1.connect(clientId.c_str(), mqtt_user, mqtt_password)) { // Use this with PW

Serial.println("[DONE]");

delay(1000);

} else {

Serial.print("failed, rc=");

Serial.print(client.state());

Serial.println(" try again in 5 seconds");

// Wait 5 seconds before retrying

delay(5000);

}

void SendMOSQData()

{

delay(500);

Serial.println("Collecting VICTRON MPPT data");

String msg_str;

char msg[50];

// Read sensor data for MOSQ

while (mySerial.read() >= 0); // Flush the input buffer of Serial1

label = "";

val = "";

while (i[0] != iset)

{

label = mySerial.readStringUntil('\t');

val = mySerial.readStringUntil('\r\r\n');

if (label == "V" && i[1] == 0) // Battery Voltage

{

if (val != "")

{

val.toCharArray(buf, sizeof(buf));

B_VOL2 = atof(buf);

float B_VOL = (B_VOL2) / 1000;

Serial.print ("Battery Voltage B_VOL: ");

Serial.println (B_VOL);

msg_str = String(B_VOL);

msg_str.toCharArray(msg, msg_str.length() + 1);

Serial.print("Publish message B_VOL: ");

Serial.println(msg);

client.publish("B_VOL", msg, true);

i[1]++;

i[0]++;

label = "";

val = "";

}

if (label == "I" && i[2] == 0) // Battery Current

{

if (val != "")

{

val.toCharArray(buf, sizeof(buf));

B_CUR2 = atof(buf);

float B_CUR = (B_CUR2) / 1000;

Serial.print ("Battery Current B_CUR: ");

Serial.println (B_CUR);

msg_str = String(B_CUR);

msg_str.toCharArray(msg, msg_str.length() + 1);

Serial.print("Publish message B_CUR: ");

Serial.println(msg);

client.publish("B_CUR", msg, true);

i[2]++;

i[0]++;

label = "";

val = "";

}

if (label == "VPV" && i[3] == 0) // Solar Panel Voltage

{

if (val != "")

{

val.toCharArray(buf, sizeof(buf));

P_VOL2 = atof(buf);

float P_VOL = (P_VOL2) / 1000;

Serial.print ("Solar Panel Voltage P_VOL: ");

Serial.println (P_VOL);

msg_str = String(P_VOL);

msg_str.toCharArray(msg, msg_str.length() + 1);

Serial.print("Publish message P_VOL: ");

Serial.println(msg);

client.publish("P_VOL", msg, true);

i[3]++;

i[0]++;

label = "";

val = "";

}

if (label == "PPV" && i[4] == 0) // Solar Panel Power

{

if (val != "")

{

val.toCharArray(buf, sizeof(buf));

P_POW2 = atof(buf);

float P_POW = (P_POW2);

Serial.print ("Solar Panel Power P_POW: ");

Serial.println (P_POW);

msg_str = String(P_POW);

msg_str.toCharArray(msg, msg_str.length() + 1);

Serial.print("Publish message P_POW: ");

Serial.println(msg);

client.publish("P_POW", msg, true);

i[4]++;

i[0]++;

label = "";

val = "";

}

if (label == "H22" && i[5] == 0) // Solar Panel Yield yesterday

{

if (val != "")

{

val.toCharArray(buf, sizeof(buf));

Y_YD2 = atof(buf);

float Y_YD = (Y_YD2) * 10;

Serial.print ("Solar Panel Yield yesterday Y_YD: ");

Serial.println (Y_YD);

msg_str = String(Y_YD);

msg_str.toCharArray(msg, msg_str.length() + 1);

Serial.print("Publish message Y_YD: ");

Serial.println(msg);

client.publish("Y_YD", msg, true);

i[5]++;

i[0]++;

label = "";

val = "";

}

if (label == "H23" && i[6] == 0) // Solar Panel Maximum Power yesterday

{

if (val != "")

{

val.toCharArray(buf, sizeof(buf));

Y_MPOW2 = atof(buf);

float Y_MPOW = (Y_MPOW2);

Serial.print ("Solar Panel Maximum Power yesterday Y_MPOW: ");

Serial.println (Y_MPOW);

msg_str = String(Y_MPOW);

msg_str.toCharArray(msg, msg_str.length() + 1);

Serial.print("Publish message Y_MPOW: ");

Serial.println(msg);

client.publish("Y_MPOW", msg, true);

i[6]++;

i[0]++;

label = "";

val = "";

}

if (label == "ERR" && i[7] == 0) // Solar Panel Error Code

{

if (val != "")

{

val.toCharArray(buf, sizeof(buf));

VIC_ERR2 = atof(buf);

float VIC_ERR = (VIC_ERR2);

Serial.print ("Solar Panel Error Code VIC_ERR: ");

Serial.println (VIC_ERR);

msg_str = String(VIC_ERR);

msg_str.toCharArray(msg, msg_str.length() + 1);

Serial.print("Publish message VIC_ERR: ");

Serial.println(msg);

client.publish("VIC_ERR", msg, true);

i[7]++;

i[0]++;

label = "";

val = "";

}

if (label == "CS" && i[8] == 0) // Solar Panel State of Operation

{

if (val != "")

{

val.toCharArray(buf, sizeof(buf));

VIC_CS2 = atof(buf);

float VIC_CS = (VIC_CS2);

Serial.print ("Solar Panel State of Operation VIC_CS: ");

Serial.println (VIC_CS);

msg_str = String(VIC_CS);

msg_str.toCharArray(msg, msg_str.length() + 1);

Serial.print("Publish message VIC_CS: ");

Serial.println(msg);

client.publish("VIC_CS", msg, true);

i[8]++;

i[0]++;

label = "";

val = "";

}

void gotosleep()

{

Serial.println("Good Bye, going to SLEEP ... zzZZZZzzz... ");

delay(5000);

ESP.deepSleep(SLEEP_LENGTH * 1000000, WAKE_RF_DEFAULT);

}

void setup()

{

Serial.begin(115200);

mySerial.begin(19200);

Serial.println();

WiFi.mode(WIFI_STA);

Serial.println("");

Serial.println("Connecting to Wifi ...");

// attempt to connect to WiFi network

WiFi.begin(WIFI_AP, WIFI_PASSWORD);

while (WiFi.status() != WL_CONNECTED) {

delay(500);

Serial.print(".");

//Uutta -->

aika++;

Serial.print(aika);

if (aika > 30) {

aika = 0;

reconnect();

}

aika = 0;

Serial.println(" Connected to Wifi");

Serial.print("IP address: ");

Serial.println(WiFi.localIP());

Serial.print("MAC: ");

Serial.println(WiFi.macAddress());

Serial.print("Setting time using SNTP");

configTime(+2 * 3600, 0, "0.fi.pool.ntp.org", "3.fi.pool.ntp.org");

now = time(nullptr);

while (now < 1510592825) {

delay(500);

Serial.print(".");

now = time(nullptr);

}

Serial.println("done!");

struct tm timeinfo;

gmtime_r(&now, &timeinfo);

Serial.print("GMT time: ");

Serial.print(asctime(&timeinfo));

Serial.print("Local time: ");

Serial.print(ctime(&now));

// CHECK_CA_ROOT

BearSSL::X509List cert(digicert);

net.setTrustAnchors(&cert);

// CHECK_PUB_KEY

// BearSSL::PublicKey key(pubkey);

// net.setKnownKey(&key);

// CHECK_FINGERPRINT

// net.setFingerprint(fp);

//

client.setServer(MQTT_HOST, MQTT_PORT);

reconnect();

}

void loop()

{

now = time(nullptr);

{

reconnect();

delay(1000);

SendMOSQData();

delay(1000);

client.disconnect();

delay(1000);

gotosleep();

delay(1000);

}

client.loop();

}

// SECRETS.h