Weather Station Ideas

[davefi]

The commercial weatherstation I use has a haptic rain sensor. It is useless for drizzle, but works extremely well for "normal" rain, I often get a phone notification before I realise it is raining. Might be another idea to investigate - except the drizzle aspect (which as previously mentioned is also an issue with the tipping bucket as well).

[Gina]

We get a lot of drizzle!  Useful information on not what to buy though 😀  Thanks.

Edited August 28, 2020 by Gina

[Gina]

Regarding Consensus Averaging :- using a ring array of 16 integers.

sum =  n(i) + n(i+1) + n(i+2) + n(i+3) + n(i+4).

Becomes

sum =  n(i) + n((i+1)%16) + n((i+2)%16) + n((i+3)%16) + n((i+4)%16)

And

W = ( n(I+1) + 2 * n(I+2) + 3 * n(I+3) + 4 * n(I+4) ) * 45 / S

Becomes

W = ( n((I+1)%16) + 2 * n((I+2)%16) + 3 * n((I+3)%16) + 4 * n((I+4)%16) ) * 45 / S

I do wonder if this is more complicated (using more space and time) than just using a 20 integer array rather than 16.  OTOH I think this is the more correct approach.  I'm not sure that the results are going to be correct around the zero direction with 20 "bins" but I'm having difficulty getting my poor old brain round this complex maths.

If the C++ (Arduino) language had nybble as a data type I could use that for the index since the modulo is 16 and save having to apply modulo.

Whether %16 uses more resources than bit logic (&15) I don't know and doubt it makes much difference in this case but from my programming experience in many languages, bit logic is usually more sparing of resources.

Edited August 28, 2020 by Gina

[tekkydave]

I used the algorithm on this web site for my calculations.

[tekkydave]

If you read Java this is my class

package piweather4jv2;

import java.util.ArrayList;
import java.util.Date;
import java.util.Iterator;
import java.util.List;

// Class constructed using instructions at: http://www.beals5.com/wx/faqs.htm

public class WindConcensus {
    
    public void readWindConcensus(Database db, int dbSensorId, FileManager fm) {
        List<Double> windDirectionResults;
        List<Double> windSpeedResults;
        List<Double> windGustResults;
        double windSpeed = 0;
        double windGust = 0;
        int[] bin = new int[20];
        boolean validConcensus = true;
        Date endDate = new Date();    // endDate of readings = now
        
        windDirectionResults = db.getRecentReadings(1, endDate, 900L, "NONE");       // Get all the values from last 15mins
        windSpeedResults     = db.getRecentReadings(16, endDate, 900L, "AVG");       // Get the Average of the values in lat 15mins
        windGustResults      = db.getRecentReadings(16, endDate, 120L, "MAX");       // Get the Maximum of the values in last 2 mins
        
        if ((windDirectionResults.size() >= 1) && (windSpeedResults.size() == 1) && (windGustResults.size() == 1)) {
            
            // Put windDirectionResults in bins
            Iterator dirIterator = windDirectionResults.iterator();
            while (dirIterator.hasNext()) {
                double degrees = (double)dirIterator.next();
                int sector = ((int) ((degrees + 11.25)/22.5))%16;
                bin[sector] += 1;
                //System.out.println("WindConcensus: " + String.valueOf(degrees) + " in sector " + String.valueOf(sector));  
            }
            // bins 16 to 19 are same as bins 0 to 3
            bin[16] = bin[0];
            bin[17] = bin[1];
            bin[18] = bin[2];
            bin[19] = bin[3];

            // Process bin outcome
            int I = 0;
            int S = 0;
            int Smax = 0;
            for (int i = 0; i < 16; i++) {
                S = bin[i] + bin[i+1] + bin[i+2] + bin[i+3] + bin[i+4];
                if (S > Smax) {
                    I = i;
                    Smax = S;
                }
                //System.out.println("bin " + i + " = " + bin[i] + " , sum = " + S);
            }
            //System.out.println("I = " + I + ", S = " + Smax);
            
            double W = ( bin[I+1] + (2 * bin[I+2]) + (3 * bin[I+3]) + (4 * bin[I+4]) ) * 45.0D / Smax;
            double D = (((I * 45.0D) + W) % 720)/2;
            
            //System.out.println("W = " + W + " , D = " + D);

            // get windSpeedResults value
            windSpeed = (double)windSpeedResults.get(0);
            //System.out.println("WindConcensus: Average Speed = " + String.valueOf(windSpeed));  

            // get windGustResults value
            windGust = (double)windGustResults.get(0);
            //System.out.println("WindConcensus: Gust Speed = " + String.valueOf(windGust));  
            
            // Check Validity of Wind Direction and Speed
            if ((D < 0.0D) || (D >= 360.0D) || (windSpeed < 0.0D) || (windSpeed >= 100.0D) || (windGust < 0.0D) || (windGust >= 100.0D)) {
                validConcensus = false;
            }
            
            Reading3 windDataReading = new Reading3(new Date(), D, windSpeed, windGust, validConcensus);  

            windDataReading.print("Wind Direction Concensus (" + windDirectionResults.size() + " readings)","Wind Speed","Wind Gust");

            if (windDataReading.isValid()) {
                windDataReading.saveDB(db, dbSensorId);
                windDataReading.saveFile(fm, dbSensorId);
            }
            
        } else {
            System.out.println("WindConcensus: No results to process.");
        }
        
    }
}

 

[Gina]

I've been examining the Consensus Averaging procedure with my Mathematical Hat on and I understand all of it except the weighted sum.

W = ( n(I+1) + 2 * n(I+2) + 3 * n(I+3) + 4 * n(I+4) ) * 45 / S

I don't understand weighting the higher index "bins" higher.  I would have thought a Gaussian shaped weighting would have been wanted rather than a linearly increasing weighting.  If there are any maths experts around, I would be grateful for an explanation.  However, the proof of the pudding... is that this method seems to work pretty well!!

Thinking about it, I may just go for the method shown rather than out think it and use a ring array.  After all "it ain't broke so don't try to fix it". 😁

[Gina]

@tekkydave that looks much like the same maths in a different language for the direction but also includes the speed.

That's web page I'm using for the algorithm.

Edited September 4, 2020 by Gina
Added an extra line

[tekkydave]

I put everything in the same class as it writes them to the database as a single row. You are probably only interested in the direction bits. 

[Gina]

I'm not using a database yet.  Also, I'm not familiar with Java and using C++ (Arduion IDE).  I like a compiled language.

I think I shall keep direction and speed data and functions separate as this aids debugging.

Edited August 28, 2020 by Gina

[Gina]

I was going to use a 2 dimensional array for the mean and gust ring array but it seems the Arduino IDE has the odd error with multi-dimensional arrays so I'll use two single dimension arrays.

[tekkydave]

19 minutes ago, Gina said:

I'm not using a database yet.  Also, I'm not familiar with Java and using C++ (Arduion IDE).  I like a compiled language.

I think I shall keep direction and speed data and functions separate as this aids debugging.

The class I posted is part of the aggregation functions in my system. I have a separate logger that just puts raw values into the db with minimal processing for speed. The aggregation functions read from the db and write back things like wind speed & direction concensus that depend on values over a set period. Also I have hourly, daily, monthly, yearly aggregation of temperature, rainfall and wind speed going on in the background. My database is at 1.5Gb already with data going back to 2016. Daily backups happen automatically

 

[Gina]

I'm going to need to rethink the periods involved in the various counts and calculations. 

The Consensus Averaging I've been looking at uses 60 samples with 5s sampling and 5m period.  If I were going to use 1m reporting, with 3s sampling that would only be 20 samples in each period.  I don't think this is enough to give good results.  If I were to use 3m integration period yet report every minute I would need 3 sets of bins and the functions that go with them.  I think I might go for a 3m reporting period for wind direction.

For the wind mean speed and gust speed, the gust integration period wants to be 3s as it already is.  An integration period of 10m is recommended by the Met Office.  A reporting period of 10m would be the easiest solution but I think I would prefer a more frequent report.  I could go back to the 1m reporting period and 10m integration using a pair of 10 integer ring arrays - one for mean and one for gust speeds.

The result of the above is that the reporting intervals are different for direction and speed.

[Gina]

I think I need to establish various intervals in void loop() and put the required data process in routines that are called at those intervals.

1. 3s - get speed count and read direction
2. 60s - get result of average and maximum of wind speed count - circulate in ring arrays and calculate rolling average and max
3. 3m - process direction data and send message to MQTT
4. 10m

Not sure the 10m period is needed.

Edited August 28, 2020 by Gina

[Gina]

Can anyone please tell me what's wrong here?

[pete_l]

Can't have variable names beginning with a number?

[Gina]

Ah!!  Thank you.  That was it - compiles now.

Pity the error didn't say that!

Edited August 28, 2020 by Gina

[Gina]

Think this should work :-

void loop() {
  if (!client.connected()) {
    reconnect();
  }
  client.loop();

  long now = millis();
  if(now - last3s > P3s) {do3sJobs; last3s = now;}
  if(now - last1m > P1m) {do1mJobs; last1m = now;}
  if(now - last3m > P3m) {do3mJobs; last3m = now;}
  if(now - last10m > P10m) {do10mJobs; last10m = now;}
}

 

[Gina]

I'm hoping this is right for the speed calculations

void do3sJobs (){
  windSpeed = ++PulseCount;  //  Accumulate mean speed
  if (PulseCount > windGust) {windGust = PulseCount;};  // Get max speed for gust
  PulseCount = 0;
  ++dirbinsArray[readDirection()]; // increment appropriate bin
}
void do1mJobs(){
  // wind speed mean and gust ring arrays and report
  meanArray[ringIndex] = windSpeed; // put windSpeed into new array index
  gustArray[ringIndex] = windGust;  // put windGust into new array index
//  windSpeed /=20;  // leave this for later
  for (int i = 0; i < 10; i++) {
    meanSpeed += meanArray[i];  // add all windSpeeds 
    if (gustArray[i] > gustSpeed){gustSpeed = gustArray[i];}; // find maximum gust speed
  }
  meanSpeed /=200;  // the first sum was over 20 values and then the second over 10 values
  sendSpeedMessages();
  ringIndex = (ringIndex+1)%10; // move ringIndex on to next location in the ring array
}

 

[Gina]

I shall need to clear the gustSpeed to zero each 1m period.  That will still keep the earlier gust values in the ring array but allow it to go after 10m.

[Gina]

Now looking at the Consensus Averaging.  It's even more complicated than it looks!  I don't think I shall be doing much more tonight.

[Gina]

OK so can someone please tell me what I've done wrong here?

Further up I have this

// Direction count bins
int bin[20] = {0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0};

Maybe it doesn't like "bin"?

[Victor Boesen]

@Gina You're multiplying by [I+3] instead of what I would imagine should be bin[I+3]??

Victor

[JamesF]

You've missed "bin" out before the "[I+3]" part.

James

[JamesF]

Now two of us have said it, so it must be true

James

[Gina]

Blummin' 'eck - so I did - must be getting tired.  Thanks both.