SID detection with UKRAA VLF radio

[Tomatobro]

I am up and running!

Two challenges, the first was getting 75 turns on the antenna and realising that the slots were not deep enough for the remaining 50 turns so had to carefully unwind and re spool  the copper wire, and the second was to adjust the receiver to get all of the signal range within the 5 volts of the 10 bit A/D converter.

I take 10 readings over 30 seconds and then record the average. There is enough memory for 11 days of data.

Experience with these EEPROM's indicates that they have two failure modes, one (rare) is that the read write front end fails and the second is that an individual bit will not set to 1. So to erase each byte is set to 255 and then read back to ensure all bits have set to 1. It takes about 15 mins to do this on this size of memory chip

[HN50]

5 hours ago, Tomatobro said:

I am up and running!

Two challenges, the first was...

...about 15 mins to do this on this size of memory chip.

That is looking good, I look forward to comparing data.  

I am slowly rewriting my code so that the MKR 1010 will do what my existing Arduino does.  Had a few issues over weekend so not quite as far along as I had hoped.   Once that is running again I will look to get using the WiFi.

[HN50]

Well, I have checked the data for the last week and not a whole lot has happened - below is the weekly GIF.

And in the average for all time the numbers continue to settle down.

This week I have been focusing on getting the MKR running as a 1:1 copy of the bare bones Arduinos that I can't currently reprogram, and can say that I now have it running on the breadboard.

It is doing pretty much what I want it to, I just need to get it to go into sleep mode but all in all I am quite pleased.  So while the Sun may have been quiet this week things have been busy on the programming side.  I have also posted my code as well at the bottom.

I will order some more stripboard so I can solder up a chassis and also a plastic project box for it all to sit in.  I am chosing plastic as if it is metal I am not sure the WiFi signal will be shielded.

Not a bad week though.

Thanks for reading.

Dave

 

#include <DS3231.h> //v1.0.2 https://github.com/NorthernWidget/DS3231
#include <Wire.h>
#include <SD.h>             //SD card library
//#include <LowPower.h>       // https://github.com/rocketscream/Low-Power

DS3231 Clock;
bool Century = false;
bool h12;
bool PMT;

#define ALRM1_MATCH_EVERY_SEC  0b1111  // once a second
#define ALRM1_MATCH_SEC        0b1110  // when seconds match
//#define ALRM1_MATCH_MIN_SEC    0b1100  // when minutes and seconds match
//#define ALRM1_MATCH_HR_MIN_SEC 0b1000  // when hours, minutes, and seconds match
byte ALRM1_SET = ALRM1_MATCH_SEC;

//Variables for SD card START
Sd2Card card;
// set up variables using the SD utility library functions:
const byte chipSelect = 6;
//CS pin for SD card
String logFile = "VLF_MKR.txt";
//Variable to hold filename to be created/written to
//Variables for SD card END

//Variables for voltage calculation START
byte interruptPin = 7;
//float value = 0.0;
int var;
int loops = 10;
float voltage;
const float referenceVoltage = 1;
//Variables for voltage calculation END

//Variables for RTC START
//bool showRTCFlags = true;
bool showRTCFlags = false;

volatile bool alarmflag = false;
byte timeComponent;
//Variables for RTC END

void setup() {
  // Start the I2C interface
  Wire.begin();
  // Start the serial interface
  Serial.begin(9600);

  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB
  }

  Serial.println("Turn off alarm 1 & 2...");
  Clock.turnOffAlarm(1);
  Clock.turnOffAlarm(2);

  Serial.println("Set alarm clock time...");
  Clock.setA1Time(0, 0, 0, 1, ALRM1_SET, true, false, false);

  Serial.println(F("Turn on alarm 1..."));
  Clock.turnOnAlarm(1);

  Serial.println(F("Attaching pin interrupt..."));
  // Attach interrupt
  pinMode (interruptPin, INPUT_PULLUP);
  digitalWrite(interruptPin, HIGH);
  attachInterrupt (digitalPinToInterrupt (interruptPin), clockTrigger, FALLING);
  Serial.print(F("Pin interrupt on "));
  Serial.print(interruptPin);
  Serial.println(F("..."));

  Serial.println(F("Initialize SD card..."));
  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect))
  {
    Serial.println(F("Card failed..."));
    // don't do anything more:
    return;
  }
  Serial.println(F("SD card initialized..."));

  if (SD.exists(logFile))
  {
    Serial.println(logFile + F(" found..."));
  }
  else
  {
    Serial.println(logFile + " !found, creating...");
    File dataFile = SD.open(logFile, FILE_WRITE);
    dataFile.println(F("D,T,V,K"));
    Serial.println(logFile + F(" created..."));
    dataFile.close();
  }

  analogReference(AR_INTERNAL1V0);
  //Set to use internal 1V reference voltage - so that the 8 bit
  //range is measured over a smaller range
}

void loop() {
  voltage = 0;
  var = 0;
  while (var < loops) {
    voltage = voltage + analogRead(A4);
    delay(5);
    var++;
  }
  voltage = voltage / loops;

  voltage = voltage * referenceVoltage / 1023;

  float temperature;
  temperature = 273.15 + Clock.getTemperature(); //((Clock.getTemperature()) / 4);

  if (alarmflag == true) {
    File dataFile = SD.open(logFile, FILE_WRITE);
    if (dataFile) {
      if (showRTCFlags) {
        Serial.println(F("AlarmFlag is true"));
      }

      printString("20", dataFile);
      timeComponent = Clock.getYear();
      printByte(timeComponent, dataFile);
      printString("/", dataFile);
      timeComponent = Clock.getMonth(Century);
      printByte(timeComponent, dataFile);
      printString("/", dataFile);
      timeComponent = Clock.getDate();
      printByte(timeComponent, dataFile);
      printString(",", dataFile);
      timeComponent = Clock.getHour(h12, PMT);
      printByte(timeComponent, dataFile);
      printString(":", dataFile);
      timeComponent = Clock.getMinute();
      printByte(timeComponent, dataFile);
      printString(",", dataFile);
      printFloat(voltage, 4, dataFile);
      printString(",", dataFile);
      printFloat(temperature, 2, dataFile);
      Serial.println("");
      dataFile.println("");

      alarmflag = false;
      dataFile.close();
    }
  }
  if (Clock.checkIfAlarm(1)) {
    if (showRTCFlags) {
      Serial.println(F("checkIfAlarm(1) is true"));
    }
  }
}

void clockTrigger()
{
  alarmflag = true;
}

//Only use these when SD card is initialised
//and has been started ready for writing...
void printByte(byte value, File dataFile)
{
  if (value < 10) {
    Serial.print("0");
    dataFile.print("0");
  }
  Serial.print(value);
  dataFile.print(value);
}

void printString(String value, File dataFile)
{
  Serial.print(value);
  dataFile.print(value);
}

void printFloat(float value, byte places, File dataFile)
{
  Serial.print(value, places);
  dataFile.print(value, places);
}
//Only use these when SD card is initialised
//and has been started ready for writing...

 

Edited June 6, 2020 by HN50
Font was a bit small

[Tomatobro]

All my laptops are tied up analysing and logging the daytime meteor shower (Arietids) at the moment but as soon as one becomes free will take a look at my SID data for comparison.

[HN50]

1 hour ago, Tomatobro said:

All my laptops are tied up analysing and logging the daytime meteor shower (Arietids) at the moment but as soon as one becomes free will take a look at my SID data for comparison.

You have quite a few things on the go!  I have added my data for this last week below.

VLF_T2_20200606.TXT

[Tomatobro]

Well all I did was to disconnect the laptop and move the rig to a "quiet" outbuilding loft. this is 24 hours starting at 14:00 hours on the 4th June.

Clearly I have more work to do as it was working great before.

[Tomatobro]

My auto graphing software is not working properly so have been sorting that out and I have reduced the plot to 12 hours per jpg as this gives me better resolution. The rig will be moved to the outbuilding and the UKRAS radio output adjusted to suit its new environment. Will see what the improvements are in a few days

Update. The auto graphing software has been sorted, the receiver adjusted and recording started at 16:00.

Edited June 10, 2020 by Tomatobro
update

[Tomatobro]

Still not getting good data. Next step is to bring it back into the house and see if I can duplicate my earlier results.

[HN50]

Hi,

Sorry for the delay, did you get to the bottom of the problem?  Do you think it is radio interference or a problem within the microcontroller?

My progress...

It has been a bit of a busy fortnight with work so I have not been able to do too much on this, I have several weeks of data to analyse and will do that tomorrow.  However today I sat down with some stuff I had ordered and soldered the chassis for the MKR 1010 to plug into.  Have not tried it yet so will do that tomorrow as well.  

I left the board bare on the left in case I need to play around with the voltage divider.  Top right is the regulated power supply that drops first to 9V and then to 5V before being fed into the Arduino.  

It looks like it should work but I will reserve judgement until I switch it on..!

Thanks for reading.

[Tomatobro]

Things are sorting themselves out. The big spikes at the start of  data1 is the washing machine when it goes onto fast spin. The receiver has been moved from the house  back into the outbuilding loft where its logging away as I type.

One small problem is that we had a short power cut which meant that on power up it started to over write the existing data so I need to add a routine that scans for the first  blank memory slot on power up and starts recording from there. I will add an LED to warn me that a power outage occurred sometime during the week. Power outages will mess up its recording times but I will live with that as they are quite rare.

[7170]

@HN50 If it were me instead of using internal ref I would invest in something like a ADS1115, you can get them for a few £ off a well known auction site. 16 Bit ADC with I2C interface. Is the output from the radio is always +ve, or have you given it a positive bias?

As voltage is a float I would take a lot more than 10 readings without the delay between and average out without worry of overflow. I would go for 1000+, not sure if it would really make much difference in this case but things do become more stable.

A very interesting project, thanks for taking the time to share it!

[HN50]

@Tomatobro glad it is running, look forward to seeing the outputs. 

@7170 thanks for your comments.  In terms of the ADS1115 I am certainly not against the idea of going to a higher resolution, at this stage I am trying to keep down the number of extra add-ons while I learn more about Arduino.  10 bit is giving reasonable results and I think that the MKR 1010 can be increased to 12 bit.  But I did make the change you suggested of taking an average of 1000 readings so am looking forwards to seeing how that affects things.

My progress...

In terms of data not too much has happened in the last fortnight, though late on the 13th the signal starts to jump about as I assume they were testing/fixing (???) something at the transmitter.  Anyway, the radio appears to be working as expected which is giving me increasing confidence.  GIF for teh last fortnight;

The signal on the 13th looks like;

Which does make me wonder if the funny readings I got way back at the beginning were something happening at the transmitter rather than within my setup.  The average data looks like the following.  Nothing especially amazing but the data continues to settle down;

It occurs to me that I should average by month as the diurnal profile will alter as the day length changes.

Coding today has gone well, the chassis I built works when powered from the laptop and before powering it from the power regulator on the chassis I checked that what was coming was indeed 5V as I didn't want to fry the Arduino.  It was, so I tried to get it logging powered by the supplied by the radio.

It did nothing.

The Arduino powered up but no logging was attempted.  After scratching my head I found the offending code. 

    while (!Serial) {
      ; // wait for serial port to connect. Needed for native USB
    }

I added that to get the Arduino to wait until the Serial port was connected so I could see the comments I added to check which parts of the code have been executed in the setup function.  Unfortunately when running from VIN/ground pins the serial port will never connect anyway so it was going into an endless loop.  I made a hasty tweak logging now occurs under battery power.  It is not waking up from sleep yet (need to work out why) but I am keen to get the MKR logging data properly so will look at this during the week.  The linear regulators get warm but nothing too bad and I will just bear this in mind at the moment, I used them as I already have them in and did not want to buy more parts than I have to.  Otherwise i can look out for a bit of copper or aluminium for a heat sink.

And just as importantly, the RTC is now set to GMT!

So all in all not bad, I just need to work out why the wake from sleep is not working.  But another week or two and I think I can start looking at connecting it to my WiFi and looking into Azure IoT...

Thanks for reading.

 

 

[HN50]

It is now logging in the roof, so I will give it a week and see what comes out...

Data up to today below.

VLF_T2_20200621.TXT

[Tomatobro]

18 hours ago, 7170 said:

Is the output from the radio is always +ve, or have you given it a positive bias?

The antenna is receiving two signals, one a ground wave and the other a sky wave. Because of the uneven travel distance the two waves arrive at different times and interfere with each other so the voltage output is a function of the combined signals. X-rays remove the sky wave component so the resulting signal can either increase or decrease depending on location. It's that sudden change in output followed by a slow recovery is what we are looking for.

The wave length is 13 meters

Edited June 21, 2020 by Tomatobro
data added

[HN50]

17 minutes ago, Tomatobro said:

The antenna is receiving two signals, one a ground wave and the other a sky wave. Because of the uneven travel distance the two waves arrive at different times and interfere with each other so the voltage output is a function of the combined signals. X-rays remove the sky wave component so the resulting signal can either increase or decrease depending on location. It's that sudden change in output followed by a slow recovery is what we are looking for.

@7170 Sorry, I missed that.  Yes, it is always positive, either 0-2.5V or 0-5V.

Dave

[7170]

One other tip, I can see you have referenced the LowPower library but don't look to use it in the code, if you do end up putting it to sleep and waking it up again it is best to do a few analogue reads and ignore the values, before you start your actual reading/averaging loop, as the first few readings can very often be spurious after waking up (especially the first reading).

[HN50]

On 21/06/2020 at 21:37, 7170 said:

ain it is best to do a few analogue reads and ignore the values, before you sta

 

On 21/06/2020 at 21:37, 7170 said:

One other tip, I can see you have referenced the LowPower library but don't look to use it in the code, if you do end up putting it to sleep and waking it up again it is best to do a few analogue reads and ignore the values, before you start your actual reading/averaging loop, as the first few readings can very often be spurious after waking up (especially the first reading).

Hi,

Thanks for the suggestion, I got the sleep mode to work (more later) and have put in a loop that reads 50 times before I actually start taking measurements.

My work this week.

I pulled the data out yesterday from the new MKR 1010 rig to see how things went...

It has been running all week as expected despite the furnace-like temperature of nearly 40 Celsius up there towards the end of the week.  So it is running successfully.

Eyeballing things I could not decide if the data looked noisier than with the Barebones Arduino, so I tried to analyse the data by plotting standard deviation of the two rigs by hour.

Things are a bit all over the place at night, while the Barebones data has a greater stdev around the maintenance window time; this only happened once in the last week for the MKR 1010 data.  But between 10:00 and 17:00 when the signal plateaus I think it is not as clear, but perhaps the MKR 1010 does have a slightly greater standard deviation than the Barebones output and I would therefore  assume noisier signal.  The Barebones data is everything from 5th May, the MKR 1010 just 5 day's worth so I might collect more data before I pass judgement.  I could look to code a 5 minute moving average within the data using an array to try and smooth the output I suppose.

Otherwise I now have the code going into deep sleep between taking measurements.  I thought it was not working as when running it from the IDE I would get my start up messages in the Serial monitor and then nothing. 

When I looked at the SD card though the Arduino had been logging data, what I think happens is that when the MKR 1010 goes to sleep it loses its connection to the previously assigned COM port and when it awakens does not reconnect.  This is different to the Bareones Arduino which appeared use the same COM port on wake-up and so kept outputting its results to the Serial Monitor.  The upshot is that no new event data gets posted to the Serial monitor for the MKR 1010 which is why I thought the sleep mode was not working.

I also find it 'a bit difficult' to compile code to the MKR 1010 when it is asleep, so it now only goes to deep sleep when it cannot find the Serial port (i.e. under battery power).  A slight reverse of the wait for Serial problem I had the other week.

    if (!Serial) {
      LowPower.deepSleep();
    }

Aside from that the spare RTC I bought seems to have a problem in that I need to look at (or cut losses and get another for a couple of quid) and I have just about all the parts now to finally get it into a plastic casing.  I will look to assemble the case for it today.

Thanks for reading.

VLF_MKR_20200627.TXT

 

Edited June 28, 2020 by HN50
Stray graph removed

[HN50]

And now fitted in a project box for a little protection.

[7170]

22 hours ago, HN50 said:

My work this week.

I pulled the data out yesterday from the new MKR 1010 rig to see how things went...

 

I think you can be very proud of your work - looks like good plots with the expected shapes and to my eye doesn't look to have a temperature variance that needs to be adjusted for (assuming it is just raw data being plotted).  I would also argue the above plots are not that noisy, so you should be able to see C/M/X class flares and maybe even B class too.

I would play with a moving average (in excel) once you have some known flares to see if that is worth applying to data once it is captured and if the flares are still all visible or not. 

Another challenge can be managing all the archive of data once you are capturing ok, so you can quickly plot specific days. I plot with a 10s capture interval and find excel struggles and even PowerBI is a bit slow. In the future i'm thinking of putting everything into a MySql DB and using python for processing and then graphing - I just need some free time!

[Tomatobro]

I made some changes to the hardware and relocated the antenna but I cannot match my earlier results. next step is to provide a ground as the only thing I can now think of is that when a laptop is connected via USB comms link the noise is reduced.

[Tomatobro]

I have added a proper earth for the antenna and its cleaned up the signal. Its back in the outbuilding loft and started logging at 14:00

[HN50]

On 29/06/2020 at 11:31, 7170 said:

I think you can be very proud of your work - looks like good plots with the expected shapes and to my eye doesn't look to have a temperature variance that needs to be adjusted for (assuming it is just raw data being plotted).  I would also argue the above plots are not that noisy, so you should be able to see C/M/X class flares and maybe even B class too.

I would play with a moving average (in excel) once you have some known flares to see if that is worth applying to data once it is captured and if the flares are still all visible or not. 

Another challenge can be managing all the archive of data once you are capturing ok, so you can quickly plot specific days. I plot with a 10s capture interval and find excel struggles and even PowerBI is a bit slow. In the future i'm thinking of putting everything into a MySql DB and using python for processing and then graphing - I just need some free time!

Hi James,

Thank you, this project has been good fun so far. 

Yes, the data I am plotting is just the raw numbers there is no other data manipulation being made to it.  I think I might try a moving average on the data as you suggest as it would be good to see whether the flare is still visible or if it gets smoothed out.

I have been logging at the minute level but I notice that you log every 10 seconds - how do you find the trade-off between increased temporal resolution of versus the data volume?  What would be the longest interval between measurements you would consider?  I had wondered about dropping the frequency to maybe once every two minutes (I might simulate with data first...) as the C class flare I detected caused the signal to drop over the course of two minutes with a recovery of maybe half an hour, so two minutes might not miss too much activity(?).  Although having detected only one flare so far I might want to see what happens for others, but once every two minutes would cut my data capture in half to 720 measurements a day.

On 03/07/2020 at 14:13, Tomatobro said:

I have added a proper earth for the antenna and its cleaned up the signal. Its back in the outbuilding loft and started logging at 14:00

Hi,

That is looking good.  Stupid question,  but how have you connected your antenna to earth?  Looking at the antenna connection in the pcb, although both ends of the loop plug in to it, it looks to me like only one end goes into the amplifier.  Did you just add a lead from one of the loop wires on your antenna to an earth connection?  Did you need to do any antenna re-tuning?   I ask as the affect on your plot is quite marked, I like tinkering and there is a water pipe not too far from my setup. 

 

I have not been able to do too much this weekend as with the good weather I have spent some time down on the allotment making raised beds and keeping wood pigeons off the sprouts.  The male to male power lead arrived so I am now able to use the power in socket on my project box and fix the lid down.  I realised I have not drilled any cooling vents above the linear regulators so I will see what that has done to the data when I pull it out tomorrow.

On a technically interesting, but visually dull, level I have been playing with the wifi on my MKR 1010.  I set up an Azure subscription* and worked through the page below;

https://create.arduino.cc/projecthub/Arduino_Genuino/securely-connecting-an-arduino-nb-1500-to-azure-iot-hub-af6470

There is not much to see but I now have an Azure hub that is able to receive data from my MKR1010 and running the Azure shell command prompt I can see the messages coming in and the data within it, which with the example Arduino sketch is just the message "Hello".  I also was able to send a message back to the MKR 1010 and see it appear in the serial monitor.  Next I need to see about merging at least some of my VLF logging code into the example sketch to see my timestamp, voltage and temperature measurement coming in. 

This is all new to me so as an initial goal I found the page below.  It does not address long term storage of the data in a SQL database but it would be good to be able to view my data in real time.

https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-get-started-with-azure-stream-analytics-to-process-data-from-iot-devices

It also occurred to me I should check that the wifi signal in the roof is strong enough to make any of this possible!  If not I may need to buy a wifi range extender and set it up in the room below.

So not that much to show today but what I have been investigating is rather exciting.

Thanks for reading,

Dave

* I want to keep a close eye on the billing as I don't want to get hit by any unexpected/unpleasant costs...

Edited July 5, 2020 by HN50
Text

[Tomatobro]

For the earth and to see the effect wrap a wire round the BNC connector and connect it to the water pipe. I noticed that when I connected my RS232 comms cable to the laptop the signal quality improved which suggested that the receiver was being grounded via the comms cable. Also I attached a high quality 0 to 3 volt analogue meter to the output so I could keep an eye on the signal strength. When this meter was connected the signal increased by almost a volt. When the comms cable was  connected the meter fell back and indicated the expected signal, disconnecting the comms lead and you could watch the signal slowly rise. I emailed the folks at UKRAA and they suggested earthing the metal case (the coax is grounded to the case from the tuning unit via its BNC connector)

I still see interference from the washing machine motor and the induction hob even though the outbuilding is 60 metres away and the machines sit on the null plane of the antenna. Also if the outbuilding strip lights are turned on I see a small increase in signal strength. I guess I will have to live with this level of interference.

Longer term will see an experiment with an earth rod driven into the ground outside to see if that works any better.

 

[7170]

16 hours ago, HN50 said:

I have been logging at the minute level but I notice that you log every 10 seconds - how do you find the trade-off between increased temporal resolution of versus the data volume?  What would be the longest interval between measurements you would consider?  I had wondered about dropping the frequency to maybe once every two minutes (I might simulate with data first...) as the C class flare I detected caused the signal to drop over the course of two minutes with a recovery of maybe half an hour, so two minutes might not miss too much activity(?).  Although having detected only one flare so far I might want to see what happens for others, but once every two minutes would cut my data capture in half to 720 measurements a day.

A good question. I started using 5s as that is what SuperSID (a python tool that can collect and submit data to Stanford as part of their Space Weather Monitor program) used by default. However I changed it to 10s to give the Pi some breathing space between samples as I monitor a large number of stations below 24khz. If I dropped it down further I would probably goto 30s, any more you risk loosing the exact start time of events when applying moving averages (if that is of interst). The output is a c2mb data file each day, so not too bad for storage but harder to plot that many samples quickly on a graph in excel or getting PowerBI to calculate a running average and then plot.

 

2 hours ago, Tomatobro said:

For the earth and to see the effect wrap a wire round the BNC connector and connect it to the water pipe. I noticed that when I connected my RS232 comms cable to the laptop the signal quality improved which suggested that the receiver was being grounded via the comms cable. Also I attached a high quality 0 to 3 volt analogue meter to the output so I could keep an eye on the signal strength. When this meter was connected the signal increased by almost a volt. When the comms cable was  connected the meter fell back and indicated the expected signal, disconnecting the comms lead and you could watch the signal slowly rise. I emailed the folks at UKRAA and they suggested earthing the metal case (the coax is grounded to the case from the tuning unit via its BNC connector)

....

Longer term will see an experiment with an earth rod driven into the ground outside to see if that works any better.

I would encourage great caution at earthing the case or connectors using a separate connection to an earth (like a rod in the ground or water pipe) unless the radio it is fully opto-isolated from your computer or it has been designed with this in mind. Or else you may inadvertently created a second earth point for your whole house (or larger local area if you have a TN-S or TN-C-S supply). Which may or may not be at the same earth point (or if an old install and something has failed the only true earth point). Also you run the risk of RCDs in the main fuse board tripping incorrectly.

Water pipes are not so good these days as supply pipes into houses and in roads are being replaced with plastic. In many cases now they won't provide an adequate ground. If it is confirmed as being safe to use a separate earth connection on your radio a separate copper rod into the ground is the way to go (I have to use this on my lighting detector). You can get the rods very cheap from DIY stores but don't connect your house earth to this dedicated rod.

Edited July 6, 2020 by 7170

[Tomatobro]

Latest SID data. Washing machine interference on the 3rd and 4th

Edited July 10, 2020 by Tomatobro
update