SID detection with UKRAA VLF radio

[HN50]

On 09/07/2020 at 23:23, Tomatobro said:

Latest SID data. Washing machine interference on the 3rd and 4th

 

Hi,

Glad you got it resolved, your plots are looking good.  It is interesting the washing machine motor has such an impact on the output.

My progress this week;

I pulled out the SD card and had a look.  As previous weeks it has been another quiet period down here in solar minimum GIF is below;

The heat from the linear regulators did not cause any problems so that is something.

The bigger piece of work has been that I took the test script that I was using to connect to Azure Hub and merged my radio logger code into it.  This I have managed and the Azure prompt showed me the following;

Excuse all the red redactions but many of the names have to be globally unique.  Anyway the data coming in looks like my data, so I think the code splice worked.  I have found that the data payload data needs to be in JSON format, so a bit of tweaking my sketch now gives an output of;

{"D":"2020-07-12","T":"15:49","V":0.001,"K":299.9,"A":0.0009}

("A" is a 5 minute moving average I am calculating in the Arduino as you pay by activity in Azure)

That being in JSON then makes the query that should copy it to SQL easier.  I say easier as I had the job set up and running (so it said) but the data was not getting into the database, so I think there is a setting wrong somewhere. 

However I am using a tool called Stream Analytics in Azure which is Platform as a Service.  It (supposedly) makes setting the job up quicker but you pay for it by the hour, and the default is 3 units running it at £0.082 per hour.  So if you are running 24 hours a day... 

I read that a much cheaper way to do it is with a tool called Azure Functions where you have to write the whole process yourself, though there will be much more of a learning curve to that.  Anyway at this stage I am just experimenting with the free trial* and will just keep investigating.

Way back though my goal had been to see if I could get the logger writing to a database in the cloud.  If I can get past this last glitch then I will have managed to go from antenna to database in one end to end process.  I have not decided on the final means of storing and visualizing the data but I will be pleased to have it working as prototype. 

I think I first thought about this 4 years ago!

Thanks for reading,

Dave

* As a backup I have had a look around at other IoT platforms :D

 

Edited July 12, 2020 by HN50
Word

[Tomatobro]

I have been trying to figure out what is the cause of the spike at 18:00 hours that has appeared in in both my old and new data. One one hand its useful as a timing marker but what switches on and off for such a brief period is a mystery. Its not the central heating that's for sure.  Maybe one of the parked cars are doing something with the door lock solenoids?

[HN50]

On 13/07/2020 at 10:20, Tomatobro said:

I have been trying to figure out what is the cause of the spike at 18:00 hours that has appeared in in both my old and new data. One one hand its useful as a timing marker but what switches on and off for such a brief period is a mystery. Its not the central heating that's for sure.  Maybe one of the parked cars are doing something with the door lock solenoids?

Did you find out what the spike was caused by?  How has the logging been going?

My progress has been a little sparse without any data being captured for nearly a month.  The issue is that I have been trying to get to grips with Azure - I can see data coming in to IoT hub, I can see my SQL database at the other end but trying to get the data into it seemed to elude me.  There are a few options - tutorials on Stream Analytics, event hubs, event grids etc that seem to offer what I need it to do but I was missing something.  Plus trying to get it working before my free trial finished and only getting to do it at the weekend was proving to take the fun out of it.

Fortuitously it turns out that with software licensing at work we get some free credit each month to experiment so that took the heat off things.  I decided to initially get it working with just the Azure web interface and not resort to programming via Visual Studio.  After having put it aside for a week I sat down today at a leisurely pace and worked through the tutorial below;

https://www.mssqltips.com/sqlservertip/6335/how-to-capture-iot-data-in-azure/

Yes, Stream Analytics costs more but at this stage I would just like to get it working.  A colleague says that you can program functions that work out much cheaper so that is a future avenue of inquiry.

I was able to get the job working but I kept getting cryptic errors that related to permissions to access the database and data types that the stream analytics job can handle (it can't do 'date' and 'time' separately, only 'datetime' so I had to merge the previously separate columns together in the MKR1010 code).  After a few hours I ran the following code;

That does not look much but that was a back-of-the-net moment.  First query is a select from the target table, it then waits 3 minutes and then runs the same query again.  There are now another three rows - data is being loaded.

I got the radio back up in the roof and have left it running this afternoon.  I downloaded Power BI and have put together a first attempt chart to see what is happening.

It is still running so I will leave it overnight and into tomorrow to see how things look.  It did occur to me that as you can't detect SIDs at night anyway it might(?) be possible to schedule Stream Analytics to stop at about 20:00 and start up at 06:00 and to load the intervening data that has built up.

So after a bit of a hiatus things are getting going again - I am now logging my data to a database.  I am sure there will be other glitches but they can be overcome. 

Thanks for reading.

Dave

[Tomatobro]

I have been doing some timing drift tests over each 11 day recording session and its significant so am incorporating a new timing system which should be done by Monday. Timing for me is critical because I start the recording run at a particular date/time and just add 30 seconds for each data point. The few milliseconds drift per data point are adding up. Also we have had some power outages which mess up the timings as the clock stops and restarts once the power comes back on.

I also have looked into the "washing machine" interference, checking that the machine frame is earthed and that the mains wiring is correct. As yet I have not tried the ground rod but its on the list to do. Mrs Tomatobro suggested that if the machine was messing up my SID data then we should get a new one..............

Once the timing issue is sorted and I have the exact timing of the mystery spike I will investigate further. But apart from the interference and spike the recordings are going well.  Just in time as it looks if the Sun is waking up at last.

[Tomatobro]

Sample graph attached

[Tomatobro]

Timing unit arrived this morning so data recording times now linked to MSF in Cumbria.

[HN50]

2 hours ago, Tomatobro said:

Timing unit arrived this morning so data recording times now linked to MSF in Cumbria.

Hi,

Get a new washing machine, I have been setting the countdown timer on ours (and the dish washer) so that it runs in the middle of the night when I can't detect SIDs anyway.    I am not sure if it has been interfering but better safe than sorry.

I am interested by your timing unit - what are you using?  I take it it is a radio controlled clock of some sort?

I fired up Power BI to have a look at what is happening (and more importantly to check the logging is still working!), and it is.

I need to experiment a bit with the dashboard as I want the date on the x axis as well.

On a tangent I have also been looking at the CosmicWatch home Muon detector here http://www.cosmicwatch.lns.mit.edu/detector.  The scintillator plastic has arrived along with a customs bill of £15.  Would be a separate build log but further down the line might be good to get that data displayed alongside this VLF data.

Thanks for reading,

Dave

[HN50]

Despite my Eeyore-ish outlook on things it is still running;

I want to add a column to my data that identifies today and yesterday, at present this chart will just plot all data for evermore.  I can have a view in my database that does this, I will also add a flag to identify the last 7 days for a separate chart. 

Something that has changed is that I keep checking my laptop to look at activity - now I don't have to brave 45 Celsius up in the roof (the graph shows that 15:00 yesterday was a little unpleasant!) or spend time shuffling data around in LibreOffice it is much easier to have a quick peek at what is going on. 

Dave

Edited August 8, 2020 by HN50
Spelling

[HN50]

It has been a bit hot to do anything today so I have been sitting in the shade a little.  The load is still running so the process seems to be reliable;

I have however been giving some thought to the diurnal profile I get and why this might be.  I found a Stanford University website and it mentions the sunrise and sunset behaviour is caused by sunlight sweeping across the path of the radio before the sun has risen at either the transmitter and receiver locations.  Likewise it mentions that the creation of the D layer attenuates waves that pass through it on the way to being reflect by the E layer.

http://solar-center.stanford.edu/SID/activities/ionosphere.html

To that end I went and found a spreadsheet that calculates solar elevation by minute (well every 3 minutes) for a given lat/long and got this into the database for my local position as well as at the location of DHO38.  I then spent some time comparing this to signal strength data for 09/08/2020 and trying to think about what I was seeing and how it may relate to what I think is happening.

Shots are below, would be interested to see what others think.  I have used terms like civil and nautical twilight only as I noticed a few of the changes occur around their boundaries. 

Thanks for reading,

Dave

 

[Tomatobro]

That Muon detector looks interesting. Shame they do not do a full kit but I look forward to your write up when works starts on it

[HN50]

On 10/08/2020 at 18:03, Tomatobro said:

That Muon detector looks interesting. Shame they do not do a full kit but I look forward to your write up when works starts on it

The scintillator and optical grease arrived the other day, will start looking at the PCB in KiCAD so hopefully will make a start soon.  Not quite sure which subject it would go under though?

My progress...

I am on holiday at the moment and I spent yesterday making some back-end changes.  Rather than logging the date and time as one column, I have reduced them to two numbers - for date an integer in the format YYYYMMDD *, and for time the number of minutes past midnight.  I did this as I would like to bring in additional data (the proposed Muon detector, GOES x-ray data from NOAA, maybe another VLF radio...) and I want to have common date and time calendar tables in the database to join to them all to.  That should make them easier to plot together.  I also removed the in-Arduino moving average as it was not calculating and in any case that could be calculated in the database.

Date table looks like (I have it going to 2040);

Time table looks like;

If I need them together I can then create a view over both that joins on DateKey and TimeKey.

I have also been practicing Power BI more and have been able to replicate the charts I have been running in LibreOffice.  I found the breadth of choices available within charts slightly narrower in Power BI but the big difference is now how much faster I can display data  without having to shuffle columns around in LibreOffice.  I have loaded into the database some GOES X-ray flux data to see how easy it is to display it alongside my data.  The feed isn't automatic so I will have to think about how I do this as it too gets updated every minute. 

I also discovered Power BI exports dashboards to a nice pdf as well.  So not a bad bit of work today.

Thanks for reading.

Dave

 

* On reflection it could have just been YYMMDD.  I am not convinced I will be working on this project in 2120 when the datekeys start being reused. 

 

Edited August 13, 2020 by HN50
Spelling

[Tomatobro]

latest data. That short pulse still appears at about the same time but its origins are unkown.

There are some very short loss of signal events. The receiver loss of power alarm was not triggered so these are caused by the transmitter going off line

The graphs are displayed in reverse chronological order (last is first)

Edited August 15, 2020 by Tomatobro
new data

[HN50]

On 15/08/2020 at 12:31, Tomatobro said:

latest data. That short pulse still appears at about the same time but its origins are unkown.

There are some very short loss of signal events. The receiver loss of power alarm was not triggered so these are caused by the transmitter going off line

The graphs are displayed in reverse chronological order (last is first)

 

Hi,

Looking good, is the new timing unit working as you hoped?  I have noticed a few sudden drops in signal strength myself so had wondered if the transmitter had stopped transmitting momentarily.  And that peak is interesting, very regular, very abrupt and yet not obvious what it is..?

There was a C class flare yesterday but I did not register it, I don't know if that is because the radio is not sensitive enough or if it was pointing away from Earth.  But it is good to know that some solar activity is happening.  I used the NOAA data I had imported to tweak my dashboard.

Not done too much myself, on a tangent I have been exploring KiCAD with a view to trying to make either another VLF radio (basically a copy of one I have made before on stripboard) or a bat detector.  I have also downloaded the Muon detector and am looking to get the pcb made for me.

Thanks for reading,

Dave

[Tomatobro]

My original design was based on starting the recording at a precise time (say 12:00:00) and the software takes 10 readings over 29 seconds and then averages them. It then waits for a pulse from a timer (at 30 second intervals) to write the data to memory. The idea being that the exact timing of each data point could be inferred from its memory position i.e. data point 120 is one hour (13:00:00) from the start point. I added an external port so that I could connect a digital counter so I could see if the time from a radio controlled clock tallied with the count number from my own internal timer. With this setup, even after careful calibration of my clock timing loop errors crept in over 11 days of stand alone data logging.

If the power fails then on startup the processor reads the first memory location and if its non zero then reads through the memory till it finds the first blank (actually the number is 255 not zero) and then starts recording from that memory address. It switches on a blue led to warn me that the receiver shut down and restarted so the timing will not tally.

You can see a blue faced radio controlled clock in the picture and I hit on the idea of having a radio controlled clock module rotate a double ended beam breaker connected to the seconds hand which passed through an opto switch. The double ended beam breaker guaranteed 30 second pulse spacings with 100% accuracy (or so I thought!)

I connected up my digital counter and set it running. After a few days I calculated the timings from the counter and found to my surprise that a discrepancy of exactly 10 minutes had occurred. A reset and run showed the same loss of time over 24 hours. What happens is in the early hours of the morning the radio clock stops and recalibrates its time from the radio signal.

I then did a similar test with a modified battery clock module and found that this unit looses no more than 2 seconds over 7 days. The opto switch is mounted on an arm that allows me to calibrate the start time with the bench clock and to advance or retard  a second or so during the 11 days of data logging. Also maintains time if the main power goes off

Crude but as it turned out very effective.

[HN50]

Just had a quick look at the dashboard while passing (I am not obsessed) and the end of day peak appears to appearing several hours ahead of normal.

I pulled in the last few hours of NOAA data and there is something happening at the moment - the start of the peak was ahead of the event but there is something happening, even if it is small.  

The start of the peak does coincide with a jump in temperature up in my roof (first screenshot) thinking back the weather changed abruptly and went from being cold to full sunshine.  I don’t know if that would explain the earlier part of the peak (it is only about 3 degrees change).  Anyway, will keep watching.

Edited August 16, 2020 by HN50
Spelling

[Tomatobro]

I will stop mine early and download the data to see if I see the same numbers in the data

[HN50]

18 minutes ago, Tomatobro said:

I will stop mine early and download the data to see if I see the same numbers in the data

Cool, am interested to see what was detected by your system.

[Tomatobro]

will post my data just after noon today

Usual washing machine interference on the 15th, unusual pattern during the night of the 17th.

Looks like the LGM's are trying a different time to get my attention (spike just after 12:00 on the 17th)

Edited August 17, 2020 by Tomatobro
graphs added

[HN50]

Thanks for posting your data.  I am not quite sure what that hump is I was getting between about 16:00 and 19:00 is but I am not sure it was a flare - the sharp end of day peak does match what you got.

Today seems more settled too (there seemed a lot of transmitter downtime yesterday);

Over the last couple of hours I have noticed a few downward ticks in the line though that happen around jumps in the GOES data. 

Though they do also tally with apparent temperature jumps in the project box that the Arduino is enclosed in... 🤔

[HN50]

I am not saying that it is all perfectly matching but having loaded the last 24 hours of GOES data more than a few of the downward ticks in my VLF data correlate with a spike in X-ray flux...

Edited August 19, 2020 by HN50
punctuation

[Tomatobro]

The BAA have not published their July SID report yet but await it with interest. My understanding is that we see a sudden change in signal followed by a slow recovery as the ionosphere rebuilds its reflective layer. Similar to around 15:00 on the 18th in the above graph.  All the graphs of SID's I have seen show a positive rise in the signal which I assume is what we see at dawn and dusk as the ground and sky phases change but to be honest I am just guessing at this point.

The June BAA report says that non of the stations saw a SID.

[Tomatobro]

The BAA have just published the July SID report, with small signal variations seen on the 4th and 8th of July. Frequencies monitored were 19.6 khz and 22.1 khz. Other than that its been a quite month.

My data for the last 7 days show nothing special.

[HN50]

On 25/08/2020 at 12:59, Tomatobro said:

The BAA have just published the July SID report, with small signal variations seen on the 4th and 8th of July. Frequencies monitored were 19.6 khz and 22.1 khz. Other than that its been a quite month.

My data for the last 7 days show nothing special.

Hi,

The last week has been pretty quiet, yesterday there seemed to be quite a bit of activity with the transmitter going on and off, and a few days before that the signal strength seemed to jump somewhat but is back down again.  There was about half an hour on the 27th from about 18:30 when I stopped receiving data at IoT hub but I will just keep an eye on that, 35 minutes over about three weeks isn't too bad.

I have been tinkering a bit with Azure behind the scenes but otherwise not so much on the radio.  I might change the sampling when I make a measurement as I take 1000 over the course of a second so I might spread it out a bit more over maybe 20 seconds to try and reduce the noise. 

I have set up my own Github account for storing source code as my laptop seems to be becoming a little temperamental and so far I have been e-mailing code to myself.   The pcb for the Muon detector arrived in a week; I was expecting it in 3 so I need to start sourcing parts.

Dave

 

 

[HN50]

Activity so far today seems more stable.

Dave

[Tomatobro]

Re the Muon detector I see that a USA source has the plastic blocks listed on Ebay......