Jump to content

SkySurveyBanner.jpg.21855908fce40597655603b6c9af720d.jpg

HN50

Members
  • Posts

    138
  • Joined

  • Last visited

Everything posted by HN50

  1. I went back and re-read how your set up works, that combination of mechanical and electronic is fascinating, I could not have come up with that myself. Is it all working as expected now? Did you leave it on 24 hour per .jpg? It has been a bit of a heavy work week so unfortunately I have not done a whole lot myself. Yesterday I sat down and amended the Arduino code to measure the minimum signal strength at the same time as I am currently measuring the average. The idea is that the transmitter should be detectable anyway however sudden jumps in signal strength due to noise from lightning strikes will effectively be ignored. So far it looks quite similar to the average but I will be interested to see the difference when there is a flare as that 'C' class flare on the 29th May caused the signal strength on my setup to drop. So I would expect the min signal to respond to a sudden drop more than the average. It occurs to me though that it won't respond to a sudden increase in signal strength and so you would only see that in the next minute. Swings and roundabouts. I will just leave it accumulating data to see what happens. Otherwise that is it, bit of a nothing post today I am afraid! I did get time yesterday to order the first set of components for the Muon detector so am hoping they should arrive this coming week. I will start a new thread in this section. Thanks for reading, Dave I also changed my report so that I get a plot of the last 4 weeks. I can't figure out why but PowerBI is determined that 12:08 on 13/09/2020 is two weeks ago and not one. SQL query is correct but I can't work out why it is plotting wrong. A week tomorrow it falls off the end of my report anyway so I might just chalk it up to 'just one of those things'... There has been a bit of an uptick in activity throughout the day as recorded by the GOES data. Will keep an eye on that.
  2. Miles Dyson didn't quite appreciate what he had built in that film so be careful if you try and switch it off..! * That is weird, I wonder why that happened? The light on the MKR flashes every time I take a measurement when running under USB power, but when under external power flashes once as expected and then stays on permanently. I have added the command to turn it off but it does exactly 3/8 of nothing... With work over the last week or so I have not done that much, but I got round to checking the data today. Helpfully the transmitter is back up and running after that interruption last week. The signal strength has gone up again after the maintenance period on the 10th so I wonder if they are transmitting at a greater power. More helpfully though I have been able to see the effect that my change in sampling has had (i.e. 1000 reads over 30 seconds rather than 1). It isn't scientific but I took two of the same day of the week n either side of my change to compare. Dark blue is with the longer sample window. It certainly looks smoother so I think is working though time will tell if it smooths out real fluctuations and unhelpfully the GOES data shows the sun has not been up to much. UKRAA suggested I could take the minimum value over the sample period as well so I might change the code to record that as well. So as ever I will leave it and watch. Otherwise I need to get ordering the Muon detector components. Thanks for reading, Dave * there will be no more film references. I only know the Terminator films and I don't see how Jurassic Park is relevant.
  3. Hi, Thank you for posting that is really useful to know, your plot looks similar to mine. So I think my radio is working properly but the transmitter isn't. I have turned logging on again so will see how this pans out. Thanks, David
  4. Thanks! Not looking entirely good...
  5. The code change has worked, analysis with Mk 1 eyeball seems to suggest a smoother output, time will tell if that also smooths out real events. Although I am not entirely keen on the plot today; I hope that a) the transmitter has been offline for an extended period rather than b) something is wrong with my radio...
  6. The transmitter seems to be down today so there is not much to see. As a result I have been making some code changes instead. I have simplified it by removing the alarm and pin interrupt. When I have finished taking measurements/sending to the MQTT broker I check the time and the number of seconds past the minute, subtract from 60 and send the Arduino to sleep for that amount of time. Not sure why I have not thought about this before, I think the low power library used with the MKR 1010 is a bit more flexible and the sleep time is variable. I have also changed how I am sampling data. Previously I was taking an average of 1000 measurements, each with 1ms between them (i.e. all done in 1 second). I mailed UKRAA and they suggested I might take measurements over a longer time to reduce lightning interference. So now I am taking 1000 with an interval of 30ms (i.e. over 30 seconds). I will be interested to see how this affects my results so as ever I will leave it running and see what happens. A bit early to tell and it would be good to see the affect on a 'normal' day. Thanks for reading, Dave
  7. Hi, That is where I went, I looked around in the UK but went for the one I found on Ebay. Seller was really helpful, I bought a block and the optical grease and with p&p it came to about £40. I did however get stung £14 for import duty. Dave
  8. Activity so far today seems more stable. Dave
  9. Hi, The last week has been pretty quiet, yesterday there seemed to be quite a bit of activity with the transmitter going on and off, and a few days before that the signal strength seemed to jump somewhat but is back down again. There was about half an hour on the 27th from about 18:30 when I stopped receiving data at IoT hub but I will just keep an eye on that, 35 minutes over about three weeks isn't too bad. I have been tinkering a bit with Azure behind the scenes but otherwise not so much on the radio. I might change the sampling when I make a measurement as I take 1000 over the course of a second so I might spread it out a bit more over maybe 20 seconds to try and reduce the noise. I have set up my own Github account for storing source code as my laptop seems to be becoming a little temperamental and so far I have been e-mailing code to myself. The pcb for the Muon detector arrived in a week; I was expecting it in 3 so I need to start sourcing parts. Dave
  10. I am not saying that it is all perfectly matching but having loaded the last 24 hours of GOES data more than a few of the downward ticks in my VLF data correlate with a spike in X-ray flux...
  11. Thanks for posting your data. I am not quite sure what that hump is I was getting between about 16:00 and 19:00 is but I am not sure it was a flare - the sharp end of day peak does match what you got. Today seems more settled too (there seemed a lot of transmitter downtime yesterday); Over the last couple of hours I have noticed a few downward ticks in the line though that happen around jumps in the GOES data. Though they do also tally with apparent temperature jumps in the project box that the Arduino is enclosed in... 🤔
  12. Cool, am interested to see what was detected by your system.
  13. Just had a quick look at the dashboard while passing (I am not obsessed) and the end of day peak appears to appearing several hours ahead of normal. I pulled in the last few hours of NOAA data and there is something happening at the moment - the start of the peak was ahead of the event but there is something happening, even if it is small. The start of the peak does coincide with a jump in temperature up in my roof (first screenshot) thinking back the weather changed abruptly and went from being cold to full sunshine. I don’t know if that would explain the earlier part of the peak (it is only about 3 degrees change). Anyway, will keep watching.
  14. Hi, Looking good, is the new timing unit working as you hoped? I have noticed a few sudden drops in signal strength myself so had wondered if the transmitter had stopped transmitting momentarily. And that peak is interesting, very regular, very abrupt and yet not obvious what it is..? There was a C class flare yesterday but I did not register it, I don't know if that is because the radio is not sensitive enough or if it was pointing away from Earth. But it is good to know that some solar activity is happening. I used the NOAA data I had imported to tweak my dashboard. Not done too much myself, on a tangent I have been exploring KiCAD with a view to trying to make either another VLF radio (basically a copy of one I have made before on stripboard) or a bat detector. I have also downloaded the Muon detector and am looking to get the pcb made for me. Thanks for reading, Dave
  15. The scintillator and optical grease arrived the other day, will start looking at the PCB in KiCAD so hopefully will make a start soon. Not quite sure which subject it would go under though? My progress... I am on holiday at the moment and I spent yesterday making some back-end changes. Rather than logging the date and time as one column, I have reduced them to two numbers - for date an integer in the format YYYYMMDD *, and for time the number of minutes past midnight. I did this as I would like to bring in additional data (the proposed Muon detector, GOES x-ray data from NOAA, maybe another VLF radio...) and I want to have common date and time calendar tables in the database to join to them all to. That should make them easier to plot together. I also removed the in-Arduino moving average as it was not calculating and in any case that could be calculated in the database. Date table looks like (I have it going to 2040); Time table looks like; If I need them together I can then create a view over both that joins on DateKey and TimeKey. I have also been practicing Power BI more and have been able to replicate the charts I have been running in LibreOffice. I found the breadth of choices available within charts slightly narrower in Power BI but the big difference is now how much faster I can display data without having to shuffle columns around in LibreOffice. I have loaded into the database some GOES X-ray flux data to see how easy it is to display it alongside my data. The feed isn't automatic so I will have to think about how I do this as it too gets updated every minute. I also discovered Power BI exports dashboards to a nice pdf as well. So not a bad bit of work today. Thanks for reading. Dave * On reflection it could have just been YYMMDD. I am not convinced I will be working on this project in 2120 when the datekeys start being reused.
  16. It has been a bit hot to do anything today so I have been sitting in the shade a little. The load is still running so the process seems to be reliable; I have however been giving some thought to the diurnal profile I get and why this might be. I found a Stanford University website and it mentions the sunrise and sunset behaviour is caused by sunlight sweeping across the path of the radio before the sun has risen at either the transmitter and receiver locations. Likewise it mentions that the creation of the D layer attenuates waves that pass through it on the way to being reflect by the E layer. http://solar-center.stanford.edu/SID/activities/ionosphere.html To that end I went and found a spreadsheet that calculates solar elevation by minute (well every 3 minutes) for a given lat/long and got this into the database for my local position as well as at the location of DHO38. I then spent some time comparing this to signal strength data for 09/08/2020 and trying to think about what I was seeing and how it may relate to what I think is happening. Shots are below, would be interested to see what others think. I have used terms like civil and nautical twilight only as I noticed a few of the changes occur around their boundaries. Thanks for reading, Dave
  17. Despite my Eeyore-ish outlook on things it is still running; I want to add a column to my data that identifies today and yesterday, at present this chart will just plot all data for evermore. I can have a view in my database that does this, I will also add a flag to identify the last 7 days for a separate chart. Something that has changed is that I keep checking my laptop to look at activity - now I don't have to brave 45 Celsius up in the roof (the graph shows that 15:00 yesterday was a little unpleasant!) or spend time shuffling data around in LibreOffice it is much easier to have a quick peek at what is going on. Dave
  18. Hi, Get a new washing machine, I have been setting the countdown timer on ours (and the dish washer) so that it runs in the middle of the night when I can't detect SIDs anyway. I am not sure if it has been interfering but better safe than sorry. I am interested by your timing unit - what are you using? I take it it is a radio controlled clock of some sort? I fired up Power BI to have a look at what is happening (and more importantly to check the logging is still working!), and it is. I need to experiment a bit with the dashboard as I want the date on the x axis as well. On a tangent I have also been looking at the CosmicWatch home Muon detector here http://www.cosmicwatch.lns.mit.edu/detector. The scintillator plastic has arrived along with a customs bill of £15. Would be a separate build log but further down the line might be good to get that data displayed alongside this VLF data. Thanks for reading, Dave
  19. Did you find out what the spike was caused by? How has the logging been going? My progress has been a little sparse without any data being captured for nearly a month. The issue is that I have been trying to get to grips with Azure - I can see data coming in to IoT hub, I can see my SQL database at the other end but trying to get the data into it seemed to elude me. There are a few options - tutorials on Stream Analytics, event hubs, event grids etc that seem to offer what I need it to do but I was missing something. Plus trying to get it working before my free trial finished and only getting to do it at the weekend was proving to take the fun out of it. Fortuitously it turns out that with software licensing at work we get some free credit each month to experiment so that took the heat off things. I decided to initially get it working with just the Azure web interface and not resort to programming via Visual Studio. After having put it aside for a week I sat down today at a leisurely pace and worked through the tutorial below; https://www.mssqltips.com/sqlservertip/6335/how-to-capture-iot-data-in-azure/ Yes, Stream Analytics costs more but at this stage I would just like to get it working. A colleague says that you can program functions that work out much cheaper so that is a future avenue of inquiry. I was able to get the job working but I kept getting cryptic errors that related to permissions to access the database and data types that the stream analytics job can handle (it can't do 'date' and 'time' separately, only 'datetime' so I had to merge the previously separate columns together in the MKR1010 code). After a few hours I ran the following code; That does not look much but that was a back-of-the-net moment. First query is a select from the target table, it then waits 3 minutes and then runs the same query again. There are now another three rows - data is being loaded. I got the radio back up in the roof and have left it running this afternoon. I downloaded Power BI and have put together a first attempt chart to see what is happening. It is still running so I will leave it overnight and into tomorrow to see how things look. It did occur to me that as you can't detect SIDs at night anyway it might(?) be possible to schedule Stream Analytics to stop at about 20:00 and start up at 06:00 and to load the intervening data that has built up. So after a bit of a hiatus things are getting going again - I am now logging my data to a database. I am sure there will be other glitches but they can be overcome. Thanks for reading. Dave
  20. Hi, Glad you got it resolved, your plots are looking good. It is interesting the washing machine motor has such an impact on the output. My progress this week; I pulled out the SD card and had a look. As previous weeks it has been another quiet period down here in solar minimum GIF is below; The heat from the linear regulators did not cause any problems so that is something. The bigger piece of work has been that I took the test script that I was using to connect to Azure Hub and merged my radio logger code into it. This I have managed and the Azure prompt showed me the following; Excuse all the red redactions but many of the names have to be globally unique. Anyway the data coming in looks like my data, so I think the code splice worked. I have found that the data payload data needs to be in JSON format, so a bit of tweaking my sketch now gives an output of; {"D":"2020-07-12","T":"15:49","V":0.001,"K":299.9,"A":0.0009} ("A" is a 5 minute moving average I am calculating in the Arduino as you pay by activity in Azure) That being in JSON then makes the query that should copy it to SQL easier. I say easier as I had the job set up and running (so it said) but the data was not getting into the database, so I think there is a setting wrong somewhere. However I am using a tool called Stream Analytics in Azure which is Platform as a Service. It (supposedly) makes setting the job up quicker but you pay for it by the hour, and the default is 3 units running it at £0.082 per hour. So if you are running 24 hours a day... I read that a much cheaper way to do it is with a tool called Azure Functions where you have to write the whole process yourself, though there will be much more of a learning curve to that. Anyway at this stage I am just experimenting with the free trial* and will just keep investigating. Way back though my goal had been to see if I could get the logger writing to a database in the cloud. If I can get past this last glitch then I will have managed to go from antenna to database in one end to end process. I have not decided on the final means of storing and visualizing the data but I will be pleased to have it working as prototype. I think I first thought about this 4 years ago! Thanks for reading, Dave * As a backup I have had a look around at other IoT platforms :D
  21. Hi James, Thank you, this project has been good fun so far. Yes, the data I am plotting is just the raw numbers there is no other data manipulation being made to it. I think I might try a moving average on the data as you suggest as it would be good to see whether the flare is still visible or if it gets smoothed out. I have been logging at the minute level but I notice that you log every 10 seconds - how do you find the trade-off between increased temporal resolution of versus the data volume? What would be the longest interval between measurements you would consider? I had wondered about dropping the frequency to maybe once every two minutes (I might simulate with data first...) as the C class flare I detected caused the signal to drop over the course of two minutes with a recovery of maybe half an hour, so two minutes might not miss too much activity(?). Although having detected only one flare so far I might want to see what happens for others, but once every two minutes would cut my data capture in half to 720 measurements a day. Hi, That is looking good. Stupid question, but how have you connected your antenna to earth? Looking at the antenna connection in the pcb, although both ends of the loop plug in to it, it looks to me like only one end goes into the amplifier. Did you just add a lead from one of the loop wires on your antenna to an earth connection? Did you need to do any antenna re-tuning? I ask as the affect on your plot is quite marked, I like tinkering and there is a water pipe not too far from my setup. I have not been able to do too much this weekend as with the good weather I have spent some time down on the allotment making raised beds and keeping wood pigeons off the sprouts. The male to male power lead arrived so I am now able to use the power in socket on my project box and fix the lid down. I realised I have not drilled any cooling vents above the linear regulators so I will see what that has done to the data when I pull it out tomorrow. On a technically interesting, but visually dull, level I have been playing with the wifi on my MKR 1010. I set up an Azure subscription* and worked through the page below; https://create.arduino.cc/projecthub/Arduino_Genuino/securely-connecting-an-arduino-nb-1500-to-azure-iot-hub-af6470 There is not much to see but I now have an Azure hub that is able to receive data from my MKR1010 and running the Azure shell command prompt I can see the messages coming in and the data within it, which with the example Arduino sketch is just the message "Hello". I also was able to send a message back to the MKR 1010 and see it appear in the serial monitor. Next I need to see about merging at least some of my VLF logging code into the example sketch to see my timestamp, voltage and temperature measurement coming in. This is all new to me so as an initial goal I found the page below. It does not address long term storage of the data in a SQL database but it would be good to be able to view my data in real time. https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-get-started-with-azure-stream-analytics-to-process-data-from-iot-devices It also occurred to me I should check that the wifi signal in the roof is strong enough to make any of this possible! If not I may need to buy a wifi range extender and set it up in the room below. So not that much to show today but what I have been investigating is rather exciting. Thanks for reading, Dave * I want to keep a close eye on the billing as I don't want to get hit by any unexpected/unpleasant costs...
  22. And now fitted in a project box for a little protection.
  23. Hi, Thanks for the suggestion, I got the sleep mode to work (more later) and have put in a loop that reads 50 times before I actually start taking measurements. My work this week. I pulled the data out yesterday from the new MKR 1010 rig to see how things went... It has been running all week as expected despite the furnace-like temperature of nearly 40 Celsius up there towards the end of the week. So it is running successfully. Eyeballing things I could not decide if the data looked noisier than with the Barebones Arduino, so I tried to analyse the data by plotting standard deviation of the two rigs by hour. Things are a bit all over the place at night, while the Barebones data has a greater stdev around the maintenance window time; this only happened once in the last week for the MKR 1010 data. But between 10:00 and 17:00 when the signal plateaus I think it is not as clear, but perhaps the MKR 1010 does have a slightly greater standard deviation than the Barebones output and I would therefore assume noisier signal. The Barebones data is everything from 5th May, the MKR 1010 just 5 day's worth so I might collect more data before I pass judgement. I could look to code a 5 minute moving average within the data using an array to try and smooth the output I suppose. Otherwise I now have the code going into deep sleep between taking measurements. I thought it was not working as when running it from the IDE I would get my start up messages in the Serial monitor and then nothing. When I looked at the SD card though the Arduino had been logging data, what I think happens is that when the MKR 1010 goes to sleep it loses its connection to the previously assigned COM port and when it awakens does not reconnect. This is different to the Bareones Arduino which appeared use the same COM port on wake-up and so kept outputting its results to the Serial Monitor. The upshot is that no new event data gets posted to the Serial monitor for the MKR 1010 which is why I thought the sleep mode was not working. I also find it 'a bit difficult' to compile code to the MKR 1010 when it is asleep, so it now only goes to deep sleep when it cannot find the Serial port (i.e. under battery power). A slight reverse of the wait for Serial problem I had the other week. if (!Serial) { LowPower.deepSleep(); } Aside from that the spare RTC I bought seems to have a problem in that I need to look at (or cut losses and get another for a couple of quid) and I have just about all the parts now to finally get it into a plastic casing. I will look to assemble the case for it today. Thanks for reading. VLF_MKR_20200627.TXT
  24. @7170 Sorry, I missed that. Yes, it is always positive, either 0-2.5V or 0-5V. Dave
  25. It is now logging in the roof, so I will give it a week and see what comes out... Data up to today below. VLF_T2_20200621.TXT
×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue. By using this site, you agree to our Terms of Use.