Sorting data from Meteor detection

[dyfiastro]

Good evening everyone

I have recently setup a basic meteor detector and I am trying to process the data.
I need to try and grab the total count (being the highest number in each hour) a little easier than doing it by hand. Can anyone help?

An example of the data format is as follows, this is imported into excel as separate columns.

06:09:50,5
06:21:40,6
06:21:47,7
06:24:02,8
06:35:39,9
06:47:55,10
06:48:37,11
06:58:02,12
07:25:47,1
08:21:26,1
08:43:17,2
08:47:16,3
08:48:00,4

The end result that I am looking for would be to have something that could be then plotted into a graph over a 24 hour period or even longer such as this

00:00 - 01:00           3
01:00 - 02:00           8
02:00 - 03:00           10

Any help with this would be great

[Tomatobro]

In the early days I used some software called Bandicam to record the screen in half hour chunks. This produced 20 half hour videos over a 10 hour period. I then fast forward through each video looking for meteor traces and made notes on a form which I then analysed. In my view this was the most accurate method as you could eliminate non meteor events such as lightning strikes, space junk, ISS passover, interferance etc from the data. There is a small charge for Bandicam.

You can also download HROFFT or RMOB software which does the analysis for you but I have found that its not as good as the Bandican recording method . The software is free.

I now use a fully automated setup which does it all for me but I often set Bandicam running as well as there is nothing like seeing the event as it happens. Be aware though that at high screen resolutions you will need 15GB of free disc space for each session.

I just remembered that sometimes unattended Laptops do unexpected actions so I always disable the internet connection and activate a piece of software called "Caffeine". This is free software and what it does is simulate a (no longer used) regular timed keypress to fool the laptop into thinking it is being used by a Human.

Edited August 1, 2019 by Tomatobro
more info

[dyfiastro]

3 hours ago, Tomatobro said:

In the early days I used some software called Bandicam to record the screen in half hour chunks. This produced 20 half hour videos over a 10 hour period. I then fast forward through each video looking for meteor traces and made notes on a form which I then analysed. In my view this was the most accurate method as you could eliminate non meteor events such as lightning strikes, space junk, ISS passover, interferance etc from the data. There is a small charge for Bandicam.

You can also download HROFFT or RMOB software which does the analysis for you but I have found that its not as good as the Bandican recording method . The software is free.

I now use a fully automated setup which does it all for me but I often set Bandicam running as well as there is nothing like seeing the event as it happens. Be aware though that at high screen resolutions you will need 15GB of free disc space for each session.

I just remembered that sometimes unattended Laptops do unexpected actions so I always disable the internet connection and activate a piece of software called "Caffeine". This is free software and what it does is simulate a (no longer used) regular timed keypress to fool the laptop into thinking it is being used by a Human.

Thanks for the reply.
I have sdr# piped to spectrum lab which is then monitoring for meteors, logging, screenshotting and recording the audio.
It also outputs as csv file that I can then import into excel. Its filtering the data from the csv file that is the main issue.

[Tomatobro]

Oh ok. I was not sure where to pitch it. Can't help with the csv file I'm afraid

[Geminids]

I note from your website that you are a keen Astro Photographer.  Are you as familiar with Radio Meteors?  I note you take the csv files from Spectrum Lab into an excel spread sheet.  The solution to hourly contain will probably involve some programming.  I use another app to do the counting using its internal programmining language and import it to Excel for what is commonly called a colourgramme plot of days against hours with the colour of the cells indicating the count.

For Excel programming the count routines can be achieved in Visual Basic which is quite a powerful language. Are you familiar with VB?

Edited August 1, 2019 by Geminids

[dyfiastro]

On 01/08/2019 at 16:33, Geminids said:

I note from your website that you are a keen Astro Photographer.  Are you as familiar with Radio Meteors?  I note you take the csv files from Spectrum Lab into an excel spread sheet.  The solution to hourly contain will probably involve some programming.  I use another app to do the counting using its internal programmining language and import it to Excel for what is commonly called a colourgramme plot of days against hours with the colour of the cells indicating the count.

For Excel programming the count routines can be achieved in Visual Basic which is quite a powerful language. Are you familiar with VB?

Hi and thanks.
Radio is very much new to me so learning on my feet so to speak.

I did Basic and Visual basic back as a teenager in the 90's but had to give it up (got banned from learning it any further for a few instances at secondary school where it may of been used for not so positive uses)
I will take a look and see if I can pipe the output from spectrum lab in a better way so that its easier.

[IanL]

You could certainly process the data in Excel by using a PivotTable. You'd need to import the data, make sure each column has text label in the first row. Now add a column on the left with a formula to extract just the hour part of the timestamp (look at the date and time functions). Then select all the data and insert a pivot table.

You drag the hour column to the rows section and the meteor count to the values section. You'll need to change the values field settings to display 'max' rather than the default 'sum'. You should now have a table of hours and maximum meteor counts, no programming required. This is all standard Excel stuff so plenty of tutorials online on each step I've described above.

Alternatively change your spectrum lab conditional actions to generate the hourly meteor count at the end of the hour rather than for each meteor event. Take a look at my post linked below for an idea of how to do this.

https://www.blackwaterskies.co.uk/2019/01/radio-meteor-scatter-improved-spectrum-lab-conditional-actions/

Edited August 6, 2019 by IanL

[Hawksmoor]

Hi dyfiastro

I have been messing about with the data I have collected from my software define radio- spectrum lab meteor catcher. I have used excel as a framework to hold the files I was interested in analysing and exported the result as a csv file. This I imported into INZite for data analysis freeware (available from Auckland University) . This is a non programmer friendly front end for the powerful R software. Once transferred to INZite it is relatively easy to clean and filter the data and then create a number of graphs to suit yourself. My son is an academic and uses R all the time to analyse extremely large data sets. INZite is much easier to use. The OU has a free short course which is run by the Stats Dept of Auckland University. I found this course both helpful and interesting but there is an online manual for INZite which is pretty comprehensive. I have recently put some info and graphs on my LVST Google sites page 're 2016 Perseid Meteor Shower' If any of this is of sounds useful please follow the link:

https://sites.google.com/yahoo.co.uk/thelvst/2016-perseid-meteor-shower

 

[dyfiastro]

7 hours ago, IanL said:

You could certainly process the data in Excel by using a PivotTable. You'd need to import the data, make sure each column has text label in the first row. Now add a column on the left with a formula to extract just the hour part of the timestamp (look at the date and time functions). Then select all the data and insert a pivot table.

You drag the hour column to the rows section and the meteor count to the values section. You'll need to change the values field settings to display 'max' rather than the default 'sum'. You should now have a table of hours and maximum meteor counts, no programming required. This is all standard Excel stuff so plenty of tutorials online on each step I've described above.

Alternatively change your spectrum lab conditional actions to generate the hourly meteor count at the end of the hour rather than for each meteor event. Take a look at my post linked below for an idea of how to do this.

https://www.blackwaterskies.co.uk/2019/01/radio-meteor-scatter-improved-spectrum-lab-conditional-actions/

Thank you very much indeed, I will take a look and get that done.

6 minutes ago, Hawksmoor said:

Hi dyfiastro

I have been messing about with the data I have collected from my software define radio- spectrum lab meteor catcher. I have used excel as a framework to hold the files I was interested in analysing and exported the result as a csv file. This I imported into INZite for data analysis freeware (available from Auckland University) . This is a non programmer friendly front end for the powerful R software. Once transferred to INZite it is relatively easy to clean and filter the data and then create a number of graphs to suit yourself. My son is an academic and uses R all the time to analyse extremely large data sets. INZite is much easier to use. The OU has a free short course which is run by the Stats Dept of Auckland University. I found this course both helpful and interesting but there is an online manual for INZite which is pretty comprehensive. I have recently put some info and graphs on my LVST Google sites page 're 2016 Perseid Meteor Shower' If any of this is of sounds useful please follow the link:

https://sites.google.com/yahoo.co.uk/thelvst/2016-perseid-meteor-shower

 

Thanks for the information and heads up, I will take a look into this as I think it maybe helpful with a few other projects that I have coming up.

[Geminids]

On 06/08/2019 at 16:50, IanL said:

 

Alternatively change your spectrum lab conditional actions to generate the hourly meteor count at the end of the hour rather than for each meteor event. Take a look at my post linked below for an idea of how to do this.

 

If you pick up interference and moon bounce as I do, there could be trouble with getting a true count of meteor actvity if you do it in Spectrum Lab - unless, that is you can do this clean up in SL.

Edited August 7, 2019 by Geminids

[IanL]

1 hour ago, Geminids said:

If you pick up interference and moon bounce as I, there could be trouble with getting a true count of meteor actvity if you do it in Spectrum Lab - unless, that is you cna do this clean up in SL.

Agreed it is problematic when the only trigger is exceeding a set SNR threshold for a set period. One will catch all sorts of spurious signals plus miss weaker ones. I haven't had time to look at or test other filtering mechanisms or triggers, though some others have produced scripts with such concepts but not seen any rigorous attempt to prove their accuracy.

Bottom line is that manual review of screenshots or recordings is necessary to get a 100% accurate count, but even so might be quicker to start with a total and subtract obviously bad data points rather than count good ones.

I also take the view that for longer term trends and amalgamation of data across multiple observers it will come out in the wash, i.e. if you're over recording it is likely to have some degree of consistency over time so trends, showers vs. sporadics, etc. would be identifiable.

By way if illustration, earlier in the year there was an interestingly large spike in activity for a couple of hours that several UK RMOB contributers recorded. On examining the data it was just a tropo or similar atmospheric condition allowing detection of the CW transmitter.  I didn't sweat for hours cleaning up data, rather waited for the data to suggest an investigation was needed.

[Geminids]

Well, I guess you could leave the Moon bounce data in ....

Just deleted wrong stream attributions.  Any thoughts on what the activity is?

Edited August 9, 2019 by Geminids

[IanL]

Can you share more details of your set-up, what software you are using, etc. How are you doing the filtering and plotting - software application, Excel, something else, e.g. interested in your Moon Bounce intelligent filter... Not that I see a lot of that here - a few tropos and maybe the odd bounce but on the whole they are much fewer and further apart - maybe 2 or 3 short events a month I'd guess.

[Geminids]

 

1 hour ago, IanL said:

Can you share more details of your set-up, what software you are using, etc. How are you doing the filtering and plotting - software application, Excel, something else, e.g. interested in your Moon Bounce intelligent filter... Not that I see a lot of that here - a few tropos and maybe the odd bounce but on the whole they are much fewer and further apart - maybe 2 or 3 short events a month I'd guess.

I am away on holiday in a few days and will give you some brief details of the software.  For all of my data manipulation, ie reading the Spectrum Lab (SL) logs, filtering, distributions, and plots I use Wavemetrics' Igor Pro which is a plotting and analysis application with a powerful programming capability.  I have written  routines using Igor Pro to do all of the tasks automatically. The top plot above is done with this - plotting is very flexible.

As you can see I do get a significant ammount of Moon Bounce events in my data; this may be because I use masthead pre-amps on my systems.  (Incidentally a snapshot of part of the RF circuit is shown on my Icon).  To filter this out I use the programme EMEPlanner to provide Moon Doppler shift at 10 minute intervals for the month.  I copy and paste the data from EMEPlanner into  Excel and run a VBA macro to format it.  It is then imported into Igor Pro.  When I have loaded some of my SL data log into Igor Pro I run my filter macro, which checks each event in the log for certain criteria.   The Moon part of the filter compares the time and frequency of the particular log entry with the interpolates  Moon Doppler shift data and, if the frequency of event matches within limits, it is filtered out.  In the example above the 4300 Moon events are filtered from total log of around 37000 events in a second or two.  The technique does require accurate time and frequency which I achieve.   It seems to work quite well and I intend to compare the patterns of my antennas with these Moon data.

The Colourgrammes are done in Excel because it is easier to use the cells to construct it.  And, as you might have guessed, the counting is done in Igor Pro, exported and then imported to Excel where a VBA macro takes the data and makes the plots shown above.

Edited August 9, 2019 by Geminids

[Stub Mandrel]

Here's my spreadsheet that I used to analyse last year's data from SpectrumLab. I just counted the hourly totals manually.

My view is that false detections will be of two kinds: Totally random, that just lift the baseline, and missed or double detections which will be random but proportional to the amount of meteors. Neither of these has a significant impact on the proportionality of the readings, so take the absolute numbers with a pinch of salt, although inspecting screen captures suggests there are few false detections.

As no detection setup will get more than a percentage of meteors and it's impossible to be sure exactly what your area of search is, I think the only scientifically valid result for  is comparing readings at different times with the same location and setup (and possibly weather). That said it's interesting to compare detections of really big meteors and to note things like ISS and plane passes.

Perseids 2018.xlsx