21cm band. Refurbishing the dish with a new Cantenna.

[Victor Boesen]

12 minutes ago, ZiHao said:

Data is received with my python code, the raw data is first filtered by a median filter of size 30, or even higher IIRC, but I didn't use those in my attached data. I gave up on this clone as even though the sharp spikes like the DC spikes is removed, I am left with some broadband peaks which is very annoying to work with. Now with the genuine rtlsdr, I am using a median filter of size 15, and it got rid of the DC spike, leaving a clean spectrum. 

Ahhh that makes sense then  I will probably need data received with my software as the processing I'm using is a little different so results wouldn't be comparable.

[SteveBz]

So here's a small step.  Broadly the arcs are 'spiral-like'.  Broadly, too, they centre at the Galactic Centre.  I think I need to step through the calculations again, because the distances don't look right and the range of velocities doesn't seem large enough.  The signs of the velocities, however, look about right.  I've used automatic Gaussian fitting with a fixed Sigma, which is probably not realistic.  I hoped it might work because I hoped the thermodynamic temperature range of the bands would be similar.

 

There's some funny stuff in the X is negative range.

Comments welcome.

Steve.

Edited April 21, 2022 by SteveBz

[SteveBz]

On 20/04/2022 at 08:32, Victor Boesen said:

I need your data 

Victor, just seen this.  You can have as much as you like of mine, but it's the old format. It that OK?

Kind regards

Steve.

[Victor Boesen]

30 minutes ago, SteveBz said:

Victor, just seen this.  You can have as much as you like of mine, but it's the old format. It that OK?

Kind regards

Steve.

Should be perfect!! Perhaps it would be the easiest to upload it to wetransfer?

[SteveBz]

8 minutes ago, Victor Boesen said:

Should be perfect!! Perhaps it would be the easiest to upload it to wetransfer?

Can you PM me your email adress then?

Steve.

[Victor Boesen]

24 minutes ago, SteveBz said:

Can you PM me your email adress then?

Steve.

Shouldn't be necessary, just upload the files to weshare and generate a sharable link 

Then I can download the data from that link.

[SteveBz]

Here it is zipped up.  Not sure about wetransfer, I don't have a subcription.

Sorry

 

Steve.

ra16-61v1.tar.gz

[Victor Boesen]

4 minutes ago, SteveBz said:

Here it is zipped up.  Not sure about wetransfer, I don't have a subcription.

Sorry

 

Steve.

ra16-61v1.tar.gz 28.11 MB · 0 downloads

Excellent, thanks a lot!! This will keep me busy for the next year or so 😅 Really helpful!

[SteveBz]

2 minutes ago, Victor Boesen said:

Excellent, thanks a lot!! This will keep me busy for the next year or so 😅 Really helpful!

There's some .pngs there that you don't really need.  You can delete them.  I've altered the 'parameters' key to hold:

 

{
    "Parameters": {
        "sample_rate": 2400000,
        "ppm": 0,
        "resolution": 11,
        "num_FFT": 10000,
        "num_med": 75,
        "ra": 57.6,
        "dec": 61.0,
        "l": "142d58m28.6283s",
        "b": "5d17m52.6504s",
        "const": "Camelopardalis",
        "DT": "20220409150351"
    },

In particular, the DT (Datetime) allows you to use @ZiHao's trick to convert to the LSR of the sample date and time.  My lat and long are about 51 N, 0 E, 88 m.

[SteveBz]

Hi,

@Victor Boesen and @ZiHao, After we collect our data we store it and analyse it.  At the moment I think we're all using different and incompatible data storage mechanisms and formats, so that when, for instance, Victor wanted to borrow ZiHao's data it was in a different format.  My data is in Victor's old format, but modified, so it was also awkward.  Can I suggest we all agree to a common standard and format?  It wouldn't be hard for any of us to change I think, but it would make our data more shareable. At the moment ZiHao appears to be using Python's NumPy format Victor is using a newer json format and I am using an older, but modified, json format.   

There are many formats out there.  I don't know NumPy format, but it might be useful.  I'm comfortable with json, but there are even other Pandas-style formats like pickle, which might be useful.

Can we have a discussion about what container (ie Numpy, json, csv, pickle etc to use) and then, separately, what format to use within that container?

While I'm happy with json, I'd also be happy to consider an alternative.

What do you think?

Kind regards,

Steve.

[SteveBz]

Hi Gents,

On a second point, my LSR correction does not seem to be working. Still.  At certain declinations, I'm about 10 km/s out, which means that slow-moving clouds can go from being moving towards to moving away (eg for 270 > l > 90 this does not make sense). 

Can I ask:

a) what declinations or values of galactic longitude are you able to access?  And what are your errors against the LAB survey?

b) Could you post your code here again please, so I can check mine against yours?

And here is mine:

def correct4LSR(self):
        # Define RV of 29.8 km/s
        rv = 29.8 * u.km / u.s 
        # Define V0 of 236 km/s
        V0 = 236 * u.km / u.s 
        # Define V0 of 220 km/s
        R0 = 8.15 * u.kpc
        # Define c0 of 299792.458 # km/s
        c0 = 299792.458 * u.km / u.s 
        for coords, record in self.raws.items(): 
            # Create Array of RAs
            ra=record['ra']
            #self.ras.append(ra)  # RA
            
            # Create Array of DECs
            dec=record['dec']
            #self.decs.append(dec) # DEC
            
            # Create Array of Peak signals
            sig_dB=max(record['sig_dBs'])
            #self.sig_dBs.append(sig_dB.max()) # peak dB value
            
            # Create Array of Speeds Lists
            speeds=record['rvs']
            #speedspeed0=speeds
            # Get time
            DT=record['DT']
            
            #Get time of exposure
            date_time_str=DT[:4]+'/'+DT[4:6]+'/'+DT[6:8]+' '+DT[8:10]+':'+DT[10:12]+':'+DT[12:14]
            dt=datetime.strptime(date_time_str,'%Y/%m/%d %H:%M:%S')
            t = Time(dt)

            loc = EarthLocation(lon=0*u.deg,lat=51*u.deg,height=88*u.m)

            #Correct for motion round the Sun
            sc = SkyCoord(record['ra']*u.deg,record['dec']*u.deg)
            vcorr = sc.radial_velocity_correction(kind='barycentric', obstime=t, location=loc)  
            vcorr= (float(str(vcorr)[:10])/1000)
            #print(vcorr)
            corrected_rv = speeds+vcorr

            #Correct for local group of stars
            observation = ICRS(ra=record['ra']*u.deg, dec=record['dec']*u.deg, \
                    pm_ra_cosdec=0*u.mas/u.yr, pm_dec=0*u.mas/u.yr, \
                    radial_velocity=0 *u.km/u.s, distance = 1*u.pc)

            new_rv = observation.transform_to(LSR()).radial_velocity 
            new_rv= (float(str(new_rv)[:10]))
            #print(new_rv)
            # Adjust Doppler velocity
            corrected_vel=float(str(new_rv)[:5])+corrected_rv
            self.raws[coords]['rvs']=corrected_vel
        return

Thanks.

Steve.

[SteveBz]

So here it is.  Still got a few funnies.  Most of the numbers were back to front and I need to trace it through and there's a strange little spur South of Sol, but otherwise nearly there.  It covers 2 quadrants (IV & III, and actually I can see quadrant II as well). So there's still a bit more work to be done, but I have been surprised at how well it's come out:

 

You can easily see the Orion spur leading into the Perseus arm and the outer arm.  There is also that blue arm, that appears on the map without a name.

Kind regards

Steve.

[Victor Boesen]

9 hours ago, SteveBz said:

Hi,

@Victor Boesen and @ZiHao, After we collect our data we store it and analyse it.  At the moment I think we're all using different and incompatible data storage mechanisms and formats, so that when, for instance, Victor wanted to borrow ZiHao's data it was in a different format.  My data is in Victor's old format, but modified, so it was also awkward.  Can I suggest we all agree to a common standard and format?  It wouldn't be hard for any of us to change I think, but it would make our data more shareable. At the moment ZiHao appears to be using Python's NumPy format Victor is using a newer json format and I am using an older, but modified, json format.   

There are many formats out there.  I don't know NumPy format, but it might be useful.  I'm comfortable with json, but there are even other Pandas-style formats like pickle, which might be useful.

Can we have a discussion about what container (ie Numpy, json, csv, pickle etc to use) and then, separately, what format to use within that container?

While I'm happy with json, I'd also be happy to consider an alternative.

What do you think?

Kind regards,

Steve.

Great idea! I am still pondering about improving the data included in the debug file. This involves including galactic coordinates, horizontal coordinates, time of observation and what else that may be useful later on.

15 minutes ago, SteveBz said:

So here it is.  Still got a few funnies.  Most of the numbers were back to front and I need to trace it through and there's a strange little spur South of Sol, but otherwise nearly there.  It covers 2 quadrants (IV & III, and actually I can see quadrant II as well). So there's still a bit more work to be done, but I have been surprised at how well it's come out:

That looks very good!

Victor

[ZiHao]

6 hours ago, SteveBz said:

Hi Gents,

On a second point, my LSR correction does not seem to be working. Still.  At certain declinations, I'm about 10 km/s out, which means that slow-moving clouds can go from being moving towards to moving away (eg for 270 > l > 90 this does not make sense). 

Can I ask:

a) what declinations or values of galactic longitude are you able to access?  And what are your errors against the LAB survey?

b) Could you post your code here again please, so I can check mine against yours?

And here is mine:

def correct4LSR(self):
        # Define RV of 29.8 km/s
        rv = 29.8 * u.km / u.s 
        # Define V0 of 236 km/s
        V0 = 236 * u.km / u.s 
        # Define V0 of 220 km/s
        R0 = 8.15 * u.kpc
        # Define c0 of 299792.458 # km/s
        c0 = 299792.458 * u.km / u.s 
        for coords, record in self.raws.items(): 
            # Create Array of RAs
            ra=record['ra']
            #self.ras.append(ra)  # RA
            
            # Create Array of DECs
            dec=record['dec']
            #self.decs.append(dec) # DEC
            
            # Create Array of Peak signals
            sig_dB=max(record['sig_dBs'])
            #self.sig_dBs.append(sig_dB.max()) # peak dB value
            
            # Create Array of Speeds Lists
            speeds=record['rvs']
            #speedspeed0=speeds
            # Get time
            DT=record['DT']
            
            #Get time of exposure
            date_time_str=DT[:4]+'/'+DT[4:6]+'/'+DT[6:8]+' '+DT[8:10]+':'+DT[10:12]+':'+DT[12:14]
            dt=datetime.strptime(date_time_str,'%Y/%m/%d %H:%M:%S')
            t = Time(dt)

            loc = EarthLocation(lon=0*u.deg,lat=51*u.deg,height=88*u.m)

            #Correct for motion round the Sun
            sc = SkyCoord(record['ra']*u.deg,record['dec']*u.deg)
            vcorr = sc.radial_velocity_correction(kind='barycentric', obstime=t, location=loc)  
            vcorr= (float(str(vcorr)[:10])/1000)
            #print(vcorr)
            corrected_rv = speeds+vcorr

            #Correct for local group of stars
            observation = ICRS(ra=record['ra']*u.deg, dec=record['dec']*u.deg, \
                    pm_ra_cosdec=0*u.mas/u.yr, pm_dec=0*u.mas/u.yr, \
                    radial_velocity=0 *u.km/u.s, distance = 1*u.pc)

            new_rv = observation.transform_to(LSR()).radial_velocity 
            new_rv= (float(str(new_rv)[:10]))
            #print(new_rv)
            # Adjust Doppler velocity
            corrected_vel=float(str(new_rv)[:5])+corrected_rv
            self.raws[coords]['rvs']=corrected_vel
        return

Thanks.

Steve.

This line

observation = ICRS(ra=record['ra']*u.deg, dec=record['dec']*u.deg, \
                    pm_ra_cosdec=0*u.mas/u.yr, pm_dec=0*u.mas/u.yr, \
                    radial_velocity=0 *u.km/u.s, distance = 1*u.pc)

I think you should enter radial_velocity=vcorr. Try not changing the 'speeds' array first, when you get the final new_rv, only then you add the value to speeds. So this line has to be removed

 corrected_rv = speeds+vcorr

 

[SteveBz]

1 hour ago, ZiHao said:

This line

observation = ICRS(ra=record['ra']*u.deg, dec=record['dec']*u.deg, \
                    pm_ra_cosdec=0*u.mas/u.yr, pm_dec=0*u.mas/u.yr, \
                    radial_velocity=0 *u.km/u.s, distance = 1*u.pc)

I think you should enter radial_velocity=vcorr. Try not changing the 'speeds' array first, when you get the final new_rv, only then you add the value to speeds. So this line has to be removed

 corrected_rv = speeds+vcorr

 

Thanks for this.  I've now updated it and it seems to work, but I've no idea why.  Here is the update:

  
    def correct4LSR(self):
        # Define RV of 29.8 km/s
        rv = 29.8 * u.km / u.s 
        # Define V0 of 236 km/s
        V0 = 236 * u.km / u.s 
        # Define V0 of 220 km/s
        R0 = 8.15 * u.kpc
        # Define c0 of 299792.458 # km/s
        c0 = 299792.458 * u.km / u.s 
        for coords, record in self.raws.items(): 
            # Create Array of Speeds Lists
            rvs=record['rvs']
            rvs0=rvs
                        
            #Get date, time and location of exposure
            DT=record['DT']
            date_time_str=DT[:4]+'/'+DT[4:6]+'/'+DT[6:8]+' '+DT[8:10]+':'+DT[10:12]+':'+DT[12:14]
            dt=datetime.strptime(date_time_str,'%Y/%m/%d %H:%M:%S')
            t = Time(dt)
            loc = EarthLocation(lon=0*u.deg,lat=51*u.deg,height=88*u.m)

            # Create Array of RAs
            ra=record['ra']
            # Create Array of DECs
            dec=record['dec']
                        
            #Correct for motion round the Sun
            sc = SkyCoord(ra*u.deg,dec*u.deg)
            rv_corr1 = sc.radial_velocity_correction(kind='barycentric', obstime=t, location=loc)  
            rv_corr1= (float(str(rv_corr1)[:10])/1000)
            #print(vcorr)
            #Correct for local group of stars
            observation = ICRS(ra=ra*u.deg, dec=dec*u.deg, \
                    pm_ra_cosdec=0*u.mas/u.yr, pm_dec=0*u.mas/u.yr, \
                    radial_velocity=0 *u.km/u.s, distance = 1*u.pc)

            rv_corr2 = observation.transform_to(LSR()).radial_velocity 
            rv_corr2= (float(str(rv_corr2)[:10]))
            # Adjust Doppler velocity
            rvs = rvs+rv_corr1-rv_corr2
            self.raws[coords]['rvs']=rvs
            
        return

The key line is

 rvs = rvs+rv_corr1-rv_corr2

Strange, no?  Also, once I plotted the final spiral plot, I had to do:

X=-X

because the Galaxy was rotating in the wrong direction and also

rvs=-rvs.

to change the blue/red shift colours. So I really need to chase these through and understand them properly and make sure they are legitimate.

Kind regards

Steve.

[SteveBz]

1 hour ago, Victor Boesen said:

Great idea! I am still pondering about improving the data included in the debug file.

I'm happy with the idea of .json, but I want to know if @ZiHao likes it too, or if he'd rather change it.

I also think it's not a debug file, because it's everything.  It's really the output data of the whole thing, so a new name would be a start

I agree with your idea if putting all the run parameters in it.

But then we have a number of questions around radial velocity or frequency and should the velocity be corrected for LSR or not.  It seems to me if you put RV you have to correct it for LSR.  If you just have frequency, then not.

Maybe it could be a runtime option (raw frequency or corrected RV).  Frequency is easier to filter for electronic noise because the spikes are always in the same place. In fact I make some artificial 'flats' from the scans in the areas of empty space and then apply them to the other areas.  It would probably work with your big spike too.

In the longer term, I'll probably want to use a big-data approach with Pandas Dataframes for the work I do.  It would be much faster, but a learning curve for any new users.

I'd like to hear other views.

Kind regards

Steve.

[Victor Boesen]

1 hour ago, SteveBz said:

I also think it's not a debug file, because it's everything.  It's really the output data of the whole thing, so a new name would be a start

I agree with your idea if putting all the run parameters in it.

But then we have a number of questions around radial velocity or frequency and should the velocity be corrected for LSR or not.  It seems to me if you put RV you have to correct it for LSR.  If you just have frequency, then not.

I agree with you on this. I will consider renaming it  
In the case of radial velocity and etc. there is nothing keeping us from just including everything 

I just came inside after doing some more software testing. I found one bug, for which I'm going to push a fix later tonight. I did some observations along the Milky Way and here's the result:

All were received with the same parameters. Besides the default parameters the following parameters were used:

50000 FFTs
10 Median

Comparing some of the observation with the lab survey it seems some are slightly off for some reason. I'm assuming that this because I haven't calculated the LSR for each observation and only corrected for the Earth's velocity. Will do some more coding to try and calculate the LSR!!

Victor

[SteveBz]

12 minutes ago, Victor Boesen said:

 

Wow, really cool. How did you track the galactic plane?

[SteveBz]

14 minutes ago, Victor Boesen said:

I'm assuming that this because I haven't calculated the LSR for each observation and only corrected for the Earth's velocity. Will do some more coding to try and calculate the LSR!!

The LSR code that I posted above seems to work. The corrections for the area I was looking at had LSR = 5 and planetary movement - 5 - or the other way round, but they were both about the same magnetude, but of different sign. To get it to match I had to reverse one of the signs, then it worked beautifully.

Steve.

[SteveBz]

19 minutes ago, Victor Boesen said:

All were received with the same parameters. Besides the default parameters the following parameters were used:

50000 FFTs
10 Median

 

Some of your pngs have better noise than others? Why is that?

[Victor Boesen]

8 minutes ago, SteveBz said:

Wow, really cool. How did you track the galactic plane?

I have programmed a simple compass and bubble level app for my phone that I use to point the antenna. Then I simply use Stellarium to get the desired Az/Alt position of my desired location and point the antenna there. I also bring a good'ol compass to double check the azimuth.

2 minutes ago, SteveBz said:

Some of your pngs have better noise than others? Why is that?

This also confused me. However, after collecting all the data I tried scanning the environment with SDR++ where it showed noise spurs some places on the sky. These directions roughly matched the direction of the noisier observations 

Thanks for the LSR info, will get started on coding it as soon as I've got some spare time from university!

Victor

[SteveBz]

I just downloaded a LAB file. The format is very old fashioned and I don't recommend it, however it's interesting to note the info they included in the file:

v_lsr [km/s], T_B [K], freq. [Mhz], wavel. [cm]
    -399.83      0.13   1422.300   21.07800
    -398.80      0.01   1422.295   21.07808

Steve

Edited April 22, 2022 by SteveBz

[Victor Boesen]

I think I may have found another, perhaps simpler, way of correcting the radial velocity with respect to the Local Standard of Rest. I followed this page, which nicely explains how SpectralCoord works and etc.

https://docs.astropy.org/en/stable/coordinates/spectralcoord.html

I've just run the following code with my observations from ealier today, and all of them seem to line up better with the H-line survey simulations.

from astropy import units as u
from astropy.time import Time
from astropy.coordinates import SkyCoord, SpectralCoord, EarthLocation, AltAz, ICRS, LSR, Galactic

def radialVelocityCorrection(self, az, alt, radial_vel):
    horizontal_coord = AltAz(alt = alt*u.degree, az = az*u.degree, pressure = 0*u.bar, obstime = self.TIME,location=self.QTH)
    eq_coord = SkyCoord(horizontal_coord.transform_to(ICRS()), radial_velocity = radial_vel*u.km/u.s,frame = 'icrs',distance = 10*u.kpc)

    # Define the observed radial velocity
    observer_loc = self.QTH.get_itrs(self.TIME)
    spectral_coord = SpectralCoord(ANALYSIS.freqFromRadialVel(radial_vel)*u.Hz,observer=observer_loc,target=eq_coord)

    freq_wrt_lsrk = spectral_coord.with_observer_stationary_relative_to("lsr")

Where the QTH is an EarthLocation describing the observer's location and TIME is the current time

self.QTH = EarthLocation(lat = lat*u.degree, lon=lon*u.degree,height=0*u.m)
self.TIME = Time.now()

Still need to test it further, but as of now it seems to lign up!

Victor

[ZiHao]

6 hours ago, SteveBz said:

Thanks for this.  I've now updated it and it seems to work, but I've no idea why.  Here is the update:

Hmmm. What did you get when you enter radial_velocity=rv_corr1 instead for this?

observation = ICRS(ra=ra*u.deg, dec=dec*u.deg, \
                    pm_ra_cosdec=0*u.mas/u.yr, pm_dec=0*u.mas/u.yr, \
                    radial_velocity=0 *u.km/u.s, distance = 1*u.pc)

And rvs=rvs+rv_corr2. As we need to correct our barycentric radial velocity to the LSR, according to the post on stackexchange. I also need to test this procedure on other parts of the milky way, perhaps after my exams in June.

 

6 hours ago, SteveBz said:

I'm happy with the idea of .json, but I want to know if @ZiHao likes it too, or if he'd rather change it.

I will try to convert the np arrays to json type if there is sharing of data required, if that's you meant by changing😅

 

6 hours ago, SteveBz said:

But then we have a number of questions around radial velocity or frequency and should the velocity be corrected for LSR or not.  It seems to me if you put RV you have to correct it for LSR.  If you just have frequency, then not.

We always correct to LSR so that we can compare our data easily with everyone observing at different locations. Since we are working with velocities most of the time, I think that is probably why we usually have velocity as the x-axis.

Edited April 23, 2022 by ZiHao

[SteveBz]

10 hours ago, ZiHao said:

will try to convert the np arrays to json type if there is sharing of data required, if that's you meant by changing😅

I was suggesting that we all try to use the same file type and the same format to share data.

But actually,  I'm now thinking that I'll just write a conversion routine to switch between formats. It's unlikely that the LAB site at Bonn will want to change 😂.

Kind regards,

Steve.