21cm band. Refurbishing the dish with a new Cantenna.

[ZiHao]

48 minutes ago, SteveBz said:

Actually, it seems to be nearly linear in Dec.  

Interesting pattern, might be the pointing error, or the code itself, not too sure... I have only one set of data for now so can't really test it reliably. We need other sets of data🧐

Edited April 18, 2022 by ZiHao

[SteveBz]

31 minutes ago, ZiHao said:

Interesting pattern, might be the pointing error, or the code itself, not too sure... I have only one set of data for now so can't really test it reliably. We need other sets of data🧐

Yes, I'm doing the Dec measurement with a digital level and I think it's always within 1 degree.  But the RA seems to vary (according to @Victor Boesen's graph) by up to 5 degrees.  Because of all the iron in the mount it's very hard to use a compass to align it.  Next clear night, I'll align it with Polaris. 

[Victor Boesen]

1 hour ago, ZiHao said:

Nope, I didn't calibrate mine, like what Victor had mentioned, 1-2ppm shouldn't cause that much of a difference.

Btw Victor, for the pyrtlsdr wrapper, in sdr.freq_correction, it seems that a value has to be entered, how did you deal with this?

That is not my impression. However, I did find that it can't be set to 0 for some weird reason. So this is how I initialize the SDR in my code:

# Return a physical serial SDR client
def rtlClient(self):
  client_sdr = RtlSdr()
  client_sdr.sample_rate = self.SAMPLE_RATE
  client_sdr.center_freq = self.CENTER_FREQ
  client_sdr.gain = 'auto'

  # For some reason the SDR doesn't want to set the offset PPM to 0 so we avoid that
  if self.PPM_OFFSET != 0:
    self.sdr.freq_correction = self.PPM_OFFSET

  return client_sdr

 

Edited April 18, 2022 by Victor Boesen

[Victor Boesen]

@ZiHao Am I correct in assuming that the radial_velocity_correction function returns the Earth's velocity in the direction of the observed area?

In the sketch above the radial_velocity_correction would then return "-velocity correction"??

Victor

[ZiHao]

11 minutes ago, Victor Boesen said:

@ZiHao Am I correct in assuming that the radial_velocity_correction function returns the Earth's velocity in the direction of the observed area?

In the sketch above the radial_velocity_correction would then return "-velocity correction"??

Victor

I think the general idea is to first correct earth's velocity to the solar system barycenter, and then we convert this to the local standard of rest, very vague explanation as I am struggling to visualize this also.
After computing the barycentric velocity correction, simply add this to your radial velocity, barycentric_rv = rv + v_correction. The text listed here https://astronomy.stackexchange.com/questions/41196/looking-for-a-routine-to-correct-for-local-standard-of-rest-lsr seems to be the only explanations I can find.

[Victor Boesen]

6 minutes ago, ZiHao said:

I think the general idea is to first correct earth's velocity to the solar system barycenter, and then we convert this to the local standard of rest, very vague explanation as I am struggling to visualize this also.
After computing the barycentric velocity correction, simply add this to your radial velocity, barycentric_rv = rv + v_correction. The text listed here https://astronomy.stackexchange.com/questions/41196/looking-for-a-routine-to-correct-for-local-standard-of-rest-lsr seems to be the only explanations I can find.

Alright, thank you. I think I understand it now 😅 So, the barycentric_rv = rv + v_correction just corrects for the Earth's orbital velocity around the sun, whereas the LSR considers the Sun's motion relative to the observed area. In my case, developing my software, I am only interested in the barycentric radial velocity since the LSR velocity must depend on the distance to the area. For the majority of users (including myself) this is unknown for the most part. Thanks for sharing your resources on here! They are very helpful Do you have a GitHub account by any chance where I can have a look at some of your stuff?

[ZiHao]

7 minutes ago, Victor Boesen said:

Alright, thank you. I think I understand it now 😅 So, the barycentric_rv = rv + v_correction just corrects for the Earth's orbital velocity around the sun, whereas the LSR considers the Sun's motion relative to the observed area. In my case, developing my software, I am only interested in the barycentric radial velocity since the LSR velocity must depend on the distance to the area. For the majority of users (including myself) this is unknown for the most part. Thanks for sharing your resources on here! They are very helpful Do you have a GitHub account by any chance where I can have a look at some of your stuff?

Actually, the transformation to the LSR velocity is somehow independent of the distance to the area, as mentioned by the guy on stackexchange. I tried playing with the distance in the code just now and the final result did not change! 
I always wanted to gather all these information such as code on a website like blogspot, github but didn't do it because I have only two 'main pieces' of code: solar drift scan and hydrogen line, perhaps I should try to add all these in no matter the quantity. I will share it here once they are ready👍

[Victor Boesen]

4 minutes ago, ZiHao said:

Actually, the transformation to the LSR velocity is somehow independent of the distance to the area, as mentioned by the guy on stackexchange. I tried playing with the distance in the code just now and the final result did not change! 

Well, guess I need to do some more reading

5 minutes ago, ZiHao said:

I always wanted to gather all these information such as code on a website like blogspot, github but didn't do it because I have only two 'main pieces' of code: solar drift scan and hydrogen line, perhaps I should try to add all these in no matter the quantity. I will share it here once they are ready

Definitely do so!! I've become quite a big fan of using GitHub for my coding projects. Great for when you need to look at some old code you suddenly find out you needed but deleted long ago!

[SteveBz]

So, just to be clear, I'm doing RV = LSR_Adjustment - Doppler_Speed.

Kind regards.

[SteveBz]

I also think that my mount is off by about 18 degrees, so at low Az there is a greater RA Error than at High Az.  Rather than waiting for the next clear night, I'm trying the correction RA_New = Delta_RA/cos(lat). For me that translates to Delta_RA ~ 25 degrees.  I get a move in the right direction, but not completely.  So I think I may leave the mount as it is and just update the Az input to H-line.py and also, retrospectively amend the RA data for my previous scans.

[Victor Boesen]

I managed to do some testing of my new rewritten software. After seeing @ZiHao using Astropy I decided to try and switch to astropy from pyephem, which made things easier when correcting for radial velocity. Here are three observations from today and a comparison to the same area from the H-line survey.

And from H-line survey. Note that it's just mirrored since they plot it as a function of increasing radial velocity, whereas I plot mine as a function of increasing frequency.

Next target:

And final target which greatly shows the correlation between my spectrum and theirs. Look at the peak which is basically centered at 0 in both cases.

Annoyingly, the noise spike is just inside the filter I apply when searching for the hydrogen line peak, so next thing on my todo-list will be removing these large spikes! This will, however, be a lot easier to work on now that I have some debug files I can test on

I was also able to test out the new UI, which I really liked, and everything worked as it should. Therefor, I have merged the rewrite branch with the main branch on GitHub
https://github.com/byggemandboesen/H-line-software

Victor

Edited April 18, 2022 by Victor Boesen

[SteveBz]

The stars are out tonight.  It's quite hard to align your dish!  At my estimate the dish is about 1.5 degrees (my thumb's width)  to the East of Polaris and Polaris itself is about .7 degrees West of NCP. Therefore the dish is within 1 degree of NCP.  It's good enough. So I don't really understand why I have this structural error.

On top of all of this, the code is becoming a nightmare and I really need to refactor it 😅

Steve.

[SteveBz]

20 minutes ago, Victor Boesen said:

I managed to do some testing of my new rewritten software. After seeing @ZiHao using Astropy I decided to try and switch to astropy from pyephem, which made things easier when correcting for radial velocity. Here are three observations from today and a comparison to the same area from the H-line survey.

And from H-line survey. Note that it's just mirrored since they plot it as a function of increasing radial velocity, whereas I plot mine as a function of increasing frequency.

Next target:

And final target which greatly shows the correlation between my spectrum and theirs. Look at the peak which is basically centered at 0 in both cases.

Annoyingly, the noise spike is just inside the filter I apply when searching for the hydrogen line peak, so next thing on my todo-list will be removing these large spikes! This will, however, be a lot easier to work on now that I have some debug files I can test on

I was also able to test out the new UI, which I really liked, and everything worked as it should. Therefor, I have merged the rewrite branch with the main branch on GitHub
https://github.com/byggemandboesen/H-line-software

Victor

Really nice, Victor.  Very good correlation.  Once I've stabilised my code again, I'll refresh my copies of H-line and give you some feedback. I'm sorry I've not done it so far.

Where are you based in terms of other houses and stuff?  Are you in the country or town?  How many FFTs are you doing and what about the Median smoothing parameter?

 

[Victor Boesen]

21 minutes ago, SteveBz said:

Really nice, Victor.  Very good correlation.  Once I've stabilised my code again, I'll refresh my copies of H-line and give you some feedback. I'm sorry I've not done it so far.

Where are you based in terms of other houses and stuff?  Are you in the country or town?  How many FFTs are you doing and what about the Median smoothing parameter?

 

Thank you Steve!! Still got some work to do with the noise spikes removal. No need to apologize for not testing it yet!

I do my observing besides multiple apartments in the city so far from ideal. Parameters used are the following:

"DSP Parameters": {
        "number_of_fft": 50000,
        "resolution": 11,
        "median": 10
}

Victor

[ZiHao]

On 18/04/2022 at 03:35, SteveBz said:

Perfect. That's what I got too.

What I don't have is the derivations for the other quadrants or outside the solar circle. Do you think they are different?

Alright, I have tried playing around with the quadrants and came up with these formulae:

Quadrant I: V_r=V_o*sinl((R_o/R_s )-1)  (basically just the same of what we have)
Quadrant II: same as Q1
Quadrant III: V_r=V_o*sinl((-R_o/R_s)+1)
Quadrant IV: same as Q3

We get these formulae if we always take the velocity component of source minus the velocity component of sun. The formulae for Q3 and Q4 differs by a negative sign, which means that the convention of redshift being positive and blueshift being negative is flipped now, you can still get the correct R_s but you would have to substitute a 'negative redshift into the equation' which is quite confusing. So just use the original formula and apply it to all quadrants, as it follows our convention of the signs.

If we were to take into account the arrows or the direction of the resolved velocity, and always take the velocity component of the object being 'chased' minus the velocity component of the object 'following', be it the sun or the source, then we will get the same formula for all quadrants.
 

Edited April 19, 2022 by ZiHao

[SteveBz]

9 hours ago, Victor Boesen said:

Thank you Steve!! Still got some work to do with the noise spikes removal. No need to apologize for not testing it yet!

I do my observing besides multiple apartments in the city so far from ideal. Parameters used are the following:

"DSP Parameters": {
        "number_of_fft": 50000,
        "resolution": 11,
        "median": 10
}

Victor

Have you tried a higher median,  like 50? It won't get ride of the spike, but I think it may smooth the noise? If your curve is smoother it's easier to do Gaussian fitting.

If you can reliably do max(freqs) you can get the mean of the highest peak and then subtract a fake gaussian (norm.pdr) from the curve. I'll post some code later.

[Victor Boesen]

13 minutes ago, SteveBz said:

Have you tried a higher median,  like 50? It won't get ride of the spike, but I think it may smooth the noise? If your curve is smoother it's easier to do Gaussian fitting.

If you can reliably do max(freqs) you can get the mean of the highest peak and then subtract a fake gaussian (norm.pdr) from the curve. I'll post some code later.

Thanks Steve, will give it a go! I tried median=20, but it looked worse. Perhaps I just need to go even further to 50-70 before it starts to look nice again. However, the reason I don't like overdoing the median filter too much is because it modifies the data. I could just calculate the max SNR, radial velocity and etc before I apply the median filter though... Some stuff to think about

[SteveBz]

On 17/04/2022 at 16:39, ZiHao said:

rv=c*((rest_freq/rest_freq)-1)*u.km/u.s

t = Time(, format='mjd')

loc = EarthLocation(lon=,lat=,height=)
ra=*u.deg
dec=*u.deg
sc = SkyCoord(ra,dec)

vcorr = sc.radial_velocity_correction(kind='barycentric', obstime=t, location=loc)  
rv = rv + vcorr

my_observation = ICRS(ra=*u.deg, dec=*u.deg, \
        pm_ra_cosdec=0*u.mas/u.yr, pm_dec=0*u.mas/u.yr, \
        radial_velocity=rv, distance = 1*u.pc)

new_rv = my_observation.transform_to(LSR()).radial_velocity

corrected_vel=corrected_vel+new_rv

I'm still struggling with this.  

Line 1 has a typo 'rest_freq/rest_freq' .  I think it should be freq/rest_freq, no?

1) rv = raw radial velocity from observation, uncorrected.

2) vcorr = the speed of the Earth round the Sun, more or less.

3) rv = rv + rcorr ... Adjusted for speed of Earth round Sun

4) my_observation = plug rv into Astropy.

5) new_rv = observation adjusted for movement of local star group

6) corrected_vel=corrected_vel+new_rv  - what's happening here?  Isn't new_rv in 5) the answer?

Tx 

Steve.

 

[ZiHao]

7 hours ago, SteveBz said:

I'm still struggling with this.  

Line 1 has a typo 'rest_freq/rest_freq' .  I think it should be freq/rest_freq, no?

1) rv = raw radial velocity from observation, uncorrected.

2) vcorr = the speed of the Earth round the Sun, more or less.

3) rv = rv + rcorr ... Adjusted for speed of Earth round Sun

4) my_observation = plug rv into Astropy.

5) new_rv = observation adjusted for movement of local star group

6) corrected_vel=corrected_vel+new_rv  - what's happening here?  Isn't new_rv in 5) the answer?

Tx 

Steve.

 

I did rest_freq/rest_freq so that the raw uncorrected velocity is zero, and the rest of the code is just to focus on correcting the zero raw uncorrected velocity, and the final new_rv will just be a single value of the zero corrected velocity. And finally we just have to add this new_rv value to the corrected_vel array which originally was an array with raw uncorrected velocities, this shifts the spectrum. You can also correct for individual raw uncorrected velocity but I find that it takes long time to compute, therefore I use the zero point so that I can directly add to the raw array to shift all the values easily. Sry for the confusion!

[Victor Boesen]

I need your data 

I just wrote a simple function to identify noise spikes in the spectrum, and have tested it with some of my own data from yesterday. It works great, however, I'd like to test it on some other data than my own to confirm it works before I do any more tweaking  

If you could possibly send some debug files (preferably with the new version on GitHub) I'll do some testing on your data. Here are the first results with my own data:

In none of the cases were any bins from the hydrogen line itself replaced/modified, only the noise spike on the left!!

EDIT: Just updated the slant correction too. Now it's corrected for slant, after the noise spikes are removed. This yields the following result:

Victor

Edited April 20, 2022 by Victor Boesen

[ZiHao]

1 hour ago, Victor Boesen said:

I need your data 

I just wrote a simple function to identify noise spikes in the spectrum, and have tested it with some of my own data from yesterday. It works great, however, I'd like to test it on some other data than my own to confirm it works before I do any more tweaking  

If you could possibly send some debug files (preferably with the new version on GitHub) I'll do some testing on your data. Here are the first results with my own data:

In none of the cases were any bins from the hydrogen line itself replaced/modified, only the noise spike on the left!!

EDIT: Just updated the slant correction too. Now it's corrected for slant, after the noise spikes are removed. This yields the following result:

Victor

Very nice! I might send some old data taken few months ago later. Curious to know, did you locate approximately the frequency bins with the noise spikes and replace the spikes with the mean of nearby values?

[Victor Boesen]

29 minutes ago, ZiHao said:

Very nice! I might send some old data taken few months ago later. Curious to know, did you locate approximately the frequency bins with the noise spikes and replace the spikes with the mean of nearby values?

Great, thanks!!

No, it's a little more complicated than that. I could "just" set a threshold which determines whether it's a noise spike or hydrogen line spike, but that really wouldn't work at large signal to noise ratios since the h-line could easily be mistaken by a noise spike. Instead, I look at the SNR difference between each bin. What's unique about most noise spikes is that they're very sharp.

The plot above shows the difference in SNR between each bin (essentially the slope). This clearly shows how much different it is compared to the hydrogen line. Then I set a threshold for the maximum allowed rate of change. I've currently set that to one standard deviation of the original data/spectrum. Then I just replace the bins in this area my some random bins located around the median of the data. This last part still needs to be optimized.

EDIT: Subtracting the bins from themselves seemed to work even better

Edited April 20, 2022 by Victor Boesen

[ZiHao]

5 hours ago, Victor Boesen said:

I need your data 

The data is here! You might want to limit the y axis from -0.0005 to 0.0040, and the data is...you will know once you open it🥴, this is taken with the clone rtlsdr, no filters applied.

source20.npy velocity.npy

[Victor Boesen]

27 minutes ago, ZiHao said:

The data is here! You might want to limit the y axis from -0.0005 to 0.0040, and the data is...you will know once you open it🥴, this is taken with the clone rtlsdr, no filters applied.

source20.npy 16.13 kB · 0 downloads velocity.npy 16.13 kB · 0 downloads

Thanks!! However, did you use my software to receive it? The DC spike is reaaaalllllyyyy bad 😅 Since it's so strong compared to the rest of the spectrum the std is not a good representation for the rest of the spectrum.

So it's not finding any of the other noise spikes.

Edited April 20, 2022 by Victor Boesen

[ZiHao]

26 minutes ago, Victor Boesen said:

Thanks!! However, did you use my software to receive it? The DC spike is reaaaalllllyyyy bad 😅 Since it's so strong compared to the rest of the spectrum the std is not a good representation for the rest of the spectrum.

So it's not finding any of the other noise spikes.

Data is received with my python code, the raw data is first filtered by a median filter of size 30, or even higher IIRC, but I didn't use those in my attached data. I gave up on this clone as even though the sharp spikes like the DC spikes is removed, I am left with some broadband peaks which is very annoying to work with. Now with the genuine rtlsdr, I am using a median filter of size 15, and it got rid of the DC spike, leaving a clean spectrum. 

Edited April 20, 2022 by ZiHao