Where is my processing going wrong ?

[SlimPaling]

I have noticed recently that after all of my processing and then zoom right into the image I am getting lines of obviously unwanted spurious small dots all running in the same direction all over my final image .... please see my attached image where I have zoomed right into my image of NGC 891.

After a bit of detective work I can see that these "seem to originate" from small fine white dots in my subs after I have calibrated them .... using MaximDL

Is there something wrong with my calibration files ?.... they are all reasonably up to date. I have been using  35 x Bias frames,  20 x Dark frames and 17 x Flat frames for each of the RGB filters.

I am using MaximDL 6 for all calibration, alignment and stacking .... and use PS for final processing.

Any suggestions would be muchly appreciated :-)

Mike

[souls33k3r]

It's what we call "walking noise". 

Try stacking the files using "Median" algorithm in MaximDL. This should get rid of such.

Hope this helps

[Uranium235]

Seems like hot pixels to me.

If your using Maxim, do the following first.

1) On each light frame, use the kernel filter and select "hot pixels" - set it to a threshold of 40% and hit OK. Its probably better to record is as an "action" first, then run it on all of the data (add files) so you dont have to repeat it a load of times - you can either overwrite original or get it to put the processed files in a new subfolder.

2) Now stack your frames using SD mask, setting = delta level, area 100%

"Walking noise" is a bit harder to get rid of, the best way I found of doing this with an 8300 chip is to calibrate the data with DSS - using flats, dark flats, and bias frames (normal darks can mess things up a bit sometimes). After which I throw it all into maxim and do the above two steps.

[wimvb]

Yes, definitely hot pixels. For a mono ccd, darks should remove hot pixels.

I have never used MaximDL, but both DSS and PI have a routine called cosmetic correction, which allows you to remove hot pixels that darks didn't catch, before the subs are stacked.

In any stacking program, you can use more aggressive pixel rejection to get rid of hot pixels (kappa sigma or windsorised sigma clipping). Median stacking has the same effect as more aggressive pixel rejection, but it has one big disadvantage. Average stacking will increase the bit depth of an image. Median stacking won't. If your stacked master is 16 bit, then you won't notice the difference. But if your stacked image is 32 bit or floating point accuracy, average stacking will get you a better image than median stacking.

[Davey-T]

Easy to auto generate a bad pixel map in Maxim using the master dark frame, you may have to fiddle with the threshold settings any missed ones can be added manually.

Dave

[Ouroboros]

30 minutes ago, wimvb said:

I have never used MaximDL, but both DSS and PI have a routine called cosmetic correction, which allows you to remove hot pixels that darks didn't catch, before the subs are stacked.

The cosmetic correction in DSS for the removal of hot pixels works well  IMO. I tried it recently on some data for which I didn't have darks and it was very effective. 

I'm wondering what the best parameters are. I used Filter Size 1 pixel and Detection Threshold 33%.  

I'm not sure what detection threshold actually means. The help file is not exactly helpful on this. It describes it as the threshold under which no correction is done. But threshold of what? 

Apparently the lower the threshold value the more corrections are done. So better start high and work down I guess. 

[wimvb]

44 minutes ago, Ouroboros said:

The cosmetic correction in DSS for the removal of hot pixels works well  IMO. I tried it recently on some data for which I didn't have darks and it was very effective. 

I'm wondering what the best parameters are. I used Filter Size 1 pixel and Detection Threshold 33%.  

I'm not sure what detection threshold actually means. The help file is not exactly helpful on this. It describes it as the threshold under which no correction is done. But threshold of what? 

Apparently the lower the threshold value the more corrections are done. So better start high and work down I guess. 

The algorithm that is used calculates the number of pixels at each intensity (using gray scale for simplicity). Most of the image will be background and give the common bell curve. The half width of the bell curve is the standard deviation of all pixelvalues from the mean. The threshold is a parameter that, multiplied by this standard deviation, determines which pixel values will be corrected. Pixels above mean plus threshold x std dev will be corrected. The lower the threshold, the more pixels close to the bell curve (background) will be corrected. The trick is to only correct hot pixels, and not destroy valid data. (The procedure in PixInsight will allow you to see a preview of all pixels that will be corrected. If this preview has any structure in it that resembles the target, you know that you're clipping valid data.)

Cosmetic correction identifies these pixels and then replaces them with the average intensity value of the surrounding pixels. At least that's how it works in PI. I think that DSS uses the same or a similar method. The exact math may be a bit more complicated.

[SlimPaling]

Wow ... you are all giving me a lot to think about here !!!

I did try out the first suggestion from Proto star by using Median stacking in Maxim ... it cleared all of the "walking noise" instantly !!! I was amazed by what a difference it has made to my final image.

I will be looking into all of the other suggestions as a matter of course .... I think there is a lot to sort out in my head!!!

Many thanks to everybody who has responded to my initial message :-)

Mike

[souls33k3r]

Proto Star? Lol I guess you meant me aka souls33k3r  Glad my suggestion helped

[SlimPaling]

6 hours ago, souls33k3r said:

Proto Star? Lol I guess you meant me aka souls33k3r  Glad my suggestion helped

Ooops! Sorry .... in my haste I misread the name :-(((

Mike

[souls33k3r]

hahahaha it's absolutely fine matey

[SlimPaling]

I have been reading some of the suggestions and comments that have been kindly sent in and trying to understand my options. I have also just done a quick bit of research on the 'net about some of this ... which has left me with some questions that I need to sort out in my head.

I do not currently have DSS ... I am trying to stick with MaximDL for my initial processing steps unless DSS does have some definite advantages that I do not currently know about.

 

Using the "Median stacking" option in Maxim has certainly removed virtually all traces of "Walking Noise" in my final images since trying it out last night ... but reading the MaximDL manual it suggests that this method can have some disadvantages ... I need to test out the other options and compare results here!

 

wimvb has said that " darks should remove hot pixels " .... it seems that some of my hot pixels are not being removed during calibration ... so my questions here are:

 

(1) Do I need to make a new set of Dark Frames???

(2) What is a good number to do? I have been using sets of 20 Darks

(3) Am I correct in thinking that a "Dark Frame" is the same as a "Dark Flat Frame" ? ... the latter type was mentioned in one of the replies above.

(4) Is it a good idea to produce Master Dark Frame before I do any calibrations?

(5) When I first started processing my subs with Maxim  a couple of months ago I did not notice any of this "Walking Noise" problem. Do these "Hot Pixels" crop up randomly and how permanent are they? Do they slowly accumulate and will I eventually have more and more of them as my sensor decays? :-(((

 

Mike

 

[Davey-T]

Hi Mike, the so called walking noise is caused by the image moving across the sensor between each sub, if your tracking is spot on theoretically the hot pixel will be in the same place on every sub and will be removed in processing, when the image is drifting then you get  the same hot pixel in a different place on each sub.

If you didn't have this problem before perhaps you were aimed at a different target / dec'  or something that didn't have as much drift.

I always load all my calibration frames into Maxim and make masters then use them for calibration. I don't think there's any point taking more than 20 calibration frames this can be verified by doing the maths if you're inclined

Once you have a master dark use it to make a bad pixel map and run all your subs through it before calibration.

All sensors have dead and hot pixels unless you pay a fortune for research grade ones so nothing to worry about, you may get a few new ones develop over time but a new bad pixel map will fix them or you can add them manually to your existing map.

DSS will not be any help to you as you've paid for Maxim it's worth making the effort to learn it's quirks, it's had the same layout for donkeys year so it's not the greatest GUI in the world but on the plus side it doesn't eat up your resources and you don't have to relearn it every time it updates, just wish it was a bit cheaper as there is a lot of new cheaper stuff out there now, I only use it because I bought it years ago when there wasn't as much choice,

Dave

 

[SlimPaling]

5 minutes ago, Davey-T said:

Hi Mike, the so called walking noise is caused by the image moving across the sensor between each sub, if your tracking is spot on theoretically the hot pixel will be in the same place on every sub and will be removed in processing, when the image is drifting then you get  the same hot pixel in a different place on each sub.

If you didn't have this problem before perhaps you were aimed at a different target / dec'  or something that didn't have as much drift.

I always load all my calibration frames into Maxim and make masters then use them for calibration. I don't think there's any point taking more than 20 calibration frames this can be verified by doing the maths if you're inclined

Once you have a master dark use it to make a bad pixel map and run all your subs through it before calibration.

All sensors have dead and hot pixels unless you pay a fortune for research grade ones so nothing to worry about, you may get a few new ones develop over time but a new bad pixel map will fix them or you can add them manually to your existing map.

DSS will not be any help to you as you've paid for Maxim it's worth making the effort to learn it's quirks, it's had the same layout for donkeys year so it's not the greatest GUI in the world but on the plus side it doesn't eat up your resources and you don't have to relearn it every time it updates, just wish it was a bit cheaper as there is a lot of new cheaper stuff out there now, I only use it because I bought it years ago when there wasn't as much choice,

Dave

 

Hi Dave ...

Thanks for this .... I had a feeling that my tracking is slightly off ... but my ASA mount is capable of improvement by using a MLPT feature, involving plate solving, that I am going to try out at the next opportunity. I am hopeful that this will improve things .... but because I am a slow learner I am taking things a step at a time :-)

At the minute, despite having a good look at the Maxim manual, I do not understand how to "make a bad pixel map" in Maxim ... but I will persevere !

Mike

 

[Davey-T]

Hi Mike, the instructions are pretty rubbish, once you have a master dark open it and the bad pixel map thingy.

Play with the threshold settings lowering the white point a bit at a time and run it on the dark frame until the hot pixels disappear.

Save it with a suitable name then run it on your subs, it can be run as a batch process if you can figure out how to do it to save doing each image separately.

HTH

Dave

[carastro]

I found that using sigma stacking removed the hot pixels.  Also dithering during capture will help by overlapping the hot pixels.

Are you guiding because there should not be so much movement between frames if you are. 

Carole 

[Stub Mandrel]

Just using darks and bias frames should remove all those dots. Don't abandon the data, just take some darks and bias frames and feed them to your stacking program.

That movement between frames is effectively dithering except the dots move in a straight line instead of being randomly arranged. You needs sigma stacking or similar if you rely on dithering alone.

 

[wimvb]

Have a look here:

It may clarify things

[ollypenrice]

On 07/12/2017 at 12:05, SlimPaling said:

 

(3) Am I correct in thinking that a "Dark Frame" is the same as a "Dark Flat Frame" ? ... the latter type was mentioned in one of the replies above.

(4) Is it a good idea to produce Master Dark Frame before I do any calibrations?

 

 

 

 

3) No, a 'dark flat frame' is a dark frame for your flats. (Why don't we call them darks for flats? That describes them unambiguously.) It is a dark taken at the same temperature and settings as your flats. Fortunately you simply don't need them. Just use a master bias as a dark for flats. At the short exposure times used for flats there will be no statistically significant difference between a master bias and a short dark. Some programmes may require you to rename your master bias, though, when used as a dark for flats.

4) I think so. Firstly there is no point in remaking darks every time you want to calibrate a new set. You also save disk space and can look at the resulting master dark before using it. If it looks dodgy you'll know to investigate. Same for flats. I always want to look at a master flat before using it. I know roughly what they should look like and sometimes they don't look right, so I do them again.

I'd have thought Sigma rather than Median would work best against your walking noise?

Olly

[Davey-T]

On ‎07‎/‎12‎/‎2017 at 13:14, SlimPaling said:

Hi Dave ...

Thanks for this .... I had a feeling that my tracking is slightly off ... but my ASA mount is capable of improvement by using a MLPT feature, involving plate solving, that I am going to try out at the next opportunity. I am hopeful that this will improve things .... but because I am a slow learner I am taking things a step at a time :-)

At the minute, despite having a good look at the Maxim manual, I do not understand how to "make a bad pixel map" in Maxim ... but I will persevere !

Mike

 

Hi Mike, there is clearly something wrong somewhere, the ASA should track unguided pretty much perfectly, have you taken any more images since this problem developed ?

Are you permanently mounted or do you set up each time ?

Dave

[SlimPaling]

1 hour ago, Davey-T said:

Hi Mike, there is clearly something wrong somewhere, the ASA should track unguided pretty much perfectly, have you taken any more images since this problem developed ?

Are you permanently mounted or do you set up each time ?

Dave

Hi Dave ....

My system is permanent and no I have not had any opportunity to do any more imaging .... it has been VERY cloudy here much of the time when I had some spare time do anything :-(

Yes ... my ASA mount is allegedly capable of some very accurate tracking ... but I am currently have a great deal of difficulty is understanding how to set this up and working properly. I am slowly working through things when I get the chance .... the manuals that came with the software to do this is not written for dummies like me !

Mike

[SlimPaling]

1 hour ago, ollypenrice said:

3) No, a 'dark flat frame' is a dark frame for your flats. (Why don't we call them darks for flats? That describes them unambiguously.) It is a dark taken at the same temperature and settings as your flats. Fortunately you simply don't need them. Just use a master bias as a dark for flats. At the short exposure times used for flats there will be no statistically significant difference between a master bias and a short dark. Some programmes may require you to rename your master bias, though, when used as a dark for flats.

Olly

Hi Olly ...

I have not tried any of this .... is it just a matter of using my Master Bias and re-naming it as a Master Dark ?... and then using it instead of my original Master Dark????

It is all a bit confusing for my small brain!!!

Mike

[ollypenrice]

10 minutes ago, SlimPaling said:

Hi Olly ...

I have not tried any of this .... is it just a matter of using my Master Bias and re-naming it as a Master Dark ?... and then using it instead of my original Master Dark????

It is all a bit confusing for my small brain!!!

Mike

I know it's confusing. But no, in my example I'm suggesting that you take a master bias and re-name a copy of it as 'Dark for flats.' This 'Dark for Flats' would be used to calibrate your flats and nothing else. 

Olly

 

[SlimPaling]

20 minutes ago, ollypenrice said:

I know it's confusing. But no, in my example I'm suggesting that you take a master bias and re-name a copy of it as 'Dark for flats.' This 'Dark for Flats' would be used to calibrate your flats and nothing else. 

Olly

 

OK .. I will give it a go next time I get a bit of time .... I have just found out that "we" have to rush down to London to deliver Chrissy presents to the grand kids before the snow storm hits us!!!

Mike

[ollypenrice]

16 minutes ago, SlimPaling said:

OK .. I will give it a go next time I get a bit of time .... I have just found out that "we" have to rush down to London to deliver Chrissy presents to the grand kids before the snow storm hits us!!!

Mike

Best of luck. Snow here as well!

Olly