DSLR: BIAS, FLATS, DARK FLATS and ?????? = brain melt down

[iapa]

Weather held most of the night and today.

I managed to get 

• 400+ hundred BIAS frames today, camera temp was+11C or +12C according to BYEOS. I want to create a good master I can use over then next 6 months.
• 150 or so FLATs: 4 layers white t-[removed word], pointing at NCP, clear even crisp wine pale blue sky (beautiful of itself :)) of ruse only with the images form last nights as I tear down after most sessions,
• 50 Dark Flats
• A load of Darks

I (try to) use Nebulosity.

Plan

• In preprocessing, BIAS will be the bias files, the LIGHTS I will select my DARKS. PreProc files will be Darks minus BIAS, batch Align and cCombine, stack these using average, no alignment. Result willl be a master Dark minus BIAS
• In preprocessing, BIAS will be the bias files, the LIGHTS I will select my FLATS. PreProc files will be Flats minus BIAS, batch Align and cCombine, stack these using average, no alignment.. Result willl be a master Flat minus BIAS
• In preprocessing, BIAS will be the bias files, the LIGHTS I will select my DARKS FLATS. PreProc files will be Flat Darks minus BIAS, batch Align and cCombine, stack these using average, no alignment.. Result willl be a master Dark Flat minus BIAS
• In preprocessing, load my master dark, Master BIAS,  then master flat with master BIAS and master dark flat. Not sure about the processing option at this point. Probably nones it's not debarred at this point. Then my lights, master bias, master dark, and 'Flat 1' (which should be flattish with dark noise and bias removed)

Where is this going to duplicate and reintroduce noise? As I can't see where it is not right.

Second and third steps will give flats with thermal issues reduced such as amp glow, and all remove the fixed signal level inherent in CCD/CMOS devices.

I realise that I am reintroducing the "shouldn't use DARKs with DSLR arguement". If I have removed the BIAS from these, I'm not doing the repeated subtraction of BIAS (or reintroducing it), so, will I need the BIAS limited DARKs if I am using Flat Darks?

There are many threads on the subject, and as soon as I think I have it, theres a flurry of contradictory comments and I loose it all again:(

LIGHTS = Data + heat noice + fixed 'electric noise + really random 5h!t +dust + vignetting

To eliminate find 'electric' noise, take BIAS

To eliminate heat noise, take darks and remove BIAS.

To reduce dust and vignetting, take flats. Take dark flats to sort the heat noise in the flats. Remove BIAS from the dark flats, remove the dark flats from the flats.

So, I have LIGHTS minus electric noise minus heat noise, dust and vignetting and a load of the random 5h!t.

Lots of these 'cleaned' lights reduces the remains stuff.

Result - clean low noise images, of star trails, aircraft lights and other rubbish..... but they're clean 

 

LOL

 

 

 

Edited November 2, 2016 by iapa

[Scooot]

I've no idea what the answer is but I found this to be informative so wonder whether it might help. If not ignore me  

http://astrofriend.eu/astronomy/tutorials/tutorial-pre-calibration-flat-dark/tutorial-pre-calibration-flat-dark.html

[iapa]

One of the references have used earlier on - found useful..

It's down to the practicalities of how to process it that is causing the brain melt (ignoring the various camps around use of DSLR darks with or without LENR)

[PhotoGav]

Mmm, it's all a bit confusing isn't it. However, I'm pretty sure that you are removing Bias too many times and I really don't think you need Dark Flats. Please don't ask me why as I struggle to get my head around it!

Here is my recipe for Nebulosity. You need Bias, Flats, Darks & Lights. It has worked for me. It's a bit long winded, but hopefully will give you good results!

1.    Create Bias Master

Batch / Align & Combine

Output Mode - ‘Save Stack’

Alignment – ‘None’

Stacking – ‘Average’

Uncheck ‘Fine-Tune Star Location’

Select All Bias Frames

Save as ‘BiasMaster’

 

2.    Pre-process Flats

Batch / Pre-Process Image Sets

Bias 1 = Select ‘BiasMaster’

Light 1 = Select all Flat Frames

Use Bias 1

 

3.    Create Flat Master

Batch / Align & Combine

Output Mode - ‘Save Stack’

Alignment – ‘None’

Stacking – ‘Average’

Uncheck ‘Fine-Tune Star Location’

Select All “pproc_” Flats

Save as ‘FlatMaster’

 

4.    Create Dark Master

Batch/Align & Combine

Output Mode - ‘Save Stack’

Alignment – ‘None’

Stacking – ‘Average’

Uncheck ‘Fine Tune Star Location’

Select All Dark Frames (Odd number is best, around 25-31 frames)

Save As ‘DarkMaster’

 

5.    Pre-Process Subs

Batch/Pre-Process Image Sets

Dark 1 – Select ‘DarkMaster’ File

Flat 1 – Select ‘Flat Master’ File

Light1 – Select Series of Subs

Use Dark1 & Flat 1 on Light1

 

6.    Normalise

Batch/Normalize Intensities

Select ‘pproc_’ subs

 

7.    De Bayer

Batch/Batch Demosaic & Square RAW Colour

Select ‘norm_pproc_’ subs

 

8.    Align Images

Batch/Align & Combine

Output – Save Each File

Alignment – Translation

Check ‘Fine Tune Star Location’

Select ‘recon_norm_pproc_’ subs

Use the Cross-Hair Cursor to select a recognisable star at high magnification towards a corner, go through all frames clicking on the correct star

 

9.    Stack Images

Batch/Align & Combine

Output – Save Stack

Alignment – None

Stacking Function – Std. Dev (1.5)

Check ‘Fine Tune Star Location’

Select ‘align_recon_norm_pproc_’ subs

Save As ‘Stacked’

[iapa]

Thanks Gav. I'll match that with what I was thinking.

What I was trying to do was explains my understanding of the process which I think is to

• remove BIAS from the FLATS, DARKS, DARK FLATS so all calibration frames are BIAS free.
• remove the DARK FLATS from FLATS to give FLATS minus dark leaving BAIS and DARK free FLATS
• removing the BIAS free DARKS, just leaves dust and vignetting to get rid of.
• using the master BIAS, master FLAT (which has no BIAS)  and master DARK (also no BIAS) should then remove BIAS, and DARK noise from each LIGHT, the (BIAS free) FLATS reducing the vignetting and dust.

 

 

[PhotoGav]

I think the problem with removing Bias from everything, as per your line one, is that you end up removing it multiple times from things later down the line. So when you remove bias from a flat and then remove the flat from the light, which has bias removed too, you are effectively removing the bias twice. Hence you need to remove the bias from the flat and then that processed flat from the light to create a light with no flat or bias in it. You then remove the simple dark stack from the light and that gives a sparkly light with no bias, dark or flat in it - perfect data for aligning and combining.

[iapa]

There's my point - if the FLAT has no BIAS I am not removing it from the LIGHT: just doing LIGHT minus FLAT minus BIAS

Lets say images captures are 'raw' and with calibration frames removed are'pure'e.g. rawDARK = pureDARK plus BIAS

So, if I am doing a

rawFLAT - BIAS = pureFLAT

rawLIGHT minus BIAS minus pureFLAT

i.e. removing BIAS once from the LIGHT.

Where there's a gotcha is subtracting unprocessed rawDARK and then BIAS; basically populating DARK, BIAS, FLAT and LIGHT fields in preprocessing.

rawLIGHT minus rawFLAT minus BIAS

as the rawDark has BIAS (i.e. pureDARK + BIAS) then you have

rawLIGHT minus BIAS minus (pureDARK + BIAS) = LIGHT minus BIAS minus pureDARK minus BIAS = LIGHT minus 2xBIAS minus DARK ; thus adding effectively BIAS back in.

That's my understanding anyway.

[Thalestris24]

DSS outlines the scheme options it uses for calibration. Interestingly, all the combinations it says are valid include darks!

Louise

[iapa]

Thanks Louise, another one of my sources which helped me under stand why the different calibration frames are used.

I am trying to work how they are used in Nebulosity 4 - to create my masters.

Had a shot last night with the steps I listed above, and a quick run through lights, they seemed better.

Proof in pudding when I get home tonight.

 

[Thalestris24]

6 minutes ago, iapa said:

Thanks Louise, another one of my sources which helped me under stand why the different calibration frames are used.

I am trying to work how they are used in Nebulosity 4 - to create my masters.

Had a shot last night with the steps I listed above, and a quick run through lights, they seemed better.

Proof in pudding when I get home tonight.

 

Hiya

Sorry, I don't know Nebulosity but the principles must be the same. The DSS diagrams show what combinations of calibration frames can be used to get a correct calibration but it isn't really explicit as to exactly how each type is dealt with. What does Nebulosity documentation say?

Louise

[iapa]

There in lies my dilemma, it says create you masters but preprocessing is useful. Doesn't have the specific steps so, as normal in this hobby, I am making it up as I go along (and making notes so I can repeat).

What I thought I needed to do, tied in with the steps that PhotoGav published - so, I cant be too badly off.

Admittedly we did go off at a tangent around whether too many BIAS removals were being done LOL.

[Thalestris24]

Hi

It seems to explain what to do and why in step 1 and 2 in the tutorial: http://www.stark-labs.com/help/nebulositytutorials/files/Processing_HowTo.pdf

If all else fails, why not use DSS?

Louise

[iapa]

Why? Comparison of the two apps. Does one give better results than the other? Theory - shouldn't.

And..... I paid for Nebulosity, hate to see the money wasted (that's be my Scots tightfistedness canniness)

[Thalestris24]

1 minute ago, iapa said:

Why? Comparison of the two apps. Does one give better results than the other? Theory - shouldn't.

And..... I paid for Nebulosity, hate to see the money wasted (that's be my Scots tightfistedness canniness)

Well, dss is free so no extra cost!  I guess different apps can use different algorithms and different levels of precision so they could give different results. DSS does at least define what it considers to be a correct (valid) combination of calibration frames. It also does it all for you - you just feed it the frames and by magic it produces a result!

Louise

[iapa]

And that's one thing I like about it - just feed it the right day and get something for manipulation - had end results that almost are Average to date (mainly due lack of calibration data which I have now).

 

[Thalestris24]

One problem with dss is that you can feed it poor data (insufficient lights, not enough calibration frames, an invalid combination of calibration frames) and it will produce a result without complaining! It relies on the user to get it right. I've no idea if it's the same with Nebulosity?

Louise

[iapa]

In DSS and Nebulosity to can review the images before processing - Nebulosity lets you rename, i run multiple passes checking for trailing, cloud cover, aircraft etc. and name each accordingly (sometimes I've been able to remove aircraft light trails).

[iapa]

@PhotoGav

Hi, was looking thought your steps again, and I notice you normalise just intensities.

Do you do the histogram as well ?

[PhotoGav]

1 hour ago, iapa said:

@PhotoGav

Hi, was looking thought your steps again, and I notice you normalise just intensities.

Do you do the histogram as well ?

Not in Nebulosity. I take the stacks into PhotoShop for processing and that is where histogram balancing takes place.

[iapa]

Thanks, so you are doing the on the final preprocessed image, not to create that image.

I've been doing the histogram matching as part of the pre-processing

Edited November 5, 2016 by iapa