An aid for CCD Flats questions

[Yoddha]

Hello,

I need your opinions and ideas!

I'm designing a new aid in APT that has to help in determining the right exposure time for the CCD flats. Here is the interface so far:

The workflow is to enter the desired ADU (which is 25-50% of the ADU of saturation). A range that is acceptable - percent of the desired ADU. The starting exposure duration, min and max exposures. Then is possible to define an use of Region of Interest for faster operation. The will be able to evaluate multiple filters in one run and then will be possible to create a plan prepopulating the determined exposure durations per every filter. All settings will be remembered for next time usage

When you hit on Run, APT starts to make exposures in the Min-Max interval trying to find these which meets the Target ADU.

So here are the questions. Am I missing something? Does it sounds flexible, easy to use and most importantly does it cover the needs to make good flats?

Currently there is no interface to control the Binning. The idea is to use the Binning set in the Camera tab. But do you think that there is need to have it in the Aid too, or it will complicate the interface for nothing?

Many thanks for your help in advance!

[narrowbandpaul]

Sounds like a nice tool.

I don't know where the idea of 25% saturation comes from. But as long as the user can adjust the desired value then that's fine. Why is the saturation ADU given as 40000? 16 bit gives you 65000. If the manufacturer gives you a camera with a gain that high without good reason then send It back.

Overall a nice idea.

Paul

[Yoddha]

Thank you Paul!

I have read several sources and think that this one summaries the things well - http://www.astrophoto.net/calibration.php

There is mentioned that the saturation ADU is the full well capacity divided by the gain. Then is advised to use 35-50% as Target ADU... As you said the Target ADU value is free to be changed by taste The value 40000 is just for a sample in this screenshot.

[narrowbandpaul]

What the author has done is take his full well in electrons and divided by the gain to get full well in ADU. This will be around 60-65000 for all 16bit cameras. So saturation ADU should always be around 60-65000.

As for the signal to use per flat. I see a lot of different numbers from different people. Most of it is old wives tales without a shred of fact. The quality of a flat can be expressed as Q=SN, where S is the signal per flat and N is the number of flats. This term actually appears in the mathematics of flat fielding. By using a low value of S you only hurt yourself. Especially when combined with just 10-15 flats as seems so common. I know of the need to avoid non linear effects but modern CCDs with anti blooming are still linear over the majority of the dynamic range. That's why I always use about 2/3rds saturation, ie around 40000 ADU. I also take a good number, 30-50. You need to minimise the injection of random noise during the process so a high quality flat is essential.

Hope that was of use.

Paul

[narrowbandpaul]

Further information from Richard Crisps site.

http://www.narrowbandimaging.com/images/Flat%20Fielding.pdf

Cheers

Paul

[Yoddha]

Hi Paul,

Many thanks! I'll remove the "Saturation ADU", it will simplifies the interface a bit

[narrowbandpaul]

Hi Ivo

No worries. I would just say saturation ADU is about 60-65000 but I don't think there is much point in displaying it. What's important is the target ADU.

Instead of quoting the absolute target ADU why not express it as a fraction of this fixed Saturation ADU. Ie specify the percentage of saturation you want rather than the actual value.

What happens if the required exposure time for your target ADU falls outside the min or max exposure. I have used an EL for narrowband flats and the S2 filter was like 30s whilst the O3 was just a few.

How does the software calculate the signal in the flat? Is it the average value or the max value? There might be some use in using the max value as then you can guarantee that no part of the flat is overexposed or nearly so.

As for the range, is this Target Signal +- Range or is it the difference between max and min? To clarify using the image you posted is it 20,000 +- 2000 or is it 20,000 +- 1000, so that max-min = 2000 centred on the target ADU. Not that this matters much but some clarification may help users.

Why not add an option to acquire the bias frames or flat darks at the same time. After all they are required for the flats to work properly. Perhaps an option for either flat darks or bias frames along with the number of these you want to acquire.

Once the software has initially found the correct integration time to satisfy the user demands does it keep the same exposure time for all images or will it seek to adjust the exposure time to always satisfy the demand?

That shouldn't be an issue for the el panel but perhaps for sky flats as the illumination falls. Let's say that during the shooting of sky flats the illumination falls outside the target range, will the software recognise this and adjust the exposure time? Or is the integration time then fixed once the capturing begins? Having the software compensate might be useful for sky flats but changing integration times would affect the usefulness of flat darks. Just a thought

Hope that helps

Cheers

Paul

[Yoddha]

Thank you again Paul! Hope that you don't mind to answer directly into your post?

 

Hi Ivo

No worries. I would just say saturation ADU is about 60-65000 but I don't think there is much point in displaying it. What's important is the target ADU.

Instead of quoting the absolute target ADU why not express it as a fraction of this fixed Saturation ADU. Ie specify the percentage of saturation you want rather than the actual value.

Looks like the common expectation is to be able to enter the desired ADU.

 

What happens if the required exposure time for your target ADU falls outside the min or max exposure. I have used an EL for narrowband flats and the S2 filter was like 30s whilst the O3 was just a few.

It will show an error message that it is not possible to get the desired ADU. Also it will let the user know which boundary is the problem - the min or max

 

How does the software calculate the signal in the flat? Is it the average value or the max value? There might be some use in using the max value as then you can guarantee that no part of the flat is overexposed or nearly so.

I think to use the averaged value. But it is interesting idea about the max value. I'll think on it.

 

As for the range, is this Target Signal +- Range or is it the difference between max and min? To clarify using the image you posted is it 20,000 +- 2000 or is it 20,000 +- 1000, so that max-min = 2000 centred on the target ADU. Not that this matters much but some clarification may help users.

The Range is percent of the ADU. If the target is 20000 and the range is 10%, then APT will accept as valid any exposure that gives averaged ADU between 18000 and 22000. This aid is part of my project APT. There is tooltip help for almost all features

 

Why not add an option to acquire the bias frames or flat darks at the same time. After all they are required for the flats to work properly. Perhaps an option for either flat darks or bias frames along with the number of these you want to acquire.

APT already has possibility to take dark and bias plans

 

Once the software has initially found the correct integration time to satisfy the user demands does it keep the same exposure time for all images or will it seek to adjust the exposure time to always satisfy the demand?

That shouldn't be an issue for the el panel but perhaps for sky flats as the illumination falls. Let's say that during the shooting of sky flats the illumination falls outside the target range, will the software recognise this and adjust the exposure time? Or is the integration time then fixed once the capturing begins? Having the software compensate might be useful for sky flats but changing integration times would affect the usefulness of flat darks. Just a thought

Hope that helps

Cheers

Paul

That is very interesting thought! Such compensation will not be an easy task... It requires continuous inspection of the real flats and also may require more calibration and parameters...

[Stephen]

Have you seen the Sky Flats Assistant plugin for Maxim DL?

Does what you're appearing to be trying to do here and works very well

[Yoddha]

Hi Stephen,

Thanks! I'll take a look. But I'm the author of APT and want to add flats automation into my project

[Stephen]

Hi Stephen,

Thanks! I'll take a look. But I'm the author of APT and want to add flats automation into my project

I know My point was maybe a conversation with the author of said plug in might be mutually beneficial

[Yoddha]

Hmmm! Interesting! I'll try to contact him

[narrowbandpaul]

Hi Ivo,

Thanks for your response. I will respond to your various points in order (replying in my post was a good idea, very easy to see what comment you were answering)

1. There might the odd user who doesn't know that the saturation ADU will be around 60-65000. They may only know that they want about 1/2 of max. Just a thought.

Also what about DSLR users? They normally have 14bit raw files. Do you show these raw files on a 16bit scale, or on the native 14bit. If 14bit the camera will of course saturate near 16000.

2. Sounds good

3. Fair enough, although I do think there is some value in quoting the max value, even just to show the user. Also perhaps the min value as well. I am thinking that the difference from the average value indicates the amount of vignetting. I.e. Max-min/average x100 would give the %vignetting. Of course it's just for interest, but people may like to know how bad theirs is. Of course even a truly flat image will show a small variation due to random shot noise but you could get a good idea as to how vignetted your images are.

4. Thanks for clarifying

5. Excellent. Though as an idea rather than having to take your flats and then reprogramme APT to take the corresponding bias or flat darks why not add 2 checkboxes, one to take a certain number of bias, and one to take a certain number of flat darks. If no checkbox is used then only the flats are taken. Some people may already have the bias or darks and so don't need to take them again. It just saves the user the hassle of setting up the flats, waiting for them to be done, then having to remember about bias/darks and then set them up. Perhaps a reminder somewhere that flats need calibrated with bias or dark frames in case they forget, get uncorredted images, and blame the software.

You might need a warning message after the flats are done and before the darks or bias are acquired. The message should tell the user to now cover the camera for the bias or darks. Might be OK with CCDs with mechanical shutters but for DSLRs any image will pop the mirror up, so it would need to be capped for the dark/bias acquisition.

Hope that made sense.

6. Yeah it may not be easy but potentially useful for sky flats. Something like....

For each flat...

1. Measure average (or max or min, doesn't matter).

2. If value within range then no change to exp time.

3. If not, then multiply exp time by Target ADU/Current ADU. This should be the exposure of the next flat

I think you need step 2 in there to stop the exp time changing for every exposure.

You porentially add a save feature too.

If Current ADU is within range then save

If not, then don't save and multiply exp time by ....

When the user is asked to specify how many flats they want it could be that you keep acquiring flats until there are that many saved flats. Ie only ones within the range specified are then saved.

I don't write software so I'm not sure how hard this is to actually implement. These could be nice features though.

Hope that was useful

Paul

[Yoddha]

Hi Paul,

You suggestions are much more than useful!

I'll use the numbering too

1. For DSLRs I use 24bits Bitmap to show the data on screen and am wondering if I can use the averaged RGB components? This will give values from 0 to 256 and in this case the flats "ADU" should be around 100-128. The RAW files are too big and heavy for run-time analysis.

3. Nice idea! But what will happen with dead and hot pixels? if there are hot that gives 65000 and dead that gives 0 then the calculation will be strange?

5. The bias and dark plans are easy to create, so maybe I'll leave the user to create them manually. The overall flow will become too heavy if I open the editor and create multiple plans. But will put a message to remind that flats need calibration There is already such reminder to cover/uncover the scope when there is switching between dark/bias and flat/light plans

6. Many thanks for the "know how"! I'll plan this for a later development, because this will need to change the plan execution flow which is too big change for the time-frame I planned for the next version...

Again, many thanks for your help!

[narrowbandpaul]

Hi Ivo.

1. Normally the green would be brightest as the QE is higher. You could quote the average or max values in each channel, thought I'm not sure if it required.

2. Good point. With short subs the chance of saturation would be low I think. It should still be valid though to use the max. If an exposure shows that a pixel is overexposed then there is no way you can get a sensible value to correct the flat. The best thing to do would be reduce the exp time so that the brightest pixel is at an appropriate level. Hope that made sense.

Alternatively you could show a histogram and the user can visually see where the peak is. Or you could median filter to remove the hot pixels.

I don't really have any software know how. Just some thoughts as to how it might be improved. Perhaps some these suggestions are easy to describe but difficult to implement.

Cheers

Paul

[Yoddha]

Hi Paul,

I'll google to see how Median filter has to be implemented. I already have code to calculate the median value of region (included in the PixelAid)...

With your help now I have the initial set of features defined When the tool is release will refine using the feedback