Calibration frames clipping data?

[Thorney]

I was out the other night to capture a widefield of the Bubble and Lobster Claw.  I got 3 hours and 20 minutes of data and stacked with 30 darks, 30 flats and 30 dark flats in DSS. When i came to process though in photoshop the data on the histogram seems to be way over to the left and clipping into it.  However i stacked the lights without any calibration frames and it leaves me room to play with on the histogram. 

Is this normal and i just need more data to move the histogram over  or have i gone wrong somewhere? Equipment used was a Redcat, ASI2600mc and Optolong L-extreme filter.

I have attached the two files in case anyone wants to take a look, the first is with and the second is without.

Many thanks.

Bubble.tif Bubble no calibration.tif

[MillHey Nebula]

What Offset /Black level did you use for your exposures?  This behaviour seems normal.  When you have stacked uncalibrated frames, the offset will still be included and so the histogram will have that additional "separation" from the left.  However, when you use calibration frames, the offset will has also been applied to the dark frames so when you do the subtraction, the offset will effectively be removed.  At this stage, this is absolutely fine and should not lead to any actual clipping of the data.

 

Edited October 6, 2021 by MillHey Nebula

[vlaiv]

This can happen - and it's not good thing - it is error in how calibration is performed.

Ideally you want to work with 32bit signed floating point when you start calibrating your data. Initially data is 16bit unsigned integer (we can't have negative number of photons hitting sensor, right?), but as soon as you start removing offset and doing calibration - due to noise, some of pixel values can end up being negative.

If calibration is performed in 16bit unsigned format - well, negative numbers will not be recorded properly and will clip to 0.

Maybe try performing calibration in another software (you can still stack calibrated data in DSS if you like) - and see if there is difference. I think that Siril will both calibrate and stack data and is free. Maybe give it a try?

[ollypenrice]

It might be a software matter as vlaiv says, but it might be more simple: how did you make the darks?  I did some experimenting and found a significant difference between darks a) made with the camera off the rig and with the metal chip cover screwed on and b) made on the scope (a refractor) with the lens cap on. The ones done properly, off-scope and under the chip cover, were measurably a little darker. The lighter darks would doubtless clip the lights.

I've read posts in which Newt users have done darks on the scope and really don't see this working when the bottom of the tube can admit light.

Olly

[vlaiv]

8 minutes ago, ollypenrice said:

The lighter darks would doubtless clip the lights.

Only if you used unsigned integer as data format. With 32bit signed floating point - you would simply end up offsetting data to negative values.

Setting levels properly will make histogram look natural (but would not correct other issues with mismatching darks).

[ollypenrice]

5 minutes ago, vlaiv said:

Only if you used unsigned integer as data format. With 32bit signed floating point - you would simply end up offsetting data to negative values.

Setting levels properly will make histogram look natural (but would not correct other issues with mismatching darks).

So having darks made with light getting in wouldn't over-subtract values from the lights?

[vlaiv]

3 minutes ago, ollypenrice said:

So having darks made with light getting in wouldn't over-subtract values from the lights?

Yes the would - but it would not lead to histogram clipping unless you are using some sort of rounding off at 0.

If using 32bit floating point data - negative values are OK. In fact - I end up with negative values in my regular image processing all the time and it does not pose a problem in any step. When I remove background - I aim for my background to have average value of 0. Due to noise floor - some pixel values are below 0 and some above 0 (and they average to 0).

This will not lead to histogram clipping though. When I import such data into Gimp - it will simply auto scale the data - taking lowest and highest pixel values (regardless of their sign) - and scaling that to 0-1 range.

Having lighter darks due to light leak - will cause other issues - like gradients or problems with flat correction - but should not cause clipped histogram by itself. Only using unsigned data format where values below 0 are clipped to 0, or deliberately clipping to 0 or some other value - will make histogram clipped.

[vlaiv]

Just now, vlaiv said:

When I remove background - I aim for my background to have average value of 0. Due to noise floor - some pixel values are below 0 and some above 0 (and they average to 0).

Just to make myself clear - I don't aim for 0 pixel value background in final image. This is still linear stage and I simply remove background sky light from the image - so I can say measure things - or get true RGB ratios without influence of atmosphere and so on ...

I can later in processing set background to any pixel value I choose to make image look natural.

[Thorney]

Thank you all for the replies. I’ve been using the ASIair for my capture software. Gain was at 100 and i believe there isn’t a way to change the offset so I’ve been running that at default which I think is set to 50.

The darks were made about a month ago with the camera attached and lens cap on. (I just used the master dark in DSS) 

I have used the darks before but it was the first time adding flats and dark flats to an image  so didn’t know if these were wrong? 
 

I’ll try add the frames to some old data to see what that the outcome is and I’ll take a look at Siril to see what that produces  

 

[vlaiv]

8 minutes ago, Thorney said:

I have used the darks before but it was the first time adding flats and dark flats to an image  so didn’t know if these were wrong?

Ok, this made me think. It is impossible for use of flats to clip histogram unless something is seriously wrong, so I downloaded fist image - and no, your histogram is not clipped.

Here are histograms per channel:

None of them is clipping. When you use flats - you just "flatten" the image and histogram becomes very narrow and sharp -as you remove vignetting from the image.

Maybe you just thought that histogram is clipping - but it's not. Here are stats on channels:

as you see - min is never 0 and above histograms don't look like clipping histograms - they have proper curve to them.

If you don't mind - I'll have quick go at processing the data as it looks nice and can't help myself

 

[david_taurus83]

The only way to take proper darks is with the camera off the imaging train and a metal cap on it. It's the only way to be sure. 

You might not be able to do that if you use flat darks but if they are done at night and you can cover the scope they should be OK for flat calibration. But long exposure darks, definitely camera off.

[vlaiv]

Here is quick process of the image - no clipping:

[Thorney]

8 minutes ago, vlaiv said:

 

Maybe you just thought that histogram is clipping - but it's not. 

 

My apologies if this is the case. I’m still at the basic stage of processing and the main histogram looked as if it was cutting into the data. 

 

12 minutes ago, vlaiv said:

 

If you don't mind - I'll have quick go at processing the data as it looks nice and can't help myself

 

Feel free, will be nice to see what can be done. 
 

7 minutes ago, david_taurus83 said:

But long exposure darks, definitely camera off.

I shall experiment with this. Thank you. 

[teoria_del_big_bang]

I agree with @vlaiv and can't see any actual clipping in the image with calibration.
There are a lot of values in the sky around 0.0007 which is small, but essentially you are using a dual pass NB filter so you can expect some pretty low values but I cannot see any pixels at 0.

In NB it is possible to end up clipping pixels in very dark areas as subtracting the master dark can take the pixel value to a -ive , I have started adding a pedastal now during pre-processing of NB images to prevent this in PI.

Steve

Edited October 6, 2021 by teoria_del_big_bang