Sanity check needed with data from new camera

[jjosefsen]

Hi,

So I've bought my first cooled astro camera, an Altair Astro Hypercam 1600M, based on the same Panasonic chip as the ASI1600MM..

I'm honestly not even sure if I have an issue with the data, or if it is just PI's autostretch that just over does it..

But this is what it looks like:

In contrast this is what a calibrated single frame looks like with a default autostretch:

The single sub looks fine I think, but the integration just looks strange, but also like it is over stretched by PI.

For the record the results are the same whether i use PixInsight for calibration and stacking or if I use Astro Pixel Processor.

I have tried many, many different stacking and normalization settings, and nothing changes it.

I am using 1.25" filters at F6.4, and with the substantially increased back focus of this camera (17,5mm) compared to the ASI1600 (6,5mm), I thought it might be vignetting, but I don't think it shows in the single subs..

 

Details:

AA Hypercam 1600M

Baader 1,25" filters in EFW mini.

Evostar 80ED @ 520mm

 

90s exposures

Gain 350 (about unity)

-15c

Offset 48

Calibrated with Flats exposed to 50% Histogram, Darks matching lights and DarkFlats matching flats and a bad pixel map.

All calibration frames in stacks of 50.

 

Link to Green master stack: 67 minutes Green master

Link to Green filter master flat: MasterFlat Green

Link to 90s master dark: Master dark 90s

 

I look forward to hearing what the experts think, something to worry about or not?

Thanks, Johannes

Edited May 15, 2019 by jjosefsen

[Adam J]

14 minutes ago, jjosefsen said:

Hi,

So I've bought my first cooled astro camera, an Altair Astro Hypercam 1600M, based on the same Panasonic chip as the ASI1600MM..

I'm honestly not even sure if I have an issue with the data, or if it is just PI's autostretch that just over does it..

But this is what it looks like:

In contrast this is what a calibrated single frame looks like with a default autostretch:

The single sub looks fine I think, but the integration just looks strange, but also like it is over stretched by PI.

For the record the results are the same whether i use PixInsight for calibration and stacking or if I use Astro Pixel Processor.

I have tried many, many different stacking and normalization settings, and nothing changes it.

I am using 1.25" filters at F6.4, and with the substantially increased back focus of this camera (17,5mm) compared to the ASI1600 (6,5mm), I thought it might be vignetting, but I don't think it shows in the single subs..

 

Details:

AA Hypercam 1600M

Baader 1,25" filters in EFW mini.

Evostar 80ED @ 520mm

 

90s exposures

Gain 350 (about unity)

-15c

Offset 48

Calibrated with Flats exposed to 50% Histogram, Darks matching lights and DarkFlats matching flats and a bad pixel map.

All calibration frames in stacks of 50.

 

Link to Green master stack: 67 minutes Green master

Link to Green filter master flat: MasterFlat Green

Link to 90s master dark: Master dark 90s

 

I look forward to hearing what the experts think, something to worry about or not?

Thanks, Johannes

I am guessing that you are unhappy with the background uniformity? Its not 100% clear from your post.

I would say that your flats might be slightly over correcting. The corners look ever so slightly brighter to me.

Also was this taken with the moon up?

Adam

 

[jjosefsen]

Just now, Adam J said:

I am guessing that you are unhappy with the background uniformity? Its not 100% clear from your post.

I would say that your flats might be slightly over correcting. The corners look ever so slightly brighter to me.

Also was this taken with the moon up?

Adam

 

Yes exactly the background, I should have been clearer, sorry.

There was moonlight yes, taken on Sunday, so 50% illuminated moon. But wouldn't the gradient be smoother from one side to the other?

[vlaiv]

Yes, something looks a bit off. I can't really be sure, but I suspect issue with offset. It is hard to tell because you posted master stack, but when I rescaled back to 0-4096 range, I got mean to be about 52  - which sort of works for offset of 48, but I got minimum value of stack to be ~1 - and that is something that I don't like - it's too low.

Issue with offset clipping might also cause problems with flat calibration.

Can you post single subs for dark, flat and flat dark?

[jjosefsen]

Thanks for looking guys..

Here is a set of 5x(lights, flats, darkflats and darks)

Data

 

The offset theory is quite interresting, I was reading something you wrote in another thread @vlaiv but, but I'm not sure i completely followed..

When I used the 130PDS for a short while, I calibrated out what appeared to be a worse vignetting with my DSLR, so I don't think the filters should cause a problem with this slow scope.

[vlaiv]

Did you do anything with you darks at all?

They are "acting" funny. Here is what I mean:

First thing to notice is that minimum in each dark sub is 0 - that should not be the case, this means offset is too low, not sure how low, or how much clipping there is, but in general you need to have values above 0 in your darks.

Second thing that I noticed - your set point temperature is not stable. First dark is set at -15C but actually taken at -17C, after that temperature starts to stabilize towards 15C with following actual values: 15.8C, 15.1C, 15C, 15C (for other darks).

Interesting thing is that your flat darks appear to have proper offset applied, or rather no zero values there:

In darks measurements I outlined standard deviation column, because it looks suspicious. Too much variation from frame to frame.

Actually - scratch that - I've just checked my 240s darks and they appear to show the same variation of standard deviation, so that is ok (although I would expect StdDev to behave more like in case of flat darks - more uniform).

Just tried to see how many pixels are there with 0 value - I can't find any by visual inspection, so it looks like there are only 1 or few. This means that it's not about the offset as flat darks show, I do however wonder why is there minimum zero value in darks (darks should have larger values than flat darks - dark current builds up over time and it is always positive).

Bottom line - can you try another stack, but exclude any calibration subs and light subs that don't have exact temperature -15C?

[jjosefsen]

1 hour ago, vlaiv said:

Did you do anything with you darks at all?

They are "acting" funny. Here is what I mean:

First thing to notice is that minimum in each dark sub is 0 - that should not be the case, this means offset is too low, not sure how low, or how much clipping there is, but in general you need to have values above 0 in your darks.

Second thing that I noticed - your set point temperature is not stable. First dark is set at -15C but actually taken at -17C, after that temperature starts to stabilize towards 15C with following actual values: 15.8C, 15.1C, 15C, 15C (for other darks).

Interesting thing is that your flat darks appear to have proper offset applied, or rather no zero values there:

In darks measurements I outlined standard deviation column, because it looks suspicious. Too much variation from frame to frame.

Actually - scratch that - I've just checked my 240s darks and they appear to show the same variation of standard deviation, so that is ok (although I would expect StdDev to behave more like in case of flat darks - more uniform).

Just tried to see how many pixels are there with 0 value - I can't find any by visual inspection, so it looks like there are only 1 or few. This means that it's not about the offset as flat darks show, I do however wonder why is there minimum zero value in darks (darks should have larger values than flat darks - dark current builds up over time and it is always positive).

Bottom line - can you try another stack, but exclude any calibration subs and light subs that don't have exact temperature -15C?

I can definitely try that tonight.

When I used my fan cooled 183m camera I also used darks, but these could easily vary over 1-3 degrees in temperature compared to the lights, and never showed this kind of problem.

But I will do a "clean" stack tonight.

[jjosefsen]

I just tried doing 5 90s darks with offset 64 and 5 0,5s darkflats with offset 64.

The minimum value of a dark frame is now 7 compared to 1 in pixinsight statistics process.

The minimum value of a darkflat is 29.. As you say, one would expect there to be more noise in the dark versus the darkflat.

Could it be some kind of on camera calibration that isn't switched off in the capture software?

[vlaiv]

Have no idea about in camera calibration thing

Btw, don't try to calibrate already captured lights with different offset darks - it will make a mess (well, you can try, but it won't be good )

Offset 64 seems to be a good option - I use it on my ASI1600 now.

 

[jjosefsen]

24 minutes ago, vlaiv said:

Have no idea about in camera calibration thing

Btw, don't try to calibrate already captured lights with different offset darks - it will make a mess (well, you can try, but it won't be good )

Offset 64 seems to be a good option - I use it on my ASI1600 now.

 

Yeah I know. Was just interrested in seeing if increasing it would make a difference with minimum values.

I am running a set of 90s darks with SharpCap now..

Edited May 16, 2019 by jjosefsen

[jjosefsen]

So this is without flats calibration:

 

And the image looks like you would expect from an image without flat calibration, but it does looke like the uneven background might be comming from the flats instead, like overcorrecting.

On another note:

Mean of 90s dark with Sharpcap: 1071 (16bit)

Mean of 90s dark with NINA: 67(16bit)

 

What is that about??

 

Edit: Wait, 4^2*67 = 1072 <-- Thats the right way to convert fra 12 bit to 16 bit right?

Edited May 16, 2019 by jjosefsen

[vlaiv]

In short 1071 / 16 = 66.9375

1600 model is 12 bit ADC sensor, which means that values are in range of 0-4096.

There are two conventions when writing lower number of bits into 16 bit format - fits that are written are 16 bit integer fits. There is MSB and LSB way of storing fewer bits - which is short for Most Significant Bit and Least Significant Bit.

With LSB - there are zeros "prepended" to actual value. With MSB zeros are "appended" to value.  This is like writing value of 27 in 4 decimal places - you can write it like 0027, or 2700 - first one is LSB, second one is MSB.

LSB keeps original value when you observe whole written number 27 == 0027, while MSB does not as 27 /= 2700. With MSB if we observe "written" value it is multiple of base of numeric system, or rather power of base of numeric system - power being number of digit places which were zeroed. In above example 2700 == 27 * 10^2, as base of numeric system is 10 and we added two zeros.

In binary form, since we are writing 12bit values in 16bit format in MSB, we are adding 4 zeros to the end, so multiplication factor will be 2^4 = 16. When 12bit value is stored in MSB format you need to divide it with 16 to get actual value.

This is why two means differ - they are in fact the same thing, only "written" differently. For imaging purposes, image will not change if you multiply everything with a constant factor, even in processing, after stacking, 32bit float point values are usually treated as 0-1 range. Only thing that you need to be careful about is if you do some absolute measurements of electron count - then you need to divide MSB values with 16 to get exact ADU to be converted to electrons via e/ADU value from Gain setting. For everything else, you don't need to worry.

So one fits was written in MSB convention - one giving 1071 mean value, and other, by Nina was written in LSB convention - no need for dividing with 16 in that case, but both values are "the same" (after conversion).

 

[jjosefsen]

Brilliant - thanks for that explanation. 🙂

What do you think of the overcorrecting flats hypothesis?

Edited May 16, 2019 by jjosefsen

[vlaiv]

Yes, I completely agree that flats are over correcting - question is why.

Over correction happens for two reasons - either signal in lights is too strong, or flats are too weak (A/B is greater than it should be, so either A is bigger, or B is smaller).

Lights can be "too strong" if:

1. dark not properly subtracted

2. there is light leak while capturing lights but not when taking flats

3. Lights are captured on higher temperature than matching darks

4.  Gain / offset / exposure length is mismatched between lights and darks in such way that lights get stronger signal (higher gain, higher offset, longer exposure)

Good news is that wrong offset that causes clipping is actually making darks stronger, so lights become weaker and that results in under correction - which is not the case here

Flats can be too weak if:

- too much flat darks is subtracted (here we can have mismatch in gain/offset/exposure or wrong offset with clipping - but examining flat darks shows its not the case)

I think that using darks at lower temperature will make them "less strong" (lower dark current) and that will leave some residual dark current in lights after dark subtraction - making them "stronger" than they should be - resulting in over correction.

[jjosefsen]

4 minutes ago, vlaiv said:

I think that using darks at lower temperature will make them "less strong" (lower dark current) and that will leave some residual dark current in lights after dark subtraction - making them "stronger" than they should be - resulting in over correction. 

Are you referring to one or two darks being at a lower temperature?

I would imagine that 1 or 2 subs out of 50 wouldn't make too much of a difference?

In either case I did try a master dark without those subs included, and it did not make a difference.

 

I will keep working on it, but I am really stomped here.. 🤨

[vlaiv]

4 minutes ago, jjosefsen said:

Are you referring to one or two darks being at a lower temperature?

I would imagine that 1 or 2 subs out of 50 wouldn't make too much of a difference?

In either case I did try a master dark without those subs included, and it did not make a difference.

 

I will keep working on it, but I am really stomped here.. 🤨

Maybe try "dark - dark" calibration to see if you can find out anything.

Stack first 25 darks (or 23 if we exclude "bad" temperature ones) and second 25 to two separate stacks and then subtract those two stacks. Result should have mean value of 0 and no perceivable patterns in the image (bias removed, amp glow removed - only pure noise with mean value of 0).

[jjosefsen]

7 minutes ago, vlaiv said:

Maybe try "dark - dark" calibration to see if you can find out anything.

Stack first 25 darks (or 23 if we exclude "bad" temperature ones) and second 25 to two separate stacks and then subtract those two stacks. Result should have mean value of 0 and no perceivable patterns in the image (bias removed, amp glow removed - only pure noise with mean value of 0).

Well s**t..

I guess it is not supposed to look like this then..

[Adam J]

8 hours ago, vlaiv said:

Just tried to see how many pixels are there with 0 value - I can't find any by visual inspection, so it looks like there are only 1 or few. This means that it's not about the offset as flat darks show, I do however wonder why is there minimum zero value in darks (darks should have larger values than flat darks - dark current builds up over time and it is always positive).

 

Is it not possible that the zero could be a single completely dead pixel?

[vlaiv]

1 minute ago, Adam J said:

Is it not possible that the zero could be a single completely dead pixel?

Quite so, but dead pixel would also show up on flat darks - and none of those have 0 valued pixels.

 

1 hour ago, jjosefsen said:

Well s**t..

I guess it is not supposed to look like this then..

What is that? Single stack of darks or two dark stacks subtracted?

How did you stack them (simple average here is best way)?

[jjosefsen]

1 minute ago, vlaiv said:

Quite so, but dead pixel would also show up on flat darks - and none of those have 0 valued pixels.

 

What is that? Single stack of darks or two dark stacks subtracted?

How did you stack them (simple average here is best way)?

Split dark stack into two.. Calibrated with average and sigma clipping - both stacks.

Subtracted one stack from the other, that was the result.

No much glow left, but alot of banding on the left side..

[vlaiv]

1 minute ago, jjosefsen said:

Split dark stack into two.. Calibrated with average and sigma clipping - both stacks.

Subtracted one stack from the other, that was the result.

No much glow left, but alot of banding on the left side..

Well, yes - banding on left and also mean ADU value of 26 - both not good signs.

I'm not familiar with PI, but is there a way to do measurement on list of files? Can you do stats on each dark file - like in batch mode, so we can see what the average ADU value of each dark is, and if there is any that is significantly different?

Either darks are not stable for some reason with this camera, or perhaps there is one dark frame that is causing trouble.

Since average of difference is positive, you can use this to deduce something about offending dark frame if there is single one - depending if you subtracted second stack from first or other way around. Let's assume you subtracted second stack from first. This means that one dark in first stack is much stronger, or one dark in second stack is much weaker.

If we can isolate one (or a few) strange darks - then you can skip those and try to do good dark - dark calibration. From fits header it might be obvious why that particular dark is not good.

[jjosefsen]

24 minutes ago, vlaiv said:

Well, yes - banding on left and also mean ADU value of 26 - both not good signs.

I'm not familiar with PI, but is there a way to do measurement on list of files? Can you do stats on each dark file - like in batch mode, so we can see what the average ADU value of each dark is, and if there is any that is significantly different?

Either darks are not stable for some reason with this camera, or perhaps there is one dark frame that is causing trouble.

Since average of difference is positive, you can use this to deduce something about offending dark frame if there is single one - depending if you subtracted second stack from first or other way around. Let's assume you subtracted second stack from first. This means that one dark in first stack is much stronger, or one dark in second stack is much weaker.

If we can isolate one (or a few) strange darks - then you can skip those and try to do good dark - dark calibration. From fits header it might be obvious why that particular dark is not good.

I redid the dark - dark substraction test, as I realized I hadn't omitted the two where the temperature was a little off.

Same result..

Here is a link to a google sheet with the dark frame stats, nothing sticks out to me.

Dark frame statistics

 

I think I need to find someone else with this camera, to compare their findings.

[Adam J]

26 minutes ago, vlaiv said:

Quite so, but dead pixel would also show up on flat darks - and none of those have 0 valued pixels.

 

What is that? Single stack of darks or two dark stacks subtracted?

How did you stack them (simple average here is best way)?

Just tried this, with my ASI1600mm pro,  I dont see this kind of banding the result is flat as a pancake, maybe just a few hot pixels that have moved as I only used two stacks of 10.

[Adam J]

[jjosefsen]

1 minute ago, Adam J said:

And that is why I'm again having this nagging feeling that something is off here..!

But that banding noise does dissapear in lights, if exposed enough.. But if dark substraction throws it back in there then..