Sanity check needed with data from new camera

[Adam J]

6 minutes ago, jjosefsen said:

And that is why I'm again having this nagging feeling that something is off here..!

But that banding noise does dissapear in lights, if exposed enough.. But if dark substraction throws it back in there then..

Exactly and you cant get away without darks on this sensor due to the amp glows and those do turn up in Ha if you dont calibrate.

Edited May 16, 2019 by Adam J

[Adam J]

I think I may have seen something similar in the past with  noise from a poor power supply. But not on my own camera, someone else maybe on cloudy nights.

Edited May 16, 2019 by Adam J

[jjosefsen]

3 minutes ago, Adam J said:

I think I may have seen something similar in the past with  noise from a poor power supply. But not on my own camera, someone else maybe on cloudy nights.

Tried the power supply it came with, and running it through my bench PSU + Pegasus Astro PPB - same thing..

[Adam J]

3 minutes ago, jjosefsen said:

Tried the power supply it came with, and running it through my bench PSU + Pegasus Astro PPB - same thing..

It could also be another source of electrical noise close to the camera. But less likely. Maybe try moving it to a different location and running off another set of darks.

[vlaiv]

21 minutes ago, Adam J said:

Just tried this, with my ASI1600mm pro,  I dont see this kind of banding the result is flat as a pancake, maybe just a few hot pixels that have moved as I only used two stacks of 10.

That is what it is supposed to look like when everything is ok - completely "flat" and mean of 0.

40 minutes ago, jjosefsen said:

Same result..

Here is a link to a google sheet with the dark frame stats, nothing sticks out to me.

Dark frame statistics

 

I think I need to find someone else with this camera, to compare their findings.

That is highly strange. Data in spread sheet indeed looks regular, but I wonder why is it reported in 0-1 range rather than 0-4095 or 0-65535 range?

Maybe PI converts to 0-1 range straight away on opening, or have you done something to those darks - any sort of processing?

I'm also wondering if it is some strange setting in PI stacking that is causing this.

If it's not too much trouble for you, could you zip/rar all of those dark subs as recorded (like you did with those previous fits, five of each) and upload them somewhere? I would like to try dark - dark test in ImageJ - straight average to see if I can figure out what is going on ...

[jjosefsen]

3 minutes ago, vlaiv said:

That is what it is supposed to look like when everything is ok - completely "flat" and mean of 0.

That is highly strange. Data in spread sheet indeed looks regular, but I wonder why is it reported in 0-1 range rather than 0-4095 or 0-65535 range?

Maybe PI converts to 0-1 range straight away on opening, or have you done something to those darks - any sort of processing?

I'm also wondering if it is some strange setting in PI stacking that is causing this.

If it's not too much trouble for you, could you zip/rar all of those dark subs as recorded (like you did with those previous fits, five of each) and upload them somewhere? I would like to try dark - dark test in ImageJ - straight average to see if I can figure out what is going on ...

I'm pretty sure that just how the PI script exported the statistics.

No trouble for me at all, i'm just grateful for the help!

Darks

[vlaiv]

1 minute ago, jjosefsen said:

I'm pretty sure that just how the PI script exported the statistics.

No trouble for me at all, i'm just grateful for the help!

Darks

Some fast internet you have there

I'll take couple of minutes for me to download, and I'll report back my findings

[jjosefsen]

1 minute ago, vlaiv said:

Some fast internet you have there

I'll take couple of minutes for me to download, and I'll report back my findings

100/100 fiber 😉

[vlaiv]

That certainly beats mine 10/1

 

[jjosefsen]

Just now, vlaiv said:

That certainly beats mine 10/1

 

😱😂

[vlaiv]

Good news / bad news situation?

When stacked with average method - two stacks, first one 24 subs, second 24 subs and then subtracted, result gives mean of 0 (or close enough), but indeed banding is visible in stretch.

Just to do "sanity check" on noise. At unity gain this sensor should have about 1.4e of read noise. At -15C dark current rate is about 0.009e/px/s, or in 90s that is ~0.81e - total noise is then about 1.65e

Now I stacked 24 frames with average method - that should give us reduction in noise by factor of ~4.9 and then I "added" two these together (well subtracted, but for noise its the same thing as adding) - which means we increased the noise by factor of ~1.414

1.65 * 1.414 / 4.9  = ~0.48

That is lower than ~0.6562, so there is more noise in there than one would expect.

I wonder why is all of this.

Btw - how did you get 26ADU mean value of two stacks subtracted? It might be the case that I did not read your images correctly - that images that you posted from dark-dark calibration - were stats visible on image from actual result or something else?

 

[jjosefsen]

4 minutes ago, vlaiv said:

Good news / bad news situation?

When stacked with average method - two stacks, first one 24 subs, second 24 subs and then subtracted, result gives mean of 0 (or close enough), but indeed banding is visible in stretch.

Just to do "sanity check" on noise. At unity gain this sensor should have about 1.4e of read noise. At -15C dark current rate is about 0.009e/px/s, or in 90s that is ~0.81e - total noise is then about 1.65e

Now I stacked 24 frames with average method - that should give us reduction in noise by factor of ~4.9 and then I "added" two these together (well subtracted, but for noise its the same thing as adding) - which means we increased the noise by factor of ~1.414

1.65 * 1.414 / 4.9  = ~0.48

That is lower than ~0.6562, so there is more noise in there than one would expect.

I wonder why is all of this.

Btw - how did you get 26ADU mean value of two stacks subtracted? It might be the case that I did not read your images correctly - that images that you posted from dark-dark calibration - were stats visible on image from actual result or something else?

 

It was the stats from the result..

Here are the files. I had the Amp Glow Reduction / Dynamic pedistal feature turned on in APP, could be the reason. Or I am doing something funky with the statistics process in PI.

Dark-Dark

 

But can we conclude that the Dark substraction works? That would leave Flats calibration right?

[vlaiv]

Dark - dark calibration works and I'm starting to suspect that you used some advanced features that messed up calibration and made problems with flats calibration.

Btw - you will have those bias stripes in your subs. Here is animation of "dancing" stripes:

This is most probably due to electronics in the camera. Above "blink" gif was created by two stacks of 24 darks stretched to show lines in each one. Lines move between dark subs, this means that they are different on each readout from camera and are likely caused by the way electronics of the camera were designed.

Back on the original topic. Can you avoid anything fancy like Amp glow reduction, dynamic pedestal, dark optimization and such - and do regular average stacking and calibration? This is just for checking, you can do fancy stuff later - this is just to see if we can get properly calibrated subs to integrate without problem and provide nice smooth image?

So - stack darks with average (ok, sigma clip is ok if you insist, but for this test, you can do without that as well ). Stack dark flats with average method. Stack flats with average method. Subtract flat dark stack from flats stack to get master flat.

Then take each light and subtract master dark and divide by master flat. Then register / align them and do simple average stacking.

[jjosefsen]

9 minutes ago, vlaiv said:

Dark - dark calibration works and I'm starting to suspect that you used some advanced features that messed up calibration and made problems with flats calibration.

Btw - you will have those bias stripes in your subs. Here is animation of "dancing" stripes:

This is most probably due to electronics in the camera. Above "blink" gif was created by two stacks of 24 darks stretched to show lines in each one. Lines move between dark subs, this means that they are different on each readout from camera and are likely caused by the way electronics of the camera were designed.

Back on the original topic. Can you avoid anything fancy like Amp glow reduction, dynamic pedestal, dark optimization and such - and do regular average stacking and calibration? This is just for checking, you can do fancy stuff later - this is just to see if we can get properly calibrated subs to integrate without problem and provide nice smooth image?

So - stack darks with average (ok, sigma clip is ok if you insist, but for this test, you can do without that as well ). Stack dark flats with average method. Stack flats with average method. Subtract flat dark stack from flats stack to get master flat.

Then take each light and subtract master dark and divide by master flat. Then register / align them and do simple average stacking.

I already tried doing a whole manual workflow in PI, where I have much more control over the subs, but I accidentally deleted everything earlier.

My master dark at the time included the darks with the "too low temperature", so I can try again without - tomorrow as it is getting super late!

And will do it without any pixel rejection in the dark flats and darks, there are no cosmic ray strikes any way.

 

I discovered the "unstable" bias or read noise a while ago, but as it can be "swamped" I figured I could live with it.

Also did a lot of tests with different capture software, power supplies, cables, etc.. to see if anything external was causing that "flicker", but I didn't find any,  so as you say it must be in the camera electronics.

[jjosefsen]

Not much change unfortunately..

Result and 10 calibrated and registered lights (.xisf)

Linear Fit Rejection, Standard normalization:

 

Linear Fit, No Normalization:

 

I still don't think it can be prescriped to light pollution, something still broken.

[jjosefsen]

Calibrated and registered .fits

[Adam J]

1 hour ago, vlaiv said:

Just to do "sanity check" on noise. At unity gain this sensor should have about 1.4e of read noise. At -15C dark current rate is about 0.009e/px/s, or in 90s that is ~0.81e - total noise is then about 1.65e

Now I stacked 24 frames with average method - that should give us reduction in noise by factor of ~4.9 and then I "added" two these together (well subtracted, but for noise its the same thing as adding) - which means we increased the noise by factor of ~1.414

1.65 * 1.414 / 4.9  = ~0.48

That is lower than ~0.6562, so there is more noise in there than one would expect.

 

I thought he used unity gain which is more like 1.7 - 1.9e as opposed to 1.4e?

Adam

[Adam J]

1 hour ago, vlaiv said:

Dark - dark calibration works and I'm starting to suspect that you used some advanced features that messed up calibration and made problems with flats calibration.

Btw - you will have those bias stripes in your subs. Here is animation of "dancing" stripes:

This is most probably due to electronics in the camera. Above "blink" gif was created by two stacks of 24 darks stretched to show lines in each one. Lines move between dark subs, this means that they are different on each readout from camera and are likely caused by the way electronics of the camera were designed.

Back on the original topic. Can you avoid anything fancy like Amp glow reduction, dynamic pedestal, dark optimization and such - and do regular average stacking and calibration? This is just for checking, you can do fancy stuff later - this is just to see if we can get properly calibrated subs to integrate without problem and provide nice smooth image?

So - stack darks with average (ok, sigma clip is ok if you insist, but for this test, you can do without that as well ). Stack dark flats with average method. Stack flats with average method. Subtract flat dark stack from flats stack to get master flat.

Then take each light and subtract master dark and divide by master flat. Then register / align them and do simple average stacking.

Any chance that many more Darks would help if those bands are changing on a sub by sub basis? Would they average out? Dithering might help too.

[vlaiv]

3 minutes ago, Adam J said:

I thought he used unity gain which is more like 1.7 - 1.9e as opposed to 1.4e?

Adam

Is it? I actually tried to look up specs but ZWO website was down at that moment ...

It still is, luckily google remembers things

Indeed, you are right, it is ~1.7, so if we add 0.81e of dark current, that will end up at ~1.92e total noise per sub. So math then follows as - 1.92 * 1.414 / 4.9 = 0.554

Still a bit short ... but I don't know if it is significant at all ...

[vlaiv]

2 minutes ago, Adam J said:

Any chance that many more Darks would help if those bands are changing on a sub by sub basis? Would they average out? Dithering might help too.

It would certainly help for master dark - it would in the end settle with lines where statistics says they are most likely, however - each light will have dancing lines also, and calibration wont remove them.

However, due to fact that they are "dancing" around - those remaining lines will be averaged out given enough light frames, and in general, dithering will help a bit.

[vlaiv]

Hm, actually, I'm not sure there is a reason for concern, here is straight forward average integration of aligned subs - it is crazy linear stretch and indeed, it shows some banding, and some residual amp glow I would say, rather than vignetting, but it is down at the level of read noise ...

Same image but with non linear stretch in gimp:

I'm going to try a different stacking method to see if I can get better results

[vlaiv]

It turns out that one light subs is heavily clouded by high altitude cloud - making all sorts of gradients.

I'm doing some algorithmic magic now to see what sort of stack I can produce from the data, will post result

[vlaiv]

Here are results:

This is gimp version, there are still amp glow elements, but can be kept in check by careful histogram manipulation.

[Rodd]

2 hours ago, vlaiv said:

Here are results:

This is gimp version, there are still amp glow elements, but can be kept in check by careful histogram manipulation.

Looks pretty good to me. Careful use of dbe With a mask could help with the light zones

Rodd

Edited May 17, 2019 by Rodd

[vlaiv]

7 hours ago, Rodd said:

Looks pretty good to me. Careful use of dbe With a mask could help with the light zones

Rodd

Indeed, DBE did occur to me, but still, I had a chance to think about it some more - I still have a feel that something is not quite right. Although with careful stacking, over correction of flats is pushed back almost to the read noise floor - adding more subs will smooth things out and it will become apparent again.

At this point, I don't have a clue what might be cause for it. I will play a bit more with data today to see if I can guess level that has been "added" to lights that is causing that - maybe that will give us clue.