Kodak KAF8300 v's Sony ICX814 *Noise comparison*

[GTom]

I feel the market is starving for a real-life review of all sub-3k$ imaging solutions for both NB and "normal" LRGB use, there are quite a lot to choose from:

- high QE sony based Atik's

- KAF-8300

- ASI1600MM-cool cmos

- sony a7s, even monochrome-modded.

Btw where are those BSI chips populating the legends with 90+% QE?

 

 

[swag72]

6 minutes ago, vlaiv said:

..................Yes, I know what you mean, I on the other hand appreciate the fact that there is a way to predict things prior to actually conducting the experiment .............

To be fair, I never set out to conduct this as an experiment.... Just just found 2 lots of data that I could compare in a real world sort of way.... well in my opinion it's a real world sort of way. Same equipment, same length of exposure time, just a different sensor.

Are you saying that as it stands for this purpose the comparison is not valid?

[vlaiv]

2 minutes ago, swag72 said:

To be fair, I never set out to conduct this as an experiment.... Just just found 2 lots of data that I could compare in a real world sort of way.... well in my opinion it's a real world sort of way. Same equipment, same length of exposure time, just a different sensor.

Are you saying that as it stands for this purpose the comparison is not valid?

Not at all, I think that comparison in it self is valid, how can it not be? You showed that in this particular case, using given equipment, conditions and processing, kaf8300 produced less noisy image. I'm just saying that under given conditions results might not be representative of general kaf8300 vs icx814 noise if there is such a thing (and I believe we need more precision in determining what are we actually comparing between). Also, I was trying to point out that more meaningful comparison would be to make sure initial conditions match as much as possible, and that final result matches as much as possible. Given that you used the same equipment part of initial conditions is satisfied, but we don't have information on external conditions, i.e. sky transparency/extinction. Also final result would be more matched if binning was performed in order to bring it to same resolution.

So if two of the above were fully satisfied we might assert that "under given conditions, kaf8300 produces less noisy image" as opposed to "given these conditions for kaf8300 and those for icx814 - kaf produces less noisy image" (still valid assertion but less useful one as comparison).

I also agree that kaf8300 should not be dismissed as noisy sensor because SNR depends on so many factors. For me sensor is a sensor is a sensor - meaning any sensor (within certain range of variations of it's characteristics - read/dark noise, qe, frequency response, ...) can be used and as long as you know characteristics of sensor and are ready to sacrifice something else (usually time) to arrive to the same result as with any other sensor.

[swag72]

As I no longer have the 2 sensors, the ability to carry out this experiment under similar controlled conditions is not possible for me......so from a scientific POV I accept that sweeping statements regarding the comaprison between the two sensors cannot be made..... however as you say, in this example it's clear to see that one sensor produced a less noisy image than another.

How about the following then as another interesting comparison (for this image only) ..... I stack x number of KAF8300 subs until the noise is as equal as I can get it on the PI script and that will show that with this data x number of subs with the KAF8300 gives a similar SNR as 14x1800s subs with the ICX814. 

[martin_h]

If I had to get another camera, I would probably get another 8300 chipped one. I know the noise in an image can be measured, but I think most people who spout on about noise tend to just look at a zoomed image, and probably that image is an hour's worth of data stretched to make a 4 hour image!

[vlaiv]

Not sure what are you trying to achieve. Maybe this will help: If I had two sets of images taken with these two sensors and I was trying to compare them for noise levels I would do the following:

I would use mean method for calibration and stacking (not sigma, median or any other but simple mean). I would bin icx814 image to produce same resolution as kaf8300 - not sure what tools are you using, most offer simple bin 2x2, 3x3, 4x4 - but there is binning algorithm that can handle fractional binning coefficients (startools has this implemented). Then I would do two point scaling to bring them to same intensity levels - this means take same region in both images, one being dark, one being bright - get median pixel value in this regions, and do some linear math to one of the images in order to match median values in both areas. Then take same segment for visual comparison and noise estimator function on that segment and compare results.

[swag72]

Just now, vlaiv said:

I would use mean method for calibration and stacking (not sigma, median or any other but simple mean). I would bin icx814 image to produce same resolution as kaf8300 - not sure what tools are you using, most offer simple bin 2x2, 3x3, 4x4 - but there is binning algorithm that can handle fractional binning coefficients (startools has this implemented). Then I would do two point scaling to bring them to same intensity levels - this means take same region in both images, one being dark, one being bright - get median pixel value in this regions, and do some linear math to one of the images in order to match median values in both areas. Then take same segment for visual comparison and noise estimator function on that segment and compare results.

I'm trying to achieve some for of best representation of noise, but without having to retake any frames...... I don't have the ability to change anything about the 14x1800s I have with the ICX814 sensor..... it is as is. No binning possible, no calibration changes etc. 

Perhaps I am looking at the impossible...... I'll stop now then!

[vlaiv]

Just now, swag72 said:

I'm trying to achieve some for of best representation of noise, but without having to retake any frames...... I don't have the ability to change anything about the 14x1800s I have with the ICX814 sensor..... it is as is. No binning possible, no calibration changes etc. 

Perhaps I am looking at the impossible...... I'll stop now then!

Ok, it's not impossible - just do the above I mentioned. You can do software bin, it will be less efficient then hardware one, but if you want to match resolutions precisely that is your only option (hardware bin is only in integer values). Even if you already have stacked 14x1800s you can do this. If you can't redo calibration and stacking, it does not matter, don't think there would be huge impact depending on stacking method used (mean generally gives best SNR, but other have their advantages and difference is small). Last thing you want to do is match intensity levels properly. If we assume both sensors are linear (and we will) - two point scaling is all you need - so again, just to make my self clear - take two regions, one being bright, one being dark - aim for uniform intensity regions, but don't worry much about it as long as each of two regions is aligned properly between icx814 and kaf8300 frames. Read of median pixel values in each region (bright and dark) and each frame (kaf and icx) - you now have two linear equations with two unknowns to solve - this will give you level and slope - so after that you should subtract level from frame and multiply by slope and after that you should get roughly the same median values in dark region for both kaf and icx and in light region for kaf and icx. When you have done all of this then just repeat what you have done in the first place - select some piece of image and compare it both visually and with noise estimator.

[Davey-T]

10 minutes ago, swag72 said:

Perhaps I am looking at the impossible...... I'll stop now then!

Stop anytime Sara you're preaching to the choir here, already got a QSI683 and planning on getting another one 

Dave

[RichLD]

Hey Sara, I really appreciate your observations, I think they are most useful. Imagine the new imager scanning this forum, wondering which setup to go for. This hobby can be very confusing when we first start can't it? Posts like this help give people the confidence to take the plunge into the money pit that is AP 

[GTom]

1 hour ago, RichLD said:

Hey Sara, I really appreciate your observations, I think they are most useful. Imagine the new imager scanning this forum, wondering which setup to go for. This hobby can be very confusing when we first start can't it? Posts like this help give people the confidence to take the plunge into the money pit that is AP 

I am in this situation now, indeed a difficult decision if you don't have 10k$ to burn. There are amazing results from the a7s, shooting 30sec unguided(!!) subs at iso3200-6400. Price is actually lower than of a kaf8300 camera and you get a full frame area, which is quite a homework to supply with appropriate filters, fittings & optics.

Personally I am seeking the next logical step from my d5100. One option is monochrome modding & cooling. This would be quite cheap but probably not as efficient as a dedicated ccd. I am also  considering to get two cameras, one for faint + small objects (414ex?) another for wide field. Speaking of wide field, question is how much is really needed. The list of 2+degree diameter objects I can't cover with my 90/600+reducer+apsc setup is not too long. The 4/3 format of the 8300 and asi1600 is really a sweet spot.

[swag72]

Just as a completion to this thread...... Thanks to Vlad @vlaiv and his suggestions about how to make a more scientific comparison he did the business to the two stacks of data to get them into a better position to make a fair comparison. 

Vlad did the following..... binning 1.4x to get the same pixel size and also doing a two point scaling so that the background and foreground levels were the same. From there I stretched them to make the comparison easier.

Based on this more scientific approach the comparison should be something more than just real life and mirror what theory suggests we should see.

You can see a larger version of the visual on my website here

So we have gone from a real world example to a scientific example ........... 

[dph1nm]

On 26/11/2016 at 21:04, swag72 said:

Vlad did the following..... binning 1.4x to get the same pixel size

Please note that non-integer binning does NOT preserve S/N. It introduces correlated noise (effectively a form of a smoothing). Only integer binning will preserve the true S/N.

NigelM

[vlaiv]

20 minutes ago, dph1nm said:

Please note that non-integer binning does NOT preserve S/N. It introduces correlated noise (effectively a form of a smoothing). Only integer binning will preserve the true S/N.

NigelM

Not sure why would you say that. I did not use any of usual resampling methods - linear, quadratic, cubic - none of classical point sample scaling techniques. Non integer binning (in my view at least, hope you will clarify your position on the matter) increases SNR same way regular binning does if you apply true binning algorithm - which in this case is area weighted average, or for this particular example you can think of it as follows (since it is bin factor of 1.4) - scale image to x5 size using "repeat pixel" technique (effectively nearest neighbour) and then do regular bin x7 (average each 7 adjacent pixels, or sum them it does not matter) . Alternatively algorithm for any ratio would be - treat image not as point samples but rather surface square pixels with certain level of signal each - superimpose new rectangular grid over existing one with squares being larger than original by binning factor. Calculate amount of signal that falls onto each new square by using surface method (how much of surface of original pixels are covered times values of each of original pixel). Result would be the same (read noise excluded) as if larger pixels have been used (this is what binning effectively does) - signal averages out, as well as noise - with signal you have loss of resolution, but with noise  - you have less of it (it averages out to 0) - hence higher signal to noise ratio at expense of resolution.

[vlaiv]

Here is an interesting result

First of all sorry about digressing from thread (but not by much, it is still relevant to actual thread topic).

I decided to run some tests to confirm my statement on fractional binning and got surprising results (still not sure what is going on here). I used ImageJ and generated 512x512 Poisson noise image centered around 10000 value. Measurement of that image gave me expected results : mean value of ~10000 and standard deviation of ~100 - just as one would expect noise being square root of 10000. Then I did 2x2 bin and 3x3 bin just to verify everything, and measurement of images gave expected results: mean of ~10000, and standard deviation of ~50 and ~33 respectively - SNR increase by 2 and 3. Then I did fractional binning with factors 1.5 and 1.333 and got quite unexpected results. In each case I expected to have SNR increase by same amount (1.5 and 1.333) but what I've got is: mean 10000 (as one would expect), and standard deviations of ~55.6 and ~58.324, giving SNR increase of ~1.8 and ~1.715 - quite unexpected. So we actually have bigger SNR gain in case of fractional binning. Either my expectation is wrong here (for fractional binning expecting to have SNR gain same as reduction in resolution as in integer values binning) - or there is problem with ImageJ and it's random number generator function (not being that random)

This shows that fractional binning works in increasing SNR, well, it works surprisingly well, more than I would expect

[swag72]

2 hours ago, dph1nm said:

Please note that non-integer binning does NOT preserve S/N. It introduces correlated noise (effectively a form of a smoothing). Only integer binning will preserve the true S/N.

NigelM

As a total non science person...... I bowed down to what Vlad said.... I have no idea what works or what doesn't........ why or how

But the 2 results speak for themselves

[dph1nm]

23 hours ago, vlaiv said:

This shows that fractional binning works in increasing SNR, well, it works surprisingly well, more than I would expect

Fractional binning effective smooths the data, so the S/N will appear to be better than expected. After all, I could smooth your 512x512 grid until every value was 10000 and achieve infinite signal-to-noise. Of course, any real signal in the data would have long since vanished.

On 29/11/2016 at 14:32, vlaiv said:

Calculate amount of signal that falls onto each new square by using surface method (how much of surface of original pixels are covered times values of each of original pixel).

Unfortunately for this method you can never know how the flux should have been distributed over the original pixel, so you cannot recover the true value which would have fallen into your new square, only an approximation to it (this is essentially bi-linear interpolation). The result is degraded data and Poisson statistics no longer work for single pixels.

NIgelM

[vlaiv]

9 minutes ago, dph1nm said:

Fractional binning effective smooths the data, so the S/N will appear to be better than expected. After all, I could smooth your 512x512 grid until every value was 10000 and achieve infinite signal-to-noise. Of course, any real signal in the data would have long since vanished.

So does the integer binning - in both cases you end up with smaller resolution - if you bin all the way to a single pixel - yes, it will be completely uniform, but that is not the point, the point was in this case to bring both sensors to same resolution, and in doing so trying to equalize noise coming from target - shot noise, smaller the pixel - less signal it will receive, hence smaller SNR, but if one is trying to asses what is the noisier sensor - we should at least exclude things that have nothing to do with sensor it self - like shot noise. Granted sensor with smaller pixels that we bin afterwards will always be at disadvantage - due to read noise. After all, binning is trading resolution for higher SNR, and it implies loss of detail due to loss of resolution (opposite of drizzling where one tries to gain resolution in oversampled case by trading in some of the SNR).

16 minutes ago, dph1nm said:

Unfortunately for this method you can never know how the flux should have been distributed over the original pixel, so you cannot recover the true value which would have fallen into your new square, only an approximation to it (this is essentially bi-linear interpolation). The result is degraded data and Poisson statistics no longer work for single pixels.

I agree on this one, and one possible way to remedy that is to do binning prior to stacking, or as integral part of stacking (when doing image registration / transformation - similar what drizzle does) - current algorithms do this by treating image as point sample matrix and often use some kind of filtering when doing transforms (one can think of interpolating techniques as filtering). In case of integer binning - one ends up with whole signal in area being kept in area. With fractional binning - we assume that signal is uniform over pixel surface - that might be a good approximation in cases of oversampling (seeing and optics psf insure good smoothness). But in any case we are introducing a bit of additional noise - by guessing that it is uniform - this is why I expected SNR after fractional bin to be a bit lower to that of simple theoretical assumption - not higher.

 

@dph1nm

Could you please give a quick glance to what I did in ImageJ testing and point out any methodology flaws, or rather explain why I'm getting lower noise than expected? I'll repeat process in a bit more detail just to make sure.

512x512 image is generated using Poisson distribution centered around 10000 - measurement is performed to asses signal and noise values (mean and standard deviation) with results of 10000 and 100 units respectively.

Initial image is copied and binned to x2 original size (256x256) - measurement is performed, and following values obtained: mean 10000, stddev 50 (as expected)

Same process is repeated with binning to x3 - values obtained: mean 10000, stddev 33.33 (as expected)

Then following is performed: on copy of initial image we do nearest neighbour scaling to 1024x1024 - ensuring that each pixel is repeated twice (no filtering is done in scaling, like it would be a case in linear, quadratic or cubic scaling), and then bin resulting image x3 to get effective image size of 341x341, measurement is performed, and obtained values are: mean 10000, stddev 55.6 (instead of expected 66.66 value).

[andrew s]

On 11/29/2016 at 15:43, vlaiv said:

This shows that fractional binning works in increasing SNR, well, it works surprisingly well, more than I would expect

As said above by NigelM (see below)

On 11/29/2016 at 13:58, dph1nm said:

Please note that non-integer binning does NOT preserve S/N. It introduces correlated noise (effectively a form of a smoothing). Only integer binning will preserve the true S/N.

NigelM

It make the S/N better as it smooths the data.

Regards Andrew

[vlaiv]

5 minutes ago, andrew s said:

It make the S/N better as it smooths the data.

Regards Andrew

Ok, first of all, don't really follow what you mean by smoothing the data.

My statement was regarding the level of noise reduction when performing fractional bin. Every binning will reduce the noise (at expense of loosing resolution - is this smoothing you are talking about?). Both integer and fractional bin will do this. Not sure what NigelM thought when he said: Only integer binning will preserve the true S/N - S/N will not be preserved - rather it will be increased when you bin (noise will decrease - hence S/N will increase). According to maths - x2 bin will increase SNR by two times - it will lower noise by factor of 2 - as I demonstrated with simple experiment - using image with Poisson noise. Likewise x3 bin will have factor of 3 in reduction of noise and hence boost of SNR. What I did not expect is that binning for example by factor of 1.5 will reduce noise by factor of 1.8, instead of factor of 1.5 - this is what I meant by saying that it works better than I expected. Still not sure why that is the case - "experiment" is easily repeatable so anyone can use ImageJ to verify these results, only question remains what is wrong with either my methodology / reasoning or idea that binning by a certain factor reduces noise by that same factor (due to planar nature of image, and quadratic dependence of noise).

[andrew s]

11 minutes ago, vlaiv said:

Ok, first of all, don't really follow what you mean by smoothing the data.

My statement was regarding the level of noise reduction when performing fractional bin. Every binning will reduce the noise (at expense of loosing resolution - is this smoothing you are talking about?). Both integer and fractional bin will do this. Not sure what NigelM thought when he said: Only integer binning will preserve the true S/N - S/N will not be preserved - rather it will be increased when you bin (noise will decrease - hence S/N will increase). According to maths - x2 bin will increase SNR by two times - it will lower noise by factor of 2 - as I demonstrated with simple experiment - using image with Poisson noise. Likewise x3 bin will have factor of 3 in reduction of noise and hence boost of SNR. What I did not expect is that binning for example by factor of 1.5 will reduce noise by factor of 1.8, instead of factor of 1.5 - this is what I meant by saying that it works better than I expected. Still not sure why that is the case - "experiment" is easily repeatable so anyone can use ImageJ to verify these results, only question remains what is wrong with either my methodology / reasoning or idea that binning by a certain factor reduces noise by that same factor (due to planar nature of image, and quadratic dependence of noise).

Sorry to be more more explicit. As it splits data from adjacent pixels into the binned pixels it correlates the information between adjacent new pixels. This reduces the noise over and above what you would expect and the noise is no longer follows Poisson statistics assuming it did to start with. When you bin whole pixels into new pixels you don't add any correlation between the new pixels.

Regards Andrew

PS I think your results are correct.

[vlaiv]

7 minutes ago, andrew s said:

Sorry to be more more explicit. As it splits data from adjacent pixels into the binned pixels it correlates the information between adjacent new pixels. This reduces the noise over and above what you would expect and the noise is no longer follows Poisson statistics assume it did to start with. When you bin whole pixels into new pixels you don't add any correlation between the new pixels.

Regards Andrew

Thanks for pointing that out. Now it is obvious what you mean by smoothing the data - both signal and noise are effectively smoothed out by means of correlation between adjacent pixels.

[MattJenko]

This topic has already gone down some technical rabbit holes, but can someone explain to me we why we are comparing calibrated frames for noise of the sensor? Isn't that simply comparing the quality of the calibration of each set of images, not the relative noisiness of each sensor, or are we saying that the fact that one sensor has worse 'technical' specs makes no difference in the real world - as that i get :).

[swag72]

If we come away from the technical rabbit holes for a moment.... the reason that I did this comparison was because of the following.

How many times have you heard people say that the Kodak sensor is so noisy when compared to the Sensors? I had the opportunity to compare an equal number of frames from each sensor and thought it would make an interesting comparison. Sadly, when I was changing my camera a few years ago, I listened to the internet rumblings abut the KAF8300 and went for the Sony sensor as I was worried about the noise of the Kodak. I guess I'm showing that it's a mute point at the end of the day and in fact when things are calibrated the Kodak KAF8300 appears to be significantly LESS noisy than the Sony. Hopefully this thread and my comparisons will stop people like me making the wrong choices.

[gnomus]

43 minutes ago, MattJenko said:

This topic has already gone down some technical rabbit holes, but can someone explain to me we why we are comparing calibrated frames for noise of the sensor? Isn't that simply comparing the quality of the calibration of each set of images, not the relative noisiness of each sensor, or are we saying that the fact that one sensor has worse 'technical' specs makes no difference in the real world - as that i get :).

I have not followed much of the technical debate in this thread, largely because it is of no interest to me.  I am interested in the final picture.  Sara's original post showed two images each with 7 hours of total integration time.  Both had been calibrated.  Since calibration is standard, then it seems to me this was a valid test (or 'experiment' if we wish to pretend that we are scientists).  I believe the main motivation behind Sara's test was the frequently cited statement that the new Sony chips are less noisy than the old-fashioned KAF 8300s.  I think she pretty well shot down that myth, certainly as far as real world results are concerned.  

What the raw subs look like seem neither here nor there.  What matters is the final result.

Incidentally, I have a KAF 8300 camera and a Sony chipped camera.  I like them both.  

[EDIT:  Oops.  I see I cross-posted with Sara who essentially made the same point.  Apologies for my unnecessary post.]