An ISO/RAW problem

[SilverAstro]

I recently took some wide field fixed tripod shots of Corona Borealis and Bootes, I was after two variables in CrB.

some were at ISO 1600 and were somewhat "bright" and and some at ISO 400 were darker, of course ! Same duration and Fratio etc.  (sadly I didnt look at the histograms, next time !! )  However, upon processing the RAWs in DCRAW I was surprised to find that the TIFFs produced were the same for 1600 and 400 !

In the composite pic below 4 instances of DeepSkyStacker show - - top left the 1600tiff, top right the 400tiff. And bottom left is the original 1600RAW loaded into the 3rd instance of DSS and bottom right the 400RAW in a 4th. ( please excuse the slight variation of size of chimney, that was just a result of squeezing four DSS windows into one IrfanView window )

So, DSS is doing something (based on the ISO?) with the RAWs before presenting them to the stacking proceedure ?  This could give a false impression of the relevance of ISO to the capture of RAWs ??

[wxsatuser]

Not sure what your getting at here.

[SilverAstro]

Sorry for being a bit obscure ! I made that post as a result of some discussion in the "NoEQ" topic recently about which ISO is best and how to use the histogram.  Here  . I didnt want to mess up that topic too much and it is of more general interest anyway? But no one followed me here !

So my post was saying that when I looked at my RAWs with DCRAW the ISO setting was having no effect , my RAWs are only affected by exposure duration and not by the ISO ( unless I have blundered somewhere!  ) and  I think that @Filroden  Here   had come to a similar conclusion ?  The display in DSS seemes to giving a false impression of the presentation of a RAW file by including the effect of the ISO setting (available in the RAW metadata).

If this is the case with other cameras then the %Histogram proceedure is irrelevant , or have I horribly misunderstood something ?

[Filroden]

As I understand it, and I'm not technical by any stretch of the imagination ISO has an effect, just not the same effect you might imagine based on experience of ISO in film photograph. For film, the higher ISO film had a higher sensitivity. It required fewer photons to activity the chemical process that resulted in signal being captured. To change ISO settings required you to physically change the film. High sensitivity film often came at the price of increased noise, an effect often embraced to give photos that lovely "grainy" effect.

For digital sensors, ISO is slightly different. If you wanted higher sensitivity you needed to change sensor, i.e. camera. So ISO changes something else akin to sensitivity. As I understand it, because sensors are typically designed for daylight photography, the analogue to digital processor that read the sensor would require many multiple photons to be "seen" before they would trigger a "count". This prevented very bright areas exceeding the capacity of the sensors. By raising the ISO, you're reducing how many photons are needed to be "seen" before they are counted (but which also means noise is more likely to be counted as signal). Eventually you will hit unity, where one photon results in a count of one.

But the technical goings on probably don't matter. What matters are that you expose the target long enough to overcome various thresholds - which ultimately gets boiled down to a signal to noise ratio. Your signal has to exceed the background conditions - caused by the moon, light pollution, current in the sensor, etc. Exposure (for a given camera/scope combination) is the only thing that matters. If your signal never exceeds that level you don't capture anything. Once it exceed it, you're then into a quality argument. Longer exposures should give more signal than noise, so the overall signal to noise ratio improves. Stacking shorter exposures works because your signal exceeds the noise and stacking improves the signal more than noise (simulating the effect of longer exposure but never quite equaling it).

So, exposures must be long enough for the signal to exceed noise. Longer exposure is usually better than shorter (the darker your skies and the better your sensor the more this is true). You want to not under or overexpose - as this either misses data (its below the noise or not even registering during the exposure) or losing data because you hit the limit of the sensor (resulting in blooming and loss of all colour information). The simplest way to see if you've done this is to aim to get your peaks away from both edges of the histogram.

There are benefits to changing ISO in DSLRs. It's been shown the read noise reduces with increasing ISO to a point. For my camera I think that you get best results at around 1600. 

I hope someone more knowledgeable than me can chip in!

Edit: I should add that this is based on my experience as an AltAz imager who is exposure limited by both mount any by pollution to about 45-90 seconds. 

[jsmoraes]

Let  see if I understood your questions.

About ISO:
with digital camera ISO means gain of the internal electronic amplifier. A kind of volume control if you think on audio amplifier. A photo with 60 seconds ISO 400 is similar to one with 30 seconds ISO 800. The same for 15 seconds ISO 1600. And son on.

I wrote similar, not equal; because of the no linear curve of the amplifier and many sources of noises.

About debayer RAW:
When you debayer the RAW photo, the software will create a color image. It will use the real RGB pixels from RAW to create an actual (artificial) color information of the area with some pixels RGB to be present on screen of computer or similar.

Thus, if with Canon we have the RAW pixel pattern:
.
.
.
RGRGRG...
GBGBGB...
.
.
.
Real image of sensor from Canon:

With computer screen we have:
R G  R G
  B     B

So, for a visual color pixel area in RAW pattern (4 pixels):
RG
BG

We will have a screen pattern (3 pixels):
R G
  B

And each software can use a different method for debayer task, and can result some difference in the final result.

The debayer task will give you a color image for screen pattern with a curve of histogram. It can be linear or not. If it is not linear it will be a kind of the best automatic adjustment of this curve. A kind of proccessed image.

About DSS (or other stacking software):

Quote

So, DSS is doing something (based on the ISO?) with the RAWs before presenting them to the stacking proceedure ?

 DSS will work with the information of a group of photos. Not only as addition of data. It will do debayer, calibration, alignment, comparison, addition of data by that comparison and with rules and many other things.

And will give you an image with linear histogram and 32 bits. The autosave.tif file.

The question isn't if DSS will do something based on ISO. It will work with the data (information) in the photo. Not only ISO parameter (gain - intensity - volume), but from saturation, color tone, white balance  and others.

note:
Sometimes you can note that the result from DSS has less color information than a single RAW image. It can add many information and reaches values very near to the full possible RGB values for an area, that will appear near to the white color.

I think that it is a false interpretation that 1 million of frames will result better result than 10 or 20 frames. Each case is a case.

[SilverAstro]

Thanks for your thoughts, very interesting, "I hope someone,,, can chip in " Yep that was the general idea cos I'm stumbling around in the dark so to speak ! with nagging thoughts and silly questions

I see what you are saying but I return to my two tiffs produced by DCRAW from the RAWs of my camera, they look the same. Well to a first approximation on my monitor etc&etc caveat, I have not done a statistical analysis on them but they dont appear to be ummm what is it 2stops different 1600vs.400 ?

I know that DCRAW (used by DSS and i think Startools and PI ) is a command line prog relying ( especially in the case of Windows) on someone to faithfully compile an executable, so it isnt the most fav. user interface but is as close as we can get to the RAWs from the cameras without DSS and histograms and things intervening ??    ,,,,

ohhh I'll pause there and see what js has just posted ,,,,,

[SilverAstro]

13 minutes ago, jsmoraes said:

Let  see if I understood your questions.

About ISO:
with digital camera ISO means gain of the internal electronic amplifier. A kind of volume control if you think on audio amplifier. A photo with 60 seconds ISO 400 is similar to one with 30 seconds ISO 800. The same for 15 seconds ISO 1600. And son on.

To prevent misunnerthingies, my two tiffs (top row) were at the same Fratio and exposure time, the only change was the ISO setting.

still reading the rest of your post, thank you, very interesting , , ,

 

[jsmoraes]

Quote

my two tiffs produced by DCRAW from the RAWs of my camera, they look the same

Negative. They are very different.

The left photo, ISO 1600, has more information than the right one, ISO 400. You can see that faint stars are more visible in the 1600 version. The bright stars have greater diameter in the 1600 version. And if you use the histogram you will see that the curves for luminosity, and colors: Red, Blue an Green are in different position and with different width.

You can get the similar result of the photo ISO 1600 if you use 4 times more exposure time with ISO 400.

Sometimes your eyes can not perceive the differences. We procces the images with 16 bits or 32 bits, but we can not see all those informations. The sofware will see, and will do a better adjustment. Mainly on the borders of  bodies: stars, clouds, etc.

After all, you will convert to 8 bits and JPG file to publish in the internet. You must to be an expert to perceive great differences between the original 16/32 bits on your computer and the image on the internet.

note:

I have a stock Canon. I work with high sensor temperature. Around 35 - 40 Celcius degrees. I prefer to work with ISO 400 and 800. Few times with ISO 1600. Never with 3200. Motif: NOISE.

[Filroden]

Changing ISO does not add any new information, it simply multiplies or divides the same information making it appear brights (increased ISO) or darker (decreased ISO). Images of the same exposure all have the same basic signal information. Whether you increase the apparent brightness using an ISO setting in the camera or increase the apparent brightness in software (e.g. by using the exposure slider in Photoshop or Lightroom) you get the same image. Caveat: that would be true in a perfect system.

Try a thought experiment. Assume you are taking a photo of a perfectly even nebula which showed a constant brightness across the entire field of view. Let's assume that your telescope is situated in a perfectly dark setting (e.g deep space) and your camera had no noise. Let's assume in 10 seconds your telescope and camera captured 8 photons in every sensor well. Our camera has a resolution of 5 x 3 (this must be 1980).

Let's assume that for this camera, ISO 100 represented 8 photons by a single count. This photo would look like this:

1 1 1 1 1

1 1 1 1 1

1 1 1 1 1 

Let's assume that for this camera, ISO 200 represented every 4 photons by a single count. This photo would look like this:

2 2 2 2 2

2 2 2 2 2

2 2 2 2 2

That is, this photo looks twice as bright as the first (so increasing ISO from 100 to 200 doubles the apparent brightness). However, both photos contain the same amount of information. When you process either image, you can make them look identical by doing a linear stretch.

Now if you increased the ISO to 800, where every photon is represented by a count then this photo would look like this:

8 8 8 8 8

8 8 8 8 8

8 8 8 8 8

So, the ISO setting makes the photos "look' different as they appear straight out of the camera, but the photos contain exactly the same information - they all represent the 8 photons actually observed in the 10 second exposure.

My brain hurts and this example can be stretched even more than my images! And I haven't even shown the effects of background or noise or internal reading issues relating to the sensor which all make this simple thought experiment much more complex.

[wxsatuser]

1 hour ago, jsmoraes said:

Let  see if I understood your questions.

About ISO:
with digital camera ISO means gain of the internal electronic amplifier. A kind of volume control if you think on audio amplifier. A photo with 60 seconds ISO 400 is similar to one with 30 seconds ISO 800. The same for 15 seconds ISO 1600. And son on.

I wrote similar, not equal; because of the no linear curve of the amplifier and many sources of noises.

This is correct for daytime photography but not AP.

What daylight togs don't realise is they have a lower SNR as they half the exposure and double the ISO.
This is one mistake that we as astrophotographers must not make.
If we double the ISO we must keep the same exposure time, it's total exposure that counts, 60seconds at ISO 400 is 4x 15secs at ISO 1600.

[jsmoraes]

Quote

If we double the ISO we must keep the same exposure time, it's total exposure that counts, 60seconds at ISO 400 is 4x 15secs at ISO 1600.

This is a common and theoretical knowledge about the amount of information.

But talking in the actual and real truth, 4 x 15 sec to do 60 seconds must be stacked. And different stacker software do different work on these task.

Take a single frame with 60 seconds ISO 800 and 30 second with ISO 1600. Analyse the histogram and come back with it.

[Filroden]

8 minutes ago, jsmoraes said:

This is a common and theoretical knowledge about the amount of information.

But talking in the actual and real truth, 4 x 15 sec to do 60 seconds must be stacked. And different stacker software do different work on these task.

Take a single frame with 60 seconds ISO 800 and 30 second with ISO 1600. Analyse the histogram and come back with it.

In your example, the 60 second exposure captures twice the signal as the 30 second exposure but the signal in the 30 second exposure has been multiplied by 2 so they both appear to be the same brightness. In this example, they appear to be the same brightness but the 30 second exposure has only half the real information. Your 30 second exposure has both the signal and the noise doubled. Your 60 second exposure has the signal doubled but has no effect on the noise.

[SilverAstro]

1 hour ago, jsmoraes said:

Negative.

Sometimes your eyes can not perceive the differences.

Yes you are right, not just my eyes but my expectations ! My mistake !! Having seen the jpg output of the camera, (see below, bottom row) and the DSS view of the two, I was expecting to see a much greater difference in the tiffs produced by DCRAW.   However, now that you make me look in more detail at them (see below, top two crops) I see that the background noise is slightly different (pinker noise at 1600 !) but interestingly not a lot of difference in the star A, being on the threshold of visibility in all 4 crops. HIP75919 at m6.45 in CdC adjacent to theta CrB

Which software are you using to look at the histogram of these ?

and I think I need a new camera ,, Gosh lots of interesting posts/reading  and my head hurts as well !

 

 

[Alien 13]

From what I understand the sensor and amplifier are both analogue devices and are subject to level and temperature issues, unity gain which is around 800 ISO for modern Canon cameras is no guarantee of best overall performance and will depend on a lot of other factors such as ambient temp, length of sub and even the type of target.

Alan

[SilverAstro]

Hmmm, that is odd, my picture did not show and I could not edit it back in,, try again :-

[Filroden]

The signal looks the same in both pictures. You can see the same star trails in both. The only difference between them is levels of noise. If you layered both pictures over each other and subtracted them, the stars would completely disappear and you'd have just noise left.

[jsmoraes]

Quote

 In this example, they appear to be the same brightness but the 30 second exposure has only half the real information.

Absolute true. Long time of exposure catch more photons than short time.

 

But ...

Quote

That is, this photo looks twice as bright as the first (so increasing ISO from 100 to 200 doubles the apparent brightness). However, both photos contain the same amount of information. When you process either image, you can make them look identical by doing a linear stretch.

Quote

so increasing ISO from 100 to 200 doubles de apparent brightness

I don't undestand like.

1)Taking a stable source of light, the incidence of amount of photons on sensor depends from time of exposition.
2) the sensor will convert this amount of photons in electrical information.
3) the ISO parameter doens't act in this convertion. This convertion depends from the technical characteristics of the sensor.
4) the ISO parameter will work with the amplifier for those electric information, as gain function. Negative or positive gain. My Canon 1100D has gain equal 1 between ISO 400 and ISO 800. I don't remember the exact theoretical value from datasheet.
5) if the source of light is a nebula, for example, when the red information is greater than blue, with double ISO the red information will be more present than blue. If the curve of amplifier was linear - and normally they aren't linear, they are logarithm - our perception of light (and colors) is logarithm.

So the appearance will be different.

I use this ressource to enhance the color of some photos. I do many shots with ISO 800 and few shots with 1600. Stack them and mix color channel in Photoshop with mask ressource to enhance some areas.

Quote

When you process either image, you can make them look identical...

irrelevant, since we can do many adjustment and enhancement... to better or to worse.

[wxsatuser]

7 minutes ago, jsmoraes said:

This is a common and theoretical knowledge about the amount of information.

But talking in the actual and real truth, 4 x 15 sec to do 60 seconds must be stacked. And different stacker software do different work on these task.

Take a single frame with 60 seconds ISO 800 and 30 second with ISO 1600. Analyse the histogram and come back with it.

Thats the general idea.

[SilverAstro]

2 minutes ago, Filroden said:

layered both pictures over each other and subtracted them, the stars would completely disappear and you'd have just noise left.

That is a good idea, I'll try that in a min.     I am getting very confused now , so which ISO should I use in future ? ! (but laters the domestic staff are yelling for their evening meal ) , , ,

[jsmoraes]

Quote

so which ISO should I use in future ?

Non sense question.

Silver, it will depend of skyglow brightness. the quality of guiding, the brightness of target (if you want trapezium from Orion nebula you will need low ISO. If you want the clouds of Orion nebula ... you will need more high ISO),  temperature, time of exposition ....

Your software that do capture may have histogram. If it hasn't, use any graphic software that has it (Gimp for example). To adjust the best ISO and time of exposition see the curve on histogram graphic. To check the quality of signal (signal/noise) see the image on the graphic software with much zoom to see the noise on the background.

[Alien 13]

59 minutes ago, SilverAstro said:

That is a good idea, I'll try that in a min.     I am getting very confused now , so which ISO should I use in future ? ! (but laters the domestic staff are yelling for their evening meal ) , , ,

For me I am limited to around 2 minutes of tracking ability so I work to match the sky conditions/sub length and ISO for around 40-50% on the camera image histogram this is almost always between an ISO of 400-1600.

Alan

[SilverAstro]

2 hours ago, jsmoraes said:

Your software that do capture may have histogram. If it hasn't, use any graphic software that has it (Gimp for example).

Yes I know, but that is not what I asked ! I asked what software you were using, because I am not seeing (in GIMP at the moment but maybe I should look in another) the result that you got from my post   (and remember I posted a screen grab in IrfanView originally, which is not the most scientific way of going about things because at the time I did not expect to get into this level of discussion !!! )   so I thought it might be a good idea to use the same software uc.

We are still all at Xpurposes, has anyone else examined their RAWs using DCRAW; and having done so are convinced that their ISO setting is varying the RAW out of their camera as reported by Dave's DCRAW *? [  EDIT later:    if so this is mine : >DCRAW -v -T -6 *.RAW   meaning -verbose write -Tiff instead of PPM write -16bit instead of 8bit ,,, anything wrong there ? is there a force ISO that I have missed ??   ]

 In other words before any in-camera software has processed and displayed any histo info and/or before any *3rd party software has intervened ? iyswim ??

[dph1nm]

To quote: "By default, dcraw writes PGM/PPM/PAM with 8-bit samples, a BT.709 gamma curve, a histogram-based white level". So I suspect your dcraw command is scaling the tiffs to have the same levels and applying a non-linear transform. So I think you need "-4" not "-6" as the option. "-4" gives a linear transform instead of a gamma curve and doesn't vary the white level.

NigelM

 

[Stub Mandrel]

Most canon cameras use 12-bits, rearranged to fit within 16 for output.

The idea of ISO is to make sure the whole range of the image fits in those 12 bits and covers as wide a range as possible.

Too high an ISO and your brighetest image pixels will need more than 12 bits and be 'bleached out'

Too low and ISO and the darkest bits of the image will be clipped to total black.

 

In astrophotography things are a bit off as we have tiny bright dots (stars) against an (almost) dark background with faint detail (DSOs) on top.

 

If the ISO is  too low then the dynamic range at the bottom of the scale is limited and the DSOs tend to disappear into the background noise.

Higher ISO increases the resolution at this bottom end but also amplifies the noise.

Brightening pictures taken at lower ISO ratings will increase the noise but can't stretch faint DSO data that isn't there.

Stacking helps, because increases dynamic range while 'averaging out' the noise.

 

So for ordinary images you want a low ISO to minimise noise. For astro images yopu want as much data as possible as you can 'stack out' the noise, so higher ISOs help.

[jsmoraes]

To capture, now I use BackyardEos. I already used AstrojanTools . Both have histogram and powerfull zoom.

To beter  analyse the quality of  image I use Canon ZoomBrowser or Photoshop CS3. As PS-CS3 doesn'  open CR2 files from Canon 1100D, I use DNG converter from Adobe. CS3 opens DNGs files.