PixInsight - newbie on trial licence

[Oddsocks]

Sorry for the delay, got called away for a moment, I have some answers for you regarding the odd readings you measured.

I had a rewrite of this post after thinking about the results of your measurements and the default settings you are using in P.I. to handle your .cr2 files and managed to find a sample .cr2 test image I could use to understand why your dark frame measurement is so high and why the saturated frame is grey and so low.

When I open the sample image in P.I. with the settings you are using in the Format Explorer and in particular "Create RAW bayer CFA image" and then measure the pixel value for black I get a value of 0.0314 which is the same that you measured in your dark frame. In the white area of the palette tray I measure 0.0800, though not saturated, it is a low score.

 

Now, if I set the .cr2 prefs in Format Explorer to default by clicking the "Debayer - RGB" button and ticking the box for "No image Flip" and then reopen the same sample image I get a maximum value of 0.0016 for the black box in the colour channels and 0.4515 in the white area of the colour palette which is much closer to expected. If I load a saturated image I am certain it would read 1.000.

 

So it looks as though your odd readings are just because of the way your .cr2 prefs are set at the moment.

Using the option "Create RAW bayer CFA image" compresses the luminance range of the image and is causing the odd appearance and pixel value measurements (I do not use this option at all for the few astro images I take with the Nikon, must have a read up about it's use).

For the purpose of testing the flats, go back to the .cr2 prefs in Format Explorer and in the edit screen push the button "Debayer- RGB" to load the defaults and tick the box "No Image Flip"

In your previous posted images you are using the STF tool coupled, it doesn't affect the underlying pixel values but it does not show the actual flat image to it's best, uncouple the STF by deselecting the icon in the STF tool that looks like a chain, top left...then click the auto button again and you should see a different representation of the flat frame.

Reload your dark frame and measure the noise floor again for each colour channel, this time you should record very low values similar to the Nikon values I posted.

Then reload the saturation image and check it now displays white and that the value is close to 1.000

Finally load up a flat and check as before that the dark corners are at least 0.100 counts above the maximum noise floor reading of the darks

Adjust the exposure time or the VLB settings until you achieve that reading.

Check the bright centre is not close to reading 1.000.

And that should be it.

I had a look in the online Canon manual for the 650, the camera settings I recalled reading about were the "Long Exposure Noise reduction" (Page 125), should definitely be turned off for astro, otherwise you end up subtracting dark current twice when you calibrate the image again with darks and bias, and not too sure about this one but think the "High Speed ISO Noise Reduction" should also be turned off for astro work.

I will watch the thread and see how you get on this time, but now I have found the reason for your odd results before I doubt you will have any further problems.

 

[julian489289]

That's fantastic dude, I will try again later.   I ticked Create RAW bayer CFA image" and pure raw settings as I had read that in one of the tutorials that these needed to be set.  In this test however for my pixel counts these need to be unticked but should I then tick them and set pure raw for my normal astro use or leave it as per this setting?

Yes I have read also that Long Exposure Noise reduction and High Speed ISO Noise Reduction should be off, so I have both turned off for astro work.

Will let you know how I get on later William, thanks again

Julian

[Oddsocks]

9 minutes ago, julian489289 said:

In this test however for my pixel counts it needs to be unticked but should I then tick it for my normal astro use or leave it unticked always?

That question is one of the difficulties with PI because there are so many different ways to approach post processing and different users have developed different approaches to reach the same end result.

I most work with LRGB images and do not use DSLR very much but when I do I use the default "Debayer - RGB" option to set the defaults for a normal colour display and tick the box "No image Flip" so that the stacking batch pre processing script does not start stacking images with differing orientations and then open the images automatically in full colour mode. If I want to work on any of the colour channels individually then I just use the "split RGB" button on the top toolbar to see each of the colour channels as a grey scale where I can then follow up with any channel specific processing.

I can't say I have ever used the "Raw Bayer CFA image option", having mostly learned from Harrys Astroshed video tutorials I can't recall that option being discussed, I will need to look on the PI resources website and see if there is an explanation for when that might be used.

Whatever works well for you and that you are comfortable with is the best mode to begin with, If you haven't watched Harry tutorials they are a great place to start:

http://www.harrysastroshed.com/index.html

[Oddsocks]

On 04/11/2016 at 07:42, julian489289 said:

but should I then tick them and set pure raw for my normal astro use or leave it as per this setting?

Julian.

Your question has been a useful exercise for me as it forced me to read up on the differing modes for handling DSLR RAW images in P.I .

Quite a long post, hope I haven't made it too complicated.

The method you have used until now ("Pure RAW" for RAW Bayer CFA) is the best for DSLR astro images that are going to be subject to further processing in P.I.

When you choose the "Debayer - RGB" option and open the .cr2 file the image is pre-processed to remove the black pedestal and the white level is automatically clipped (unless you select those options in the edit prefs tab to not remove pedestal and clip) plus the camera white balance is used. The image is displayed in P.I. as it would appear as a normal DSLR image in an image processing program like Photoshop or Lightroom etc. Since this is the background I came to P.I. from it appeared natural to me to continue with this mode for handling the few Nikon NEF RAW images I had.

For visualising and measuring the flats and darks this mode is still valid, even though the dark pedestal is removed and the white may be clipped we are still only interested in ensuring the flat exposure results in a black level above that for the noise floor of the dark and a white level that is not saturated, it also means the pixel readout mode covers a wide numerical range and allows very small changes in the flat exposure to be seen as tens of digits change rather than just one or two digits changed. This mode is a very quick way of determining if the flat is good or not.

When you set up the .cr2 prefs by using the "Pure Raw" option in the Edit Prefs tab you no longer remove the black pedestal, clip the whites or use camera white balance and the image mode is "Create RAW Bayer CFA image", "No image Flip" is automatically selected as well. The batch pre-processing script uses this mode automatically, and ignores any prefs you might have set up for the .cr2 files because it is a faster and less memory intensive method of handling the big DSLR files if they are processed as Debayered RGB (three times smaller for RAW Bayer CFA than for DeBayer RGB) importantly it also results in an output image that still includes the black pedestal and unclipped whites and no camera white balance, the "Create RAW Bayer CFA" output file is, as near as possible, the actual real pixel values.

So for handling a single DSLR raw, say looking at one that someone posted or linked to on this site or one of your own that you wanted to quickly appraise you could leave your .cr2  prefs set to "Debayer - RGB" and you will see the image just as it would appear in Photoshop. If you wanted to look at the image critically with the black pedestal included and the whites unclipped then you would need to use the "Pure RAW" option in .cr2 prefs and then see the RAW Bayer CFA image displayed. (see also below the paragraph beginning "If I understood the documentation" for an alternative approach)

In this state the RAW Bayer CFA image is monochrome so you would need to debayer it manually, then split into separate RGB channels, then carry out "Linear FIT" to colour balance, then recombine RGB channels. What you end up with is an image that is linear and uncompressed containing all the raw pixel data including black pedestal and unclipped whites, the image is rather flat, dull and not colour balanced in this state and needs manual adjustment of the STF tool to make the image look like a natural debayered RGB image looks. Automatic STF wont work here because the black pedestal is included and the whites are unclipped.

 

I think you could sum up to say if you wanted to load up a set of lights, darks, flats and bias frames and intended to work on these using the manual registration and calibration tools in P.I. then the mode to set in .cr2 prefs should be "Pure RAW" then you are working with linear data and can choose when to debayer and recombine channels depending what additional processing steps you will be applying. When using the Batch Preprocessing Script this mode is occurring automatically.

If I understood the documentation I read this morning correctly and you wanted to work on a set of lights, darks, flats and bias frames manually but prefer to see them already in the combined RGB mode as colour images then choose "Debayer - RGB" as the default in edit prefs for the .cr2 then tick the boxes "No highlights clipping", "No Black point correction" and "No white balance", the result will be the same as a "Create Raw Bayer CFA image" in terms of linearity but the image has already been interpolated and debayered into a single colour image. It would need splitting into individual RGB channels for Linear FIT so that it can be colour balanced and then recombined so the only advantage of this mode is one less step needed to colour convert than if you used the RAW Bayer CFA option in the first place!

If you want to process a single image with no calibration files then you can choose "Debayer - RGB" to visualise and work on the image as it would appear in Photoshop, or choose "Pure RAW" if you need to examine the image critically and have better control over the post processing tools that you apply.

Now I understand this difference better I will be looking at using the "Pure RAW" mode more frequently from now on for DSLR images and you should stick to this mode too unless you have a particular need for a Photoshop-like view  (quick check of the flat exposures for example).

Thanks for making me read up on this, has been really useful.

William.

(The image posted below shows a comparison of "Debayer - RGB" mode with simple STF applied, left side image, versus on the right, "Pure RAW" with separate Debayer, RGB channel split, Linear FIT, RGB combine and manual STF applied. Both images end up looking similar but the "Pure RAW" mode gives you greater control and possibilities over the post processing than simple "Debayer - RGB" does.

 

 

[julian489289]

Thanks William that's a good read.

I have redone my dark, saturated flat and tried two further flats at 1/20 & 1/10 with the VLB set at full 255.

The results were much closer to your Nikon values;

Dark 1/20 ;  R:0.0005 G:0.0005 B:0.0005

Flat 5 seconds ; R:1.000 G:1.000 B:1.000

Flat 1/20 Corner     R:0.0436 G:0.1011 B:0.0727   just under 1/2 histogram

               Corner     R:0.0435 G:0.1006 B:0.0834

               Corner     R:0.0391 G:0.0934 B:0.0741

               Corner     R:0.0385 G:0.0980 B:0.0728

               Centre     R:0.0473 G:0.1050 B:0.0838

               Centre     R:0.0442 G:0.1091 B:0.0824

Flat 1/10 Corner     R:0.0870 G:0.2089 B:0.1654   2/3 histogram

               Corner     R:0.0772 G:0.2030 B:0.1647

               Corner     R:0.0972 G:0.2164 B:0.1694

               Corner     R:0.0910 G:0.2098 B:0.1740

               Centre     R:0.0996 G:0.2247 B:0.1831

               Centre     R:0.1008 G:0.2338 B:0.1857

Looking at the corners being 0.1 above the noise floor of the dark (0.0005/0.0005/0.0008) then am i correct in thinking that the 1/10 is the best flat values of the two and maybe try 1/5 exposure to increase the levels to get closer to 0.1 above??

Kind Regards

Julian

 

[Oddsocks]

Taking all the measurements into account the 1/10 sec with the 2/3 Histogram should be fine. A few of the red channel readings are a bit low, that is a consequence of the colour temperature of the laptop screen. While working in the linear response zone for the detector if you use 1/5 sec then all the readings will double and that would be too high.

Your flat frames are remarkably "flat". For a large DSLR sensor.  Out of curiosity, which optical system is your camera coupled to when taking the flats, the LX10, the 102ED or something else?

[julian489289]

The 102 triplet  refractor

[Oddsocks]

On 05/11/2016 at 10:37, julian489289 said:

The 102 triplet  refractor

It is a shame ES don't publish a more informative spec sheet for the 102. Looking at your flats result it could have quite a large imaging circle, it would be interesting to see the coverage on a full frame sensor. The shape of the flat is very different to my TS 100 Quad which is almost triangular in shape. 

Hope you can put the flat problem to bed now.

For DSLR images, next time you use the Batch Preprocessing Script, take the output bias master and use the PI Super Bias tool on it.

Run the Batch Preprocessing script again but this time load the new Super Bias master and tick the box to use masters instead of loading the original individual or master bias frames.

The Super Bias master is a smoothed version that helps reduce noise in the calibrated image, it mimics a source bias master that was made from hundreds of bias subs with as few as twenty or thirty bias subs.

 

[julian489289]

Thanks again William, I will use the 1/10s @ 255 setting in VLB to get my flats in future.

I have noted though, with the ES 102 that towards the edges the star shapes do become a little more elongated than the centre ones.

I have not been using the batch pre-processing script as I have been doing them separately per the tutorials that I have seen.  I thought that you got a better result that way?  But, I have been using the super bias tool though.

[Oddsocks]

2 hours ago, julian489289 said:

I have noted though, with the ES 102 that towards the edges the star shapes do become a little more elongated than the centre ones.

I have not been using the batch pre-processing script as I have been doing them separately per the tutorials that I have seen.  I thought that you got a better result that way?  But, I have been using the super bias tool though.

To correct the coma induced elongated edge stars a flattener would be required, but that is the case for any triplet optic, you need a minimum of four lens elements to produce a flat image plane.

I have used both batch preprocessing and carrying out all the calibration steps manually and never have been able to produce a better image at the end of the process. Perhaps thats just down to my laziness, or too many of these....

 

[julian489289]

That's great I will try the batch process method as that maybe much easier for me.  

However, I wish to add t my M33 with further data.  Can I process and stack frames taken on a different nights?  In that, although same ISO I mean the frames will be different in focus, temperature etc etc.  Can I use different ISO also, thinking of dropping to ISO 400 to get better dynamic range and less noise????

If so, can you do that in BP and if so how?

[wimvb]

If you want to combine data from different sessions, your first concern should be framing of the target. It helps if you align the edge of the sensor with the RA axis:

Align on a bright star and do a 30 sec exposure. After 5 seconds have passed, press the RA+ button on the handcontrol (rate = sidereal) and keep it pressed for the rest of the exposure.

This will result in a very distinctive trail on that image. Turn the camera and repeat until the trail is parallell with the edge of the sensor.

Next imaging session, you do the same (of course).

To improve GoTo accuracy, use a barlow and a short fl eyepiece, ideally with crosshair.

I found that these two methods can give me an accuracy of <100 pixels mismatch between imgaing sessions with my AZ-EQ6 mount and a DSLR (14.6 Mpixels). (That's on the conservative side, I've had 50 pixel mismatch.)

It is possible to combine images with different exposure times etc. PixInsight will group images according to settings. Just make sure that you take darks and flats for each session.

If you are uncertain about how to combine images from different sessions, you can always calibrate each session individually and then combine all sessions when you register and integrate the images. This will give you best control over the integration process. In BPP, tick the box "calibrate only". Then use StarAlignment and ImageIntegration separately.

 

Good luck,

[julian489289]

Thanks Wim, but I usually use astro tortilla to align my targets each night but have been also trying the blind solve in APT.

Just was wondering if PI would automatically recognise light & calibration frames from different dates, ISO etc and deal with those accordingly or whether I would need to input these in a different way?

[wimvb]

3 hours ago, julian489289 said:

Thanks Wim, but I usually use astro tortilla to align my targets each night but have been also trying the blind solve in APT.

Just was wondering if PI would automatically recognise light & calibration frames from different dates, ISO etc and deal with those accordingly or whether I would need to input these in a different way?

I guess then that you leave the camera attached to the scope? Otherwise your camera can still be rotated differently between sessions. Scope/Mount alignment doesn't correct this.

PI batch script does not care about a date or time stamp on the images. It (only) differentiates between exposure times (and maybe ISO, haven't tried this). Your best bet is to calibrate the lights from one session with their own calibration frames, and then align all images at once with StarAlignment. I'm not even sure if PI will pair darks and lights if you have two different exposure times. You can always experiment with a limited number of subs (say, 5 per session), to see if it works.