Please help with image Integration & Calibration with OSC in PixInsight

[oymd]

Good afternoon everyone! I hope you are enjoying the weather this fine weekend!

I have been using DSS to stack my DSO images, and then processing the stacked FIT file in PI. I am still very much a newbie, and have received a lot of help on the forums. Also reading through Inside PixInsight book.

So, last week I imaged M63 over 2 nights. All subs were 240 seconds @ -10C. Took Flats & Dark Flats separately over the two nights. The darks were all from my calibration library., 240s @ -10C.

I stacked them all in DSS using Groups, Main Group was Darks, Group one were Ls, DFs & Fs from first night, and Group two were Ls, DFs & Fs from second night.

I had never done imaging over separate nights before, and the resulting FIT file was a DISASTER:

I was told it is because over the two nights the camera had different orientation on the target, and was advised to stack in PixInsight instead.

So, I chose just the images of ONE of the nights, and followed a tutorial on youtube.

 

Which brings me to my 2 questions:

 

A- At what stage do I DEBAYER the Lights subs? I looked up some tutorials on Youtube, and also in the book. It seems in PI, the process is:

 

1-Use ImageIntegration to stack DARKS to create MASTER DARK

2-Use ImageIntegration to stack FLAT DARKS to create MASTER FLAT DARK

3-Use ImageCalibration to stack FLATS with MASTER FLAT DARK to create CALIBRATED FLATS

4-Use ImageIntegration to stack CALIBRATED FLATS to create MASTER FLAT

5-Use ImageCalibration to stack LIGHTS with MASTER DARK + MASTER FLAT to create CALIBRATED LIGHTS

6-Use StarAlignment to stack CALIBRATED LIGHTS to create REGISTERED LIGHTS

7-Use ImageIntegration to stack REGISTERED LIGHTS to create MASTER LIGHT

 

Unfortunately, after going through the full hour tutorial, it turned out to be for a MONO camera. My camera is a OSC 294MC Pro, so the resultant image was GRAY.

At which point in the above 7 steps do I DEBAYER my LIGHT SUBS?

And where does cosmetic correction come in as well?

 

B- If I want to do this process with subs from BOTH nights, how will I go about that in PI? Darks will be common to both nights, but Ls, Fs & DFs will be different?

Many thanks for any help!!

 

 

Edited May 9, 2020 by oymd

[Adreneline]

I would keep it simple to start with and use data from one night and use the BatchPreprocessing or the newer WeightedBatchPreprocessing script. Load your light and calibration frames and make sure the CFA images box is ticked. Generally accept the default settings and go from there - as advocated by Rogelio Bernal Andreo in his new Mastering PixInsight book.

The purists would take the steps you’ve outlined above but the scripts generally work very well and it’s a good way to get going.

Good luck. 

Edited May 9, 2020 by Adreneline
Typo

[oymd]

25 minutes ago, Adreneline said:

I would keep it simple to start with and use data from one night and use the BatchPreprocessing or the newer WeightedBatchPreprocessing script. Load your light and calibration frames and make sure the CFA images box is ticked. Generally accept the default settings and go from there - as advocated by Rogelio Bernal Andrew in his new Mastering PixInsight book.

The purists would take the steps you’ve outlined above but the scripts generally work very well and it’s a good way to get going.

Good luck. 

Thanks for your reply. I will try that later when I get home. 
 

But just to understand the process above, I have already done all those 7 steps. Would be helpful if you can point out when should DEBAYER process be done?

i will avoid combining data from separate nights to avoid complexity, just as you suggested. 
 

many thanks

 

[Adreneline]

As I so rarely use the full process the best thing I can do is refer you to these tutorials which take you methodically through every step of calibrating and stacking your lights, i.e.

When I first started with a dslr and the osc(CCD) these tutorials were a godsend to me; hopefully they will help you.

Hope you won't see this as a cop-out on my part but the above is a more complete and thorough answer than I will give. It's a little dated inasmuch that WeightedBatchPreprocessing has come along since these tutorials were posted.

Adrian

 

 

Edited May 9, 2020 by Adreneline
Clarification

[MarkAR]

You can combine multiple nights imaging into one but because of the large rotation you will have to crop quite a bit by the looks of it.

I tried the "proper method" for one integration and I couldn't find anything on where to put in Cosmetic Correction either. Not sure about Debayer (I'm mono) but it looks like it comes between 4 and 5.

[wimvb]

DeBayering should be done AFTER calibration and BEFORE image registration. As for imaging over several nights. Do you need to take the camera off? In order to get repeatable results, align the sensor with the ra or dec direction.

Take a 30 s exposure. About 5 seconds in, start slewing in ra+ only at 1x sidereal speed. This will give star trails. Examine and rotate your camer "into" the trails. Repeat untill the trails are parallell to the longer edge of the sensor. This only takes a few minutes to do.

[david_taurus83]

^^^ This. Once you've calibrated your lights, then debayer them all. (Optional, but after debayering I load them all into Blink to have a look for bad ones and discard as needed.) Then pick one to align all the rest to, then integrate. I used to use the LVA tutorial and it's similar to the book but I dont bother using the drizzle or normalisation routines as they take soooooo long unless you've got a supercomputer!

[wimvb]

9 hours ago, david_taurus83 said:

but I dont bother using the drizzle or normalisation routines

Correct. Drizzle is almost never needed for amateur images. Only if the stars in your subs look very "blocky", ie are undersampled, can drizzle be of use. And then only if the shape of stars bothers you. In all other instances, drizzle does virtually nothing to improve resolution. But it does increase noise and image size.

 

[oymd]

22 hours ago, Adreneline said:

I would keep it simple to start with and use data from one night and use the BatchPreprocessing or the newer WeightedBatchPreprocessing script. Load your light and calibration frames and make sure the CFA images box is ticked. Generally accept the default settings and go from there - as advocated by Rogelio Bernal Andreo in his new Mastering PixInsight book.

The purists would take the steps you’ve outlined above but the scripts generally work very well and it’s a good way to get going.

Good luck. 

Update,

Yesterday night stayed up to 3am, and went through the Purist approach, with Warren Keller's book open, and managed to do the whole thing!

Took 4 hours though, and obviously just ONE set of files, as still do not know how to combine images from 2 different nights.

Four hours!! DSS takes like 20 minutes??

Anyways, I have just opened BPP, to try it the easy way. I realise there is a NEWER WeightedBPP, but it seems it came out AFTER the InsidePixInsight book was printed, and its not in the book.

So, sticking with BPP for now.

The confusing part is that the book does not mention FLAT DARKS? where do they go? There's only tabs for Bias, Fs, Ds & Lights.

[oymd]

16 hours ago, wimvb said:

DeBayering should be done AFTER calibration and BEFORE image registration. As for imaging over several nights. Do you need to take the camera off? In order to get repeatable results, align the sensor with the ra or dec direction.

Take a 30 s exposure. About 5 seconds in, start slewing in ra+ only at 1x sidereal speed. This will give star trails. Examine and rotate your camer "into" the trails. Repeat untill the trails are parallell to the longer edge of the sensor. This only takes a few minutes to do.

 

15 hours ago, david_taurus83 said:

^^^ This. Once you've calibrated your lights, then debayer them all. (Optional, but after debayering I load them all into Blink to have a look for bad ones and discard as needed.) Then pick one to align all the rest to, then integrate. I used to use the LVA tutorial and it's similar to the book but I dont bother using the drizzle or normalisation routines as they take soooooo long unless you've got a supercomputer!

Thanks, I will try that trick the next clear night I do imaging.

Any thoughts on BPP?

I am about to give it a go, PI book has no mention or Weighted BPP, so sticking with BPP. There is NO TAB for Flat Darks?

[Adreneline]

29 minutes ago, oymd said:

So, sticking with BPP for now.

The differences with WBPP are not that great but best to go with BPP first. A quick search on the web will reveal PI tutorials and help on using WBPP.

Dark-Flats can be used in place of Bias - least that is what I do. Bias frames are not recommended for CMOS cameras but Dark_Flats will do the job as the exposure times are likely to be short.

Combining different imaging sessions with the same optics/camera should not be a problem, the BPP and WBPP can both cope with that - you will just have to accept cropping the final image.

I think it is a case of keeping all this in perspective. DSS is a very capable piece of software and does a very good job of calibrating and integrating images. PI allows for much greater control of the process at every stage along with the likes of cosmetic corrections, etc. I have to confess I now use APP all the time for calibration and integration because it is so very easy to combine different imaging sessions from different nights even with different optics/camera; it also handles mosaics with amazing ease and in my view, with my modest equipment, produces an integrated image that is every bit as good as PI or DSS. Once I've got my calibrated and integrated master light I move straight to PI to post-process the image and then often finish up in PS to do a few minor cosmetics.

There are plenty of packages like DSS, PI and APP on the market (e.g. StarTools, Nebulosity, AstroArt) - it is a case of finding the one that suits you and your method of working best.

Good luck!

Adrian

[oymd]

Thanks Adrian

well, BPP just completed.....after about 45 minutes processing. 
 

I was greeted with a failure message..

failed to calibrate lights. Console suggests there was no master supplied, when the book clearly explained that a master is not necessary, and I can use my original FIT files out of the camera. 
 

I am seriously thinking of throwing in the towel and skipping PI. It looks like an amazing bit of software, but the time and involvement it demands is just too consuming. 
 

I will have a look at APP, as it keep popping up and everyone seems to recommend it. 
 

I’m still on my trial licence in PI, but growing frustrated with it. 

[MarkAR]

Not sure if you've seen this vid.

 

[wimvb]

1 hour ago, oymd said:

I was greeted with a failure message..

failed to calibrate lights. Console suggests there was no master supplied, when the book clearly explained that a master is not necessary, and I can use my original FIT files out of the camera. 

You can skip calibration masters or frames, and PI will only warn you. But you have to deselect the use of a master calibration file in BPP.

Btw, "the Book" was written before the days of WBPP. Therefore: no mention of it. Maybe in a future edition.

[oymd]

2 hours ago, Adreneline said:

The differences with WBPP are not that great but best to go with BPP first. A quick search on the web will reveal PI tutorials and help on using WBPP.

Dark-Flats can be used in place of Bias - least that is what I do. Bias frames are not recommended for CMOS cameras but Dark_Flats will do the job as the exposure times are likely to be short.

Combining different imaging sessions with the same optics/camera should not be a problem, the BPP and WBPP can both cope with that - you will just have to accept cropping the final image.

I think it is a case of keeping all this in perspective. DSS is a very capable piece of software and does a very good job of calibrating and integrating images. PI allows for much greater control of the process at every stage along with the likes of cosmetic corrections, etc. I have to confess I now use APP all the time for calibration and integration because it is so very easy to combine different imaging sessions from different nights even with different optics/camera; it also handles mosaics with amazing ease and in my view, with my modest equipment, produces an integrated image that is every bit as good as PI or DSS. Once I've got my calibrated and integrated master light I move straight to PI to post-process the image and then often finish up in PS to do a few minor cosmetics.

There are plenty of packages like DSS, PI and APP on the market (e.g. StarTools, Nebulosity, AstroArt) - it is a case of finding the one that suits you and your method of working best.

Good luck!

Adrian

Hi Adrian

Ok, I've decided I'm not giving up...

So, I took your suggestion, and will use the Flat Darks in the BIAS tab.

I've just loaded BPP, and here's what I've done. It's processing now, so should find out in an hour or so if this will be successful:

I have 2 sets of Ls, DFs & Fs, and one set of Ds

All Ls & Ds are matched @240s -10C

Each set of lights has corresponding Fs & DFs, which are matched.

In Bias: Loaded DFs from imaging session 1

In Darks: Loaded Ds

In Flats: Loaded Fs from imaging session 1

In Lights: loaded Ls from imaging session 1 (30 subs)

Clicked CFA, and chose RGGB in Bebayer

 

Decided to try my luck, and add the subs from the SECOND IMAGING SESSION using Groups under CUSTOM TAB

 

Chose BIas, and named it in the FILTER option as "OLD": Loaded DFs from imaging session 2

Chose Dark, and named it in the FILTER option as "OLD": Loaded Ds from common pool (THIS WAS REJECTED by PI, as it informed me that the darks were already loaded. Makes sense. So no NEW Darks here)

Chose Flat, and named it in the FILTER option as "OLD": Loaded Fs from imaging session 2

Chose Light, and named it in the FILTER option as "OLD": Loaded Ls from imaging session 2 (23 subs)

 

I noticed that in the FILE COLUMNS for DARKs, FLATS & LIGHTs, I had TWO groups of files, the ones I added initially, and then the group from the other session that i labelled as "OLD" underneath them.

In the BIAS column, both groups of DFs were in ONE group, with no separation. Repeated the BIAS process, and gain, no separation between the dark flats based on name.

 

 

Clicked DIAGNOSTICS and it said all was OK

Hit RUN...

We will see.....

 

 

[wimvb]

4 minutes ago, oymd said:

So, I took your suggestion, and will use the Flat Darks in the BIAS tab.

 

Won't this calibrate your darks with Flat Darks?

PixInsight groups darks according to exposure time (these groups are called "bins" in BPP). It will then use darks that best match lights and flats to calibrate light frames and maybe even flat frames.

[oymd]

52 minutes ago, wimvb said:

You can skip calibration masters or frames, and PI will only warn you. But you have to deselect the use of a master calibration file in BPP.

Btw, "the Book" was written before the days of WBPP. Therefore: no mention of it. Maybe in a future edition.

Yes, I've noticed that WBPP is only in the newer book, the one on the other thread.

Wim, by the way, I did try te MLT tutorial you sent me, and while the MLT part worked, I just could not get the Median transform part to work? You mentioned in your tutorial to try to increase the STRENGTH to 7 or 10 on the first layer, but the console window in PI had no STRENGTH option? Just THRESHOLD & AMOUNT? Just could not get it to work yesterday night.

[oymd]

3 minutes ago, wimvb said:

Won't this calibrate your darks with Flat Darks?

PixInsight groups darks according to exposure time (these groups are called "bins" in BPP). It will then use darks that best match lights and flats to calibrate light frames and maybe even flat frames.

I have no idea Wim

I am just following Adrian's advice.

If you think this is wrong, so where should I add my Dark Flats then? There is no Dark Flats option in BPP?

[oymd]

Ok, its done, and looks like it's worked!!??

BPP produced TWO Master Lights:

One of the 30 Light subs

Second one of the COMBINED 53 Light subs...

 

The one on the left is the COMBINED file of all the 53 subs.

And it looks better than the result I got with DSS, at least there are no visible artifacts?

Would be great if someone have a look at it, and if free, have a go at processing it? It might shed some light whether the BPP process I did was OK?

Fingers crossed...if the file turns out OK, I will be very very pleased and encouraged.....

Many thanks to you all for your help...

 

 

 

light-FILTER_OLD-BINNING_1.xisf

[oymd]

There seems to be much more noise though in the COMBINED file after stretching:

3 hours ago, Adreneline said:

The differences with WBPP are not that great but best to go with BPP first. A quick search on the web will reveal PI tutorials and help on using WBPP.

Dark-Flats can be used in place of Bias - least that is what I do. Bias frames are not recommended for CMOS cameras but Dark_Flats will do the job as the exposure times are likely to be short.

Combining different imaging sessions with the same optics/camera should not be a problem, the BPP and WBPP can both cope with that - you will just have to accept cropping the final image.

I think it is a case of keeping all this in perspective. DSS is a very capable piece of software and does a very good job of calibrating and integrating images. PI allows for much greater control of the process at every stage along with the likes of cosmetic corrections, etc. I have to confess I now use APP all the time for calibration and integration because it is so very easy to combine different imaging sessions from different nights even with different optics/camera; it also handles mosaics with amazing ease and in my view, with my modest equipment, produces an integrated image that is every bit as good as PI or DSS. Once I've got my calibrated and integrated master light I move straight to PI to post-process the image and then often finish up in PS to do a few minor cosmetics.

There are plenty of packages like DSS, PI and APP on the market (e.g. StarTools, Nebulosity, AstroArt) - it is a case of finding the one that suits you and your method of working best.

Good luck!

Adrian

 

2 hours ago, MarkAR said:

Not sure if you've seen this vid.

 

 

1 hour ago, wimvb said:

You can skip calibration masters or frames, and PI will only warn you. But you have to deselect the use of a master calibration file in BPP.

Btw, "the Book" was written before the days of WBPP. Therefore: no mention of it. Maybe in a future edition.

There seems to be much more noise though in the COMBINED Master file.

[wimvb]

45 minutes ago, oymd said:

If you think this is wrong, so where should I add my Dark Flats then? There is no Dark Flats option in BPP?

That's why I use the "manual" workflow:

• dark integration
• flat dark integration
• flat calibration
• light calibration
• light registration
• light integration

All with these settings (will load in workspace 1 in PI)

RGBIntegrationProcess.xpsm

But, you can set the exposure tolerance in BPP to zero and load the flat darks into the darks section. PI should then group them correctly.

[Star101]

27 minutes ago, oymd said:

Ok, its done, and looks like it's worked!!??

BPP produced TWO Master Lights:

One of the 30 Light subs

Second one of the COMBINED 53 Light subs...

 

The one on the left is the COMBINED file of all the 53 subs.

And it looks better than the result I got with DSS, at least there are no visible artifacts?

Would be great if someone have a look at it, and if free, have a go at processing it? It might shed some light whether the BPP process I did was OK?

Fingers crossed...if the file turns out OK, I will be very very pleased and encouraged.....

Many thanks to you all for your help...

 

 

 

light-FILTER_OLD-BINNING_1.xisf 401.52 MB · 2 downloads

that file appears to be debayered wronly on my PI. It comes out RED. 

Did you set the debayer matrix to RGGB? 

[wimvb]

1 hour ago, oymd said:

Yes, I've noticed that WBPP is only in the newer book, the one on the other thread.

Wim, by the way, I did try te MLT tutorial you sent me, and while the MLT part worked, I just could not get the Median transform part to work? You mentioned in your tutorial to try to increase the STRENGTH to 7 or 10 on the first layer, but the console window in PI had no STRENGTH option? Just THRESHOLD & AMOUNT? Just could not get it to work yesterday night.

Strength should be threshold, my bad.

Here's an image I'm currently working on where I used this recipe.

This is an extreme crop, so you can see the individual pixels. Chroma noise reduction was done AFTER colour calibration. As you can see, the colour is much more neutral after noise reduction, but the image looks as noisy as before. This is intensity or lightness noise. There still is a weak greenish large scale structure left after noise reduction. Also note that stars were not affected.

These are the settings I used for this image

I got away with less noise reduction in this image. But since I didn't use all 8 layers, there was still some structure left afterwards.

[oymd]

12 minutes ago, Star101 said:

that file appears to be debayered wronly on my PI. It comes out RED. 

Did you set the debayer matrix to RGGB? 

Yes...

I think once you do ABE it becomes normal in color, or on STF click the CHAIN button to unlink RGB channels

Here's my outcome...

Struggled with the noise..

 

M63 Final Processing.tif

[wimvb]

44 minutes ago, oymd said:

There seems to be much more noise though in the COMBINED Master file.

That may very well be the STF. Apply the same STF settings to both images. A better way to evaluate noise is either the script under Image analysis, or the standard deviation in image statistics.