SER vs AVI - file size

[Tommohawk]

Short version is I'm struggling to get claimed frame rates with my OmeGON Velox 385 at full res. It seems OK when I restrict ROI, which for me is normal when doing planets, but when doing lunar I like to use entire full res frame.

The only capture software which seems to work with Omegon is Toupsky - and to be fair its really good. Today I updated to a new version and the frame rate seemed much improved - reaching around 130 as claimed. However, it still doesnt download at that rate.. only about 35 fps. There no RAM allocation of buffer as in Firecapture BTW.

Trying a few different things, I discovered that if I used SER format, I got much better rates - the full 130 fps. For some reason the timestamp/FPS isnt recognised by PIPP, but simple maths shows the number of frames is correct for 130 FPS.

However... I notice the SER frame size is rather smaller that the AVI file. Is that correct? Maybe its because its not debayered? Now if that were true, it would be an obvious reason for using SER in preference to AVI - much smaller file sizes. But I've read all I can on this and nobody seem to mention that SER files are smaller than AVIs.

So am I missing something??? Grateful for any help!

[vlaiv]

SER files can be smaller than AVI - depending what format you are using.

AVI is usually using RGB 8bit format and that is 24bit per pixel.

You can roughly calculate the size of AVI like this height x width x number_of subs x 3 byte.

If you record 8bit RAW SER file - it will be x3 smaller than avi as it will record height x width x number_of_subs x 1 byte.

If you record 16bit RAW SER file - it will be 2/3 size of AVI as it will record height x width x number_of_subs x 2 byte.

SER can also record debayered data and then it will be about the same size as AVI unless AVI is compressed in that case it will be smaller.

Here is screen shot from SER specification describing format:

It can record mono, OSC in different bayer matrix format or RGB / BGR debayered data

[Tommohawk]

Hi Vlaiv and thanks for that the maths seems to fit with my situation. 

Today I fired it up again just to recheck before tidying everything away and it all went wonky for some reason and froze... not sure why. Anyhow, I did some further tests and the maths you  outline stacks up very nicely.

With Toupsky I  can select RGB24 or RAW, and separately I can select 8bit or 12bit, and also AVI or SER. All combinations seem to be allowed. Most folk agree that when doing many frames 8 bit is fine, so that would suggest RGB24, SER 8bit... correct?

Previously I've used AVI just because the files open with a wider range of software, and also because with Toupsky for some reason the Timestamp which gives the FPS isnt recognised in PIPP or in Windows metadata. Maybe an issue with Toupsky. But I can live without that because the FPS is easy to calculate.

Also I didn't notice the file size difference with SER and AVI in Firecapture and have just rechecked that - AVI and SER files are same size. I can also see that Firecapture seems to capture debayered - it reports it in PIPP as mono.

So after all that, my question is.... if undebayered SER files are smaller than mono SER and smaller than AVI, why doesn't everyone work with undebayered SER?

Smaller files and presumably faster capture speeds, no?

 

[vlaiv]

9 minutes ago, Tommohawk said:

So after all that, my question is.... if undebayered SER files are smaller than mono SER and smaller than AVI, why doesn't everyone work with undebayered SER?

Smaller files and presumably faster capture speeds, no?

I would advocate use of undebayered SER as format of choice primarily because AS!3 uses bayer drizzle to debayer data when stacking. This keeps resolution corresponding to pixel size. Other debayer methods blur image tiny bit and you want the sharpest image possible to start with when processing after stacking.

Speed difference can be due to storage speed.

Regardless what format is used for storage, data should be downloaded RAW from camera. Only difference there is 8 vs 16 bit. After data is transferred over USB - then it is debayered if you select RGB24 and written into file. If your storage disk can't support data throughput required and there is no memory buffer to store unprocessed subs - your capture speed will drop.

In any case - using RAW SER format is the best option.

As far as 8bit / 16bit goes - I'd need to run some tests on that one to see the difference. I have feeling that it should be different and that 16bit should be better - but how much, and under which circumstances - I can't tell without checking.

Using higher gain will lessen the difference between 8bit and 16bit though.

[Tommohawk]

Ok I need some time to digest that! But if working with undebayered data is best why does fire capture used debayered capture?

[vlaiv]

12 minutes ago, Tommohawk said:

Ok I need some time to digest that! But if working with undebayered data is best why does fire capture used debayered capture?

I guess not all will want to work with AutoStakkert!3 so there is option to get color data right away.

Maybe someone just wants to shoot a movie of what the Moon or Jupiter looks like thru their telescope and post it on youtube for example. They expect video to be already debayered.

[Tommohawk]

I guess so but you'd think a "casual" photographer would just stick to AVI.

There's lots of threads out there about folks struggling to achieve download speeds and struggling with monster file sizes, so I can't see why Torsten has opted for debayered with Firecapture... it seems odd, especially given what you say about AS!3 preferring undebayered.

[vlaiv]

3 minutes ago, Tommohawk said:

I guess so but you'd think a "casual" photographer would just stick to AVI.

There's lots of threads out there about folks struggling to achieve download speeds and struggling with monster file sizes, so I can't see why Torsten has opted for debayered with Firecapture... it seems odd, especially given what you say about AS!3 preferring undebayered.

See this thread:

https://www.cloudynights.com/topic/614178-new-224mc-settings-for-firecapture/

Especially post #7 - by Torsten

[Tommohawk]

OK thats really handy info and makes perfect sense. I've used FC quite a bit with AVI and SER, and never realised that! 

FC wont handle my Omegon camera, so I'm stuck with Toupsky for that - although to be fair Toupsky seems very good. If I return to FC when using the ZWO cameras I'll need to remember that!

Thanks Vlaiv

 

[Tommohawk]

Ummmm.... now I have another odd problem!

Tonight I grabbed some more images using RAW SER. If I shove them into AS!3, and use autodetect for the debayer pattern, I get a green image. If I manually select alternative matrix patters, I still get a coloured image. Processing through AS!3 makes no difference - image is still green. 

I can use white balance adjustments when capturing, but this makes no difference to the captured image - so presumably the WB only affects the viewed image when capturing.

What threw me is that in PIPP, the image appears gray - but PIPP by default converts to greyscale. If I untick that options, the image is green again. 

Is this normal behaviour, and if so how do I get my image back to normal colour? Should I choose grayscale in AS13 under colour options? Would this affect the way AS!3 processes the image? Or should I process it as colour (using "autodetect" and just convert to grayscale in PS?

Most of my planetary imaging over the years has been done in mono and none of this is an issue. The limited amount of OSC imaging I've done I used AVIs with RGB 24, and that seems to work OK.

Interestingly, I've just checked, and when doing AVIs in RGB24, the white balance option definitely does affect the capture - without WB its green, with WB its grey.

GRateful for any further thoughts - its driving me slightly bonkers!

 

 

[vlaiv]

Yes it will be green because of relative responses of each channel.

This is easily fixed in post processing with color balance. For accurate color representation, you need to do full color correction with color correction matrix - but that is probably too much work / too complicated

Registax provides very decent results with auto white balance.

Here is one of my captures with ASI178:

It has that strange green cast after stacking. In Registax I do white balance and it makes nice color:

then a bit of wavelets / resize ...

starting to look nice ...

If you want full color correction thing - it goes something like this:

 

[Tommohawk]

OK great, thanks for that!

I dont think I'd used that feature in Registax before, so thats good to know.

Just to be 100% clear - in the past I've often run files through PIPP first before AS!3, partly to use both quality selection methods and partly because I found PIPP better at rejecting some dud frames. Whilst doing this, I also (unknowingly) had the convert to monochrome option checked. Are you saying that this is inadvisable, and that AS!3 would work better with the colour image? 

 

[ONIKKINEN]

Photoshop has auto color and auto tone features aswell, these have worked on Lunar and Planetary for me so far. The downside is that they introduce clipping, specifically 0.10% of blacks and whites are clipped.

You can run the same tool without the clipping introduced by using it from the Curves menu. Click options and you get the auto color correction options menu pop up. Do Enhance per channel contrast with 0 and 0 for clipping and then do the find dark and light colors with 0 and 0 for clipping. Almost every time the colourbalance turns out nice afterwards.

[vlaiv]

14 minutes ago, Tommohawk said:

Just to be 100% clear - in the past I've often run files through PIPP first before AS!3, partly to use both quality selection methods and partly because I found PIPP better at rejecting some dud frames. Whilst doing this, I also (unknowingly) had the convert to monochrome option checked. Are you saying that this is inadvisable, and that AS!3 would work better with the colour image? 

Not really sure what you are asking, but this is what I would recommend with regards to PIPP and AS!3.

Capture raw data and save as SER

In PIPP do your preprocessing / calibration / frame rejection and also save as SER.

Make sure you have following PIPP options set:

Don't debayer and protect bayer pattern.

This will make sure bayer pattern makes it all the way to AS!3 which will then use Bayer drizzle for debayering (best resolution / detail).

[Tommohawk]

26 minutes ago, ONIKKINEN said:

Photoshop has auto color and auto tone features aswell, these have worked on Lunar and Planetary for me so far. The downside is that they introduce clipping, specifically 0.10% of blacks and whites are clipped.

You can run the same tool without the clipping introduced by using it from the Curves menu. Click options and you get the auto color correction options menu pop up. Do Enhance per channel contrast with 0 and 0 for clipping and then do the find dark and light colors with 0 and 0 for clipping. Almost every time the colourbalance turns out nice afterwards.

 

 

Brilliant - thanks for that. As it happens in this instance I am doing lunar, so relatively easy to just convert to grayscale. But since my last post I realised that I will also require colour when doing Jupiter and MArs later this year. I do occaisonally use autocolour because my colour vision isnt great - but I wasn't aware of the clipping issue or the workaround you mentioned, so that's a very helpful tip.

28 minutes ago, vlaiv said:

Not really sure what you are asking, but this is what I would recommend with regards to PIPP and AS!3.

Capture raw data and save as SER

In PIPP do your preprocessing / calibration / frame rejection and also save as SER.

Make sure you have following PIPP options set:

 

Don't debayer and protect bayer pattern.

This will make sure bayer pattern makes it all the way to AS!3 which will then use Bayer drizzle for debayering (best resolution / detail).

Great, thanks. I thought that is what you meant but wanted to be 100% sure. I'll be sure to tick those options.

The point I was trying to make is that in PIPP under "processing options" there is a convert to monochrome option - this was ticked by default I think and of course it means all the funny colour issues disappear.  But from what you say it looks best to leave undebayered colour for AS!3.

Hopefully I'll get round to posting some images later, though with mist and poor seeing they wont be great. Thanks again!