PIPP - planetary imaging pre-processor software

[cgarry]

PIPP is a simple command-line program that I wrote to help with pre-processing planetary images.

PIPP's functions include:

• Load a sequence of images from either AVI video files or bitmap image files.
  
• Debayer raw frames from colour cameras to produce colour frames.
  
• Check each frame contains a planet that is completely on the image and discard any frames that do not.
  
• Check for and discard overexposed frames.
  
• Centre the planet in the frames.
  
• Offset the centred planet.
  
• Crop each frame around the centred planet.
  
• Estimate the quality of each frame and reorder the processed frames in order of quality.
  
• Keep only the number of best quality frames specified by the user.
  
• Split colour frames into R, G and B frames.
  
• Save processed frames as a sequence of bitmap (.bmp) images ready for stacking.
  

It is still very early days for PIPP and up until now only a few test users have had access to the program. I may come to regret saying this, but I think PIPP is now stable enough for a first formal release.

So, if anyone is interested then I have put together a quick website for PIPP that can be found here:

PIPP Planetary Imaging PreProcessor

Cheers,

Chris

[samtheeagle]

Just going to give it a try now Thanks for the contribution. I'll report back on how it goes.

[cgarry]

Just going to give it a try now Thanks for the contribution. I'll report back on how it goes.

That is great, thanks. Looking forward to any feedback that you have.

Cheers,

Chris

[samtheeagle]

Good news and bad news

It has run all ok, no crashes or anything... But I am getting a lot of the output bmp files as just completely black images. A very high percentage of them, maybe 50%?

Here's the console info from my test run:

C:\Astro\_To Be Sorted>pipp -quality=500 -colour "Jupiter 30_09_2011 23_29_15.avi"

PIPP (v1.0)

Counting input frames:

WARNING: AVI File uses non-recommended codec YUY2 (RGB24 recommended)

Total input frames: 1636

Processing 1636 frames:

100% done

Quality processing:

Renamed 500 images

Summary:

PIPP version: v1.0

Total input frames: 1636

Frames discarded by quality: 1136

Total output frames: 500

Output Frame Type: Colour

Output Directory: Jupiter 30_09_2011 23_29_15/

Runtime: 19 seconds

I'm running Windows 7 Home Premium. Shout if there's any more info you need.

[tonyowen]

I tried your program with some avi files I have.

I (deliberately) used only a few of the options and it worked okay.

As per an earlier response, I got some blank/dark frames - but maybe they're on the original avi.

The -rgb splitter worked okay for the files used by me.

My avi files were reported as using a non-recommended codes I420 - I'm not certain as to how I change the codex that my spc880nc webcam uses under Sharpcap

I tried your program with an avi of the moon. The program followed its routine but yielded no frames!!

Does the program only work with a full disk??

Why will it not work with an avi that has only a portion of the moon's disk??

Will it work if the avi has only a portion of (say) Jupiter's or Saturn's disk?

I tried my moon avi with the option -nctr and again had no frames in the end.

The programs seems to do what is claimed, and I intend to explore it more.

regards

Tony Owen

[cgarry]

Thanks for the testing guys and especially the AVI file samtheeagle.

I have now fixed the blank frame bug and uploaded a new version. This bug only occurred when the image was not cropped, something my testing missed!

As for the warning about using the I420 codec, that is just a friendly warning that can be ignored if there is no alternative. I may remove the warning if it bugs users too much.

I have never thought to use the program with a moon image, there is no reason why it should not work in principle but a little bit of a code rewrite will be needed as the quality estimation code currently takes some information provided by the planet centring logic. I shall get onto that! This will also true for a portion of Jupiter or Saturn's disk being on the frame.

Cheers,

Chris

[samtheeagle]

Version 1.1 is happily processing the troublesome avi now I'll let you know if I hit any more issues.

[cgarry]

V1.2 is now available with support for quality estimation of objects that are not completely within the image. I have tested it on some old moon AVIs I had and it worked for me.

Cheers,

Chris

[tonyowen]

downloaded program v1.3 and latest manual.

The latest version will process an avi of a portion of the moon - thank you.

However, the program seems to reduce the brightness of the sections of the avi selected.

I tried the program with several moon webcam recordings - but all seemed to become darker after processing.

I'm using only the quality option to keep things simple.

Planetary webcam recordings do not suffer from the (apparent) reduction in brightness.

I used the same avi twice with your program and ended up with a (for example) run02 & run02p.

However when running this same avi a third time the program indicated that it had been processed but the resulting file was not displayed, NOR was an earlier file overwritten.

What does the suffix 'p' mean?

and what becomes of successive processing of the same avi?

For instance I might want to run the one avi using many variations of the options and expect to retain all resulting files.

keep up the good work

regards

Tony

[cgarry]

Hi Tony,

There was a bug in the output directory name generation, but that was fixed in v1.2. The bug was that the directory name would start off as "./pipp" and if the if the name PIPP generated was shorter than this ("run02" in your case) then the remaining characters would be left at the end of the name ( giving "run02p" in your case). However, I believe this bug was fixed in v1.2 and I definitely cannot recreate it here with v1.3. I can only guess that you actually ran an old version of PIPP or the "run02p" directory was hanging around from some earlier tests. This should be obvious since PIPP does report its version and the output directory used.

Successive processing of the same AVI should just write over the older files, at least it does on my Win7 machine. If you want to re-run the processing multiple times then you would be best to use the -outdir=directory_name command line option to specify a different directory each time rather than accept the default option.

Actually, I do have in mind the idea that PIPP should not use the same output directory twice when auto-generating directory names, possibly by adding a count to the end of the directory name if it already exists. PIPP could then write a text file to that directory with details of the options used for the processing. I am open to other ideas on how to do this.

Right, onto your moon issue. I don't think PIPP is actually dimming your frames, but I would not be surprised if the quality detection code was favouring the dimmer images for some reason as this code may be working outside of its comfort zone analysing the whole frame rather than just a small part of the frame surrounded by dark sky. Would it be possible to get hold of one of your moon AVIs so that I can see the issue? The only moon AVIs I have are quite short. I am sure that with a little tweaking this is fixable though.

Thanks again for your feedback.

Cheers,

Chris

[cgarry]

V1.4 of PIPP has been uploaded to the website. There are a few minor feature additions but the main reason for the release is to fix a bug found in the debayer algorithm.

Cheers,

Chris

[Gina]

Love the website - very clear, nice colours, pleasant and easy to use Great program usage examples too. This looks a brilliant application and should be extremely useful. I am used to command line control as most of my software starts off this way.

I have nothing to try it with as yet as I'm still setting up my equipment and need some clear skies to test things but we've had so few of those lately! Things are looking so exciting for the future I think this is the most patience testing hobby I have found yet I have seen images of Jupiter on my setup with similar detail to your source files and find it truly amazing how much detail you have extracted from those fuzzy and wobbly video files.

[tonyowen]

Chris, you may still have a bug when your program is re-run without deletion.

I ran an avi twice using different values of -q

The result was frames in the sequence:

002 0002 003 0003 etc

I'm running v1.3 and have manual v0.5.

I've just noticed that v1.4 is available so I'll download that for further uses.

regards

Tony

[cgarry]

Hi Tony,

Yes, v1.3 was a little untidy in that respect. v1.4 should solve this as PIPP will no longer use an existing directory. It also writes a simple log file to let you know what options were used.

Thanks for the comments Gina, it is nice to know I am not the only person who prefers the command line. A GUI version will be along at some point though to keep everyone else happy! Am I correct in saying that you are Linux user? What type do you run out of interest?

Cheers,

Chris

[Gina]

Thanks for the comments Gina, it is nice to know I am not the only person who prefers the command line. A GUI version will be along at some point though to keep everyone else happy! Am I correct in saying that you are Linux user? What type do you run out of interest?

Cheers,

Chris

Not too bothered either way TBH. For my stuff I generally get the low level stuff sorted out first and get things working then later may develop a GUI but often the GUI is a lot mork work than the rest

Yes, I use Linux, Mint flavour Ubuntu took a wrong turn IMO so I went over to Linux Mint which is based on Ubuntu but continued on the original track.

[tonyowen]

Chris

I use a hand guided terrestrial refractor - consequently I have to move the telescope (un-smoothly??) to keep the image on the laptop screen.

I note that using version 4 of pipp for a avi of 6780 frames in the form of:-

pipp -q=4000, -ctr -ms=5 -qmin=2 -qmax=3 -qinc=1

- does not prevent the inclusion of badly formed images of the planet (in this case Mars.)

Attached are three post pipp processed frames that are streaks, dumbells and disfigured 'planetary disks'.

Is there any control value that would limit the disk distortion when processing the quality element?

Also, is there a means of producing the pipp output as an avi file? The reason I ask is that loading many thousands of bmp files into Registax is time consuming.

regards

Tony

0001.bmp

0081.bmp

0218.bmp

[cgarry]

Hi Tony,

Can you try running again with the -debugfind option set and post the same output images? It would be useful to know what PIPP thinks the object looks like.

I am getting reports that v1.4 is also having a few issues detecting objects partially on the frame. This seems to be down to the automatic planet pixel threshold calculation generating a value that is too high. The -minpixel=xxxx option can be used to override this issue.

Unfortunately PIPP does not output AVI files yet, but that is in the pipeline. When I load the BMP files in Registax I do the following:

* Click the 'Select' button on Registax to open the 'Open Files' window.

* Navigate to the folder with all the required BMPs in it.

* Click on one of the BMP files.

* Press CTRL and A keys together to select all the files.

* Click the 'Open' button and all the files are opened.

There is actually no mechanism to reject objects that are a strange shape, but the quality algorithm should demote them as it currently favours the smallest objects.

Cheers,

Chris

[tonyowen]

will do asap

regards

Tony

[tonyowen]

Hi Tony,

Can you try running again with the -debugfind option set and post the same output images? It would be useful to know what PIPP thinks the object looks like.

Chris

As promised:

pipp -q=1500 -ctr -ms=5 =qmin=2 -qmax=3 -qinc=1 -debugfind dubmarsc+d.avi

This was the actual processed instruction that gave the 'erroneous' bmp 0010081 & 0218. (The 6780 frame file mentioned in my post refers to another set of instructions)

Looking at the (new) results 0001 is still distorted, but 0081 & 0218 are not!

I attached 0001 and some other distorted images together with 0800/81 & 82 and 0127/218 & 219; together with the log file.

regards

Tony

Attached Images

0001.bmp

0003.bmp

0006.bmp

0080.bmp

0081.bmp

0082.bmp

0217.bmp

0218.bmp

0219.bmp

pipp_log.txt

[cgarry]

Many thanks for that Tony, that is just the data I needed. The algorithm to calculate a suitable minpixel value for planet detection is not quite there yet if would seem. Your AVI is actually quite a difficult one to handle because of the bright background and small planet. But it is the difficult AVIs I am interested in right now.

For a few reasons I think the planet finding logic needs a rework if it is going to work on a range of images. I know how I want it to work so it is just a matter of time, typing and testing.

Although the automatic detection algorithms should do the job, I think we can go some way to getting it to work with a modified command line. Something like this should help:

pipp -q=1500 -ctr -ms=15 -mp=150 -qmin=2 -qmax=3 -qinc=1 dubmarsc+d.avi

Cheers,

Chris

[tonyowen]

But it is the difficult AVIs I am interested in right now.

I know how I want it to work so it is just a matter of time, typing and testing.

I know exactly how it is - you think of everything and then something comes out of nowhere.

I'll give your suggestion a try and keep you informed

By the way thanks for the short cut to moving a large number of files into Registax.

regards

Tony

[SamAndrew]

downloading it now, i'll keep an eye on this thread and let you know how it goes

[tonyowen]

Something like this should help:

pipp -q=1500 -ctr -ms=15 -mp=150 -qmin=2 -qmax=3 -qinc=1 dubmarsc+d.avi

Chris

I ran your suggested options line on two avis (c+d and f+g)

The poor frames previously posted (dubmarsc+d) have disappeared or been corrected. However some frames (within the 1500) contain distorted images. I've included a couple (0636 & 0641).

I also ran the options using a 6780 frame avi, requesting q=4000. In this case I've some bad frames at the start and some are attached (0000 & 0001).

I've attached both log files with their name changed to identify the avi processed.

For information/interest I've also attached the Registax v5 outcome of the 1500 c+d frames and the 4000 f+g frames. Registax run on auto with no adjustments at all.

regards

Tony

0000.bmp

0001.bmp

0636.bmp

0641.bmp

pipp_logc+d.txt

pipp_logf+g.txt

[tonyowen]

Chris,

I've run one avi six time through pipp v 1.5 each one of which fails to select any images.

Maybe I've got conflicting option settings??

I've attached each of the log files

please advise

regards

Tony

pipp-log01.txt

pipp_log02.txt

pipp_log03.txt

pipp_log04.txt

pipp_log05.txt

pipp_log06.txt

[cgarry]

Hi Tony,

Thanks for that debug info. I think that should be enough for me to track it down.

Cheers,

Chris