Tartan in mono images!

[philhilo]

Evening all,

I am calling on the great combined experience in Stargazers Lounge to try and stop me spending yet another evening (tomorrow) creating a tribute to Scotland, namely Tartan images! Someone must have had this issue as I am not running an exotic set up. I have tried to detail everything and would be happy to 

I got a ZWO ASI1600MM Pro in the autumn after using a DSLR for a couple of years and have since been plagued with a tartan pattern creeping into most images. I have researched this at length (weeks) and have seen various suggested causes, but this seems to be a problem normally associated with debayering in colour imaging so what is it doing in my images? It does not appear to be a display issue as it is properly baked into the image whatever screen it is displayed on (I am used to seeing a chessboard pattern when opening the files in Fits Liberator, which instantly vanishes when I start to enlarge)

I can usually find it in lights and it is in my flats. It doesn't appear to be frost, many were taken in warm conditions. . Is it to do with my running at gain 139 and offset 20? The ASI1600MM is acting as a hub for the ASI1200MM mini guide camera.

• 200PDS f4.5 Newt
• ZWO ASI 1600MM Pro imaging
• ZWO ASI 1200MM mini guide feeds into 1600 hub
• Dithered
• Controlled by Astrophotography Tool (APT) on laptop
• Stacked in DSS using Startools recommended settings
• Processed in Startools putting the data in as linear unstretched or using the compose module for multi channel images
• Gain 139 Offset 20
• T = -30c or -20c
• Full images calibrated with 50 flats, 50 dark flats, 50 darks

Images attached are:

Fully processed Cone in Hubble palette - very tartan 11hrs 300s H, 3hrs 300s O 3hrs 300s S

Part processed Cone in Ha, no flats - plenty of tartan bottom right

M81 red only stack (also epic gradient as well, LP I think....but that's another story).

Master flat, Gain 139 offset 20 stretched in Startools, lots of tartan in the midtones.

 

What I have established:

Pattern seen in:

• In Lights, Less prominent in galaxy BB data - small bright objects, and even bright nebula like the Pacman, but more prominent in faint mid tone images like Cone nebula.
• In Flats, so no guiding so unlikely interference from guide cam.
• Seen in Startools on 1st stretch so not coming from some later process
• Flats at 20k adu using electro luminscent panel so all pixels well illuminated - minimum values of 
• Lights at different ambient temps, some +10c as well as during the cold snap so unlikely freezing in optics
• Its all mono so not sure how debayering would get involved?
• Pattern in stacks yes, single frames - not sure, maybe, but its not obvious. Certainly some chequerboard pattern at a pixel level in some single images.

So is it something to do with:

• Offset too low?
• Driver issue?
• DSS setting?
• Camera issue?
• Startools issue?

Cone short Ha 21 stack.tiff Ha 21.02 NO FLATS part processed.tiff m81 red tartan cicle.tiff MasterFlat_Gain139 Off 20 tartan.tiff

[geordie85]

Are you sure you're not trying to debayer your images? Even though they are mono, if you have debayer selected you'll get a tartan like image.

I can't see your images as I'm on my phone. Maybe you could upload them as jpegs?

Edited February 21, 2021 by geordie85

[philhilo]

1 minute ago, geordie85 said:

Are you sure you're not trying to debayer your images? Even though they are mono, if you have debayer selected you'll get a tartan like image

Cheers Geordie,

Where is that likely going to be, I don't recollect ever having to specify bayered or debayered in any software - Drivers/APT/DSS/Startools? 

No option in Startools, can't see anything in DSS.

Phil

[vlaiv]

Poor polar alignment combined with bilinear interpolation in DSS.

First and last images are rotated by few degrees - you can check this in DSS as it solves for alignment - it displays rotation of each frame.

When frames need very small angle of rotation and in combination with bilinear interpolation - that effect happens.

What settings is StarTools recommending for DSS? Maybe we can tweak something to get better results, but in principle - try to avoid that slight field rotation over the course of the night.

Btw, I'm not seeing the same effect in your flats.

[symmetal]

 

1 hour ago, vlaiv said:

Btw, I'm not seeing the same effect in your flats.

The flats do show the same sort of patterning vlaiv.

I would suspect a power supply issue. What is the source for the 5V USB supply for the 1600 as maybe it's struggling powering the guide camera as well, along with excessive interference noise on the 5V. Are you using a USB2 or USB3 cable to connect the 1600 to your computer. Try taking a flat without the guide camera connected to see if it's any different. Also try a different cable and/or computer to power the camera. Inserting a powered USB hub to supply the camera will also see if it's a power issue. I don't think the cooler 12V can power the 1600 camera hub though I'm not certain.

Alan

Edited February 22, 2021 by symmetal

[philhilo]

8 hours ago, vlaiv said:

Poor polar alignment combined with bilinear interpolation in DSS.

First and last images are rotated by few degrees - you can check this in DSS as it solves for alignment - it displays rotation of each frame.

When frames need very small angle of rotation and in combination with bilinear interpolation - that effect happens.

What settings is StarTools recommending for DSS? Maybe we can tweak something to get better results, but in principle - try to avoid that slight field rotation over the course of the night.

Btw, I'm not seeing the same effect in your flats.

Hi Vlaiv,

These are the DSS settings Startools recommends:

• Choose No White Balance Processing in the RAW/FITS dialog
• Choose Bilinear Interpolation for the Bayer Matrix Transformation algorithm
• Save your final stack as 32-bit/channel integer FITS files, with adjustments not applied.
• Stack with Intersection mode - this reduces (but may not completely eliminate) stacking artifacts
• Do not choose Drizzling, unless you are 100% sure your that; your dataset is undersampled, you have shot many frames, and you dithered at the sub-pixel level between every frame
• Turn off any sort of Background Calibration
• Some users have reported that they need to check the 'Set black point to 0' checkbox in the 'RAW/FITS Digital Development Process Settings' dialog to get any workable image.
• Choose Kappa Sigma rejection if you have more than ~20 frames, use Median if you have fewer.
• Ensure hot pixel removal is not selected on the Cosmetics tab.

That's what I am using. 

The Cone images utilised different data sets from multiple sessions, even an offcentre set (ran out set rig off for target - forgot I had slightly reframed the object the previous session). 

Where I have multiple sessions of the same subs, but utilising the same calibration frames, I have always processed them in DSS as one group, even more so now I have a dark library, could this be an issue?

OK, had another look at that flat and that could be the screen effect - it radically changes depending on the scale (I have started to see tartan in everything), At full scale it looks like pimples on a golf ball, about 3 pale squares across the dust bunny, zoom in a bit and it becomes a real fierce diagonal grid about 9 squares across the dust bunny.

I had a look at the 'M81 red stack', some 223 30s subs over 3 hours in one evening. I can see the angle creeps steadily down by about 0.1 degree over the 1st couple of hours, then has a lurch of another 0.1 over a few frames just before the meridian, then creeps slowly about 0.01 degrees every 5 minutes for the last 30 minutes. Does that tally with the sort of rotation you were thinking of?

[vlaiv]

7 hours ago, philhilo said:

These are the DSS settings Startools recommends:

You don't need half of those settings. You are now using mono camera and it's using fits files. There is no debayering, white balancing and all that stuff any more.

Just make sure you have this checkbox unchecked like so:

Other things in this dialog box have no relevance (raw tab is for DSLR files anyway).

If you are going to use Kappa Sigma rejection - you need to do background calibration. Especially if you combine data from multiple nights where you can have substantially different conditions like transparency and LP levels.

7 hours ago, philhilo said:

Where I have multiple sessions of the same subs, but utilising the same calibration frames, I have always processed them in DSS as one group, even more so now I have a dark library, could this be an issue?

That is good approach - just make sure you have proper flats. You can reuse flats between sessions only if you have permanent setup, but if you disassemble things between sessions - take flats for each session.

Darks are ok as long as you keep parameters the same - temperature, gain/offset and exposure length (or have master dark for each combination you'll be using).

7 hours ago, philhilo said:

OK, had another look at that flat and that could be the screen effect - it radically changes depending on the scale (I have started to see tartan in everything), At full scale it looks like pimples on a golf ball, about 3 pale squares across the dust bunny, zoom in a bit and it becomes a real fierce diagonal grid about 9 squares across the dust bunny.

That is what I saw as well - artifact that is more related to nearest neighbor interpolation depending on zoom level than actual pattern in the image.

There is one thing that confuses me about that master flat. It is color image yet all there channels are the same. Why is that?

No need to save color image for mono camera flat, and if you wanted to send all three channels as a single file - why are all channels the same?

Even if there is a bit of pattern in flat because of pixel sensitivity distribution (that is factory process defect - I had checker board pattern on my ASI1600 in Ha wavelength but it calibrates out fine) it should calibrate out fine.

Here is master flat how I see it. Not much of pattern, is there?

8 hours ago, philhilo said:

I had a look at the 'M81 red stack', some 223 30s subs over 3 hours in one evening. I can see the angle creeps steadily down by about 0.1 degree over the 1st couple of hours, then has a lurch of another 0.1 over a few frames just before the meridian, then creeps slowly about 0.01 degrees every 5 minutes for the last 30 minutes. Does that tally with the sort of rotation you were thinking of?

When I measure angle of pattern in one M81 image - it is about 2.5 degrees:

So there is difference of few degrees between some subs. Maybe pre / post meridian?

I'm going to simulate this effect for you now - so you can see it that it happens under some circumstances:

I just took gaussian noise and rotated it by 2.5 degrees using linear interpolation. Some of your images need to be rotated to be stacked and noise in them creates this pattern.

What you could try:

1. How do you calibrate?

2. You could see if there is distinct difference in orientation between subs before and after meridian swap, if so stack only one group and see if there is effect

3. Use different stacking software that has advanced interpolation methods that don't produce this effect

[philhilo]

12 hours ago, symmetal said:

 

The flats do show the same sort of patterning vlaiv.

I would suspect a power supply issue. What is the source for the 5V USB supply for the 1600 as maybe it's struggling powering the guide camera as well, along with excessive interference noise on the 5V. Are you using a USB2 or USB3 cable to connect the 1600 to your computer. Try taking a flat without the guide camera connected to see if it's any different. Also try a different cable and/or computer to power the camera. Inserting a powered USB hub to supply the camera will also see if it's a power issue. I don't think the cooler 12V can power the 1600 camera hub though I'm not certain.

Alan

Thanks Alan,

I am going to take both yours and Vlaiv's approachs in the hope that one works (as in a bortle 8 area life is hard enough without extra non signal - especially after spending £££ on upgrading everything!) . 

As for that flat, not sure I believe my own eyes any more, I think I can see pattern, but it does do strange things when enlarged, suggesting a screen effect - not sure what to think! 

The cooler comes straight off the dew heater hub, but the camera is separate and comes through the USB3 cable from the laptop into the 1600, then the guide camera comes out via the 1600 hub.

I will run some flats this afternoon to see what effect unplugging the guide camera has, and as we have some clear skies tonight will run some sequences with the guide running through the 1600, and some running it through a separate cable, alas I don't have a spare USB3 cable.

Also will be working through Vlaivs suggestions (and cooking dinner and keeping 3 yo under control lol)

Will update later.

Thanks again.

Phil

[philhilo]

Hi Vlaiv,

1st up thanks for your help on this, really appreciate your time and effort on this - I don't often ask for help but this one has me stumped.

 

Yes, that is clear - that was my 1st suspect.

Interesting about the background calibration, it had occurred to me that different sessions were going to suffer from quite different seeing (even more so in high lp). Will be playing with that (not the 1st time that processing documentation isn't quite up to speed, but they aren't rules).

I do have a permanent set up, and tend to keep the camera set in the same position for extended periods whilst working on a particular target - usually a nebula for which I need a particular angle for framing, and a galaxy for later in the night when orientation is less important (no rotator....yet).

On 22/02/2021 at 10:37, vlaiv said:

Here is master flat how I see it. Not much of pattern, is there?

Not sure, I think I can still see a pattern, but as I said to Alan I am going to try both your approach's as I want a solution, life is hard enough with Bortle 8 lp (M81 circle) without tartan to add to the non signal!

 

 

On 22/02/2021 at 10:37, vlaiv said:

I'm going to simulate this effect for you now - so you can see it that it happens under some circumstances:

WOW! That is very dramatic and very much in line with my issue.

Interestingly I have just done a quick dirty 1 hr on the rosette, 12 images with very little rotation and there is no tartan effect,

 

 

 

 

 

[philhilo]

Evening all, apols for the delay (life, 3 year old etc), rude of me not to put something up, even if only a 'working on it', but there was a lot to work on,, however I have an answer, well at least a solution to the biggest part of the problem

1st up, thanks to Geordie for your help, very much appreciated in times of arghhhh!  An extra big thanks to you Vlaiv for your detailed response (as ever), it really got me thinking

I still cant quite figure if there is a tartan in the flats, it is there, but definitely does strange things once you enlarge it.

I did try flats with and without the guide camera plugged in, but no difference. Alas I don't have my number two laptop set up for imaging (well sort of but still have to resolve a few settings)

I thought about a new USB3 cable but held off on the spending money option till later

To address Vlaiv's suggestions:

How do I calibrate, 50 of each - 

• Flats for each filter and any rotational change using APT flats aid at whatever recommended % of ADU using a flat panel,
• Dark flats at same settings as flats
• Darks at same settings as lights - done in extreme dark checking that the histogram is close to zero, checking for any hints of light leaks

I do seem to have an issue with the rig maintaining polar alignment - I use Sharpcap and despite getting it to 'excellent' it still clearly drifts, more so after a flip. Having a mount loaded to the max probably isn't helping, and  I think there is a question around flexure in the tube of the 200mm Newt that could be addressed with a beefier dovetail, and another added to the top (even heavier for the mount, lol). I might also need to have a look at the level too. I did try stacking separate before and after sets, however it still had the tartan effect, but there was rotation in each set.

Using a different stacker - I was imaging the rosette, 6 hours of Ha looks great, no problems, 30 mins of S was good (no rotation), 1hr of S tartan starting to appear, 2 hours of S and any fainter areas looked like a Scotsman's kilt, brutal tartan! 

 i downloaded the free trial of APP, stacked the 3 hours of S, processed the result in Startools and NO TRACE OF TARTAN. So it appears that DSS doesn't handle the slight field rotation exactly as Vlaiv suggested. What surprises me is that more people haven't had this issue as I have flagged it up in other places and got no answers to a very glaring effect, no subtle slightly bigger or smaller, shades of colour, just glaring and completely show stopping tartan in a set up and processes that felt very standard. My dilemma now is do I fork out the annual fee (£60) for APP or buy it (£200), at which point with the cost of Startools I am getting close to the cost of PI - however there is the headspace issue. My quiet time is very limited until next winter - that might be the time to get PI, even though I loath the idea of paying top dollar for something with such a badly designed UI -  but that's another story.

Attached is an APP stacked and then processed in Startools version of the Ha only data I originally posted - no tartan (although there are faint linear features at 90' in the base of the cone - which it turns out are real).

Hopefully won't be back with this problem, and thanks again for all your help, or anyone that even looked and couldn't.

Cheers, 

Phil

(I might well post a final image up)

Full_Cone_Ha_21.01-Luminance-session_1 180m stack APP process ST.tiff

[bottletopburly]

Have a try of Astap for stacking which is free and many use for platesolving , I have used for stacking with dslr and it’s simple to set up for that I’m sure @han59 can give advice for setting up parameters for using with mono set up .

[philhilo]

7 hours ago, bottletopburly said:

Have a try of Astap for stacking which is free and many use for platesolving , I have used for stacking with dslr and it’s simple to set up for that I’m sure @han59 can give advice for setting up parameters for using with mono set up .

Hi, yes I stacked in ASTAP as well (I had forgotten it stacked, it normally just says hello when plate solving as you say). On a very quick, 'throw it in, hit go, stack' I  think the result was not as good as APP but it needs another go, especially as it is free (although so is DSS but I seem to be coming to the end of the road with that). One potential bonus with APP is folks use it clear heavy gradients before putting the image into Startools, as ST does seem to struggle - I can vouch for that with Jag Land Rover, B'ham airport, Solihull and Birmingham around me, I have gradients! This arrangement is bizarre though, it is like buying a Mercedes sports car (APP), taking the wheels off and putting them on a Mini (ST) for trackday racing, or spending slightly more and buying a very hard to drive Porsche (PI). Very odd. I guess it is a more extreme version of using Sharpcap for polar aligning but not using its camera control +++ capabilities (which is exactly what I do as I use APT).

I notice you are a fellow 200PDS and IDAS LP D2 owner not far away in E Mids - I wanted to check out the E Mids astronomy club before Covid arrived. Would be good to compare notes on getting the most out of the 200PDS.

Cheers,

Phil

[bottletopburly]

I’m sure Ivo would be interested in tuning st wipe if struggling with heavy gradient if app seems better on heavy gradients  @jager945, I have / had been struggling with flats on the 200pds using Nikon D5300 but hopefully I’m solving that due to Nikon using lossy compression it leaves heavy ringing , luckily thread on CN user marksmeddley seems to solved that writing a program to covert files to Adobe dng then run through his program so fingers crossed I’m hoping flats issue is solved ,my 200pds has autofocus fitted , flocked tube  bobs knobs , wide dovetail , primary fan and recently got a concentre to collimate , and a slip ring above top ota ring , D2 filter seems to work but obviously filters the yellow channel  which is currently on my frac t-ring I need to mount back in the 2” holder and screw back on my CC for use on the 200pds .

[philhilo]

30 minutes ago, bottletopburly said:

I’m sure Ivo would be interested in tuning st wipe if struggling with heavy gradient if app seems better on heavy gradients  @jager945, I have / had been struggling with flats on the 200pds using Nikon D5300 but hopefully I’m solving that due to Nikon using lossy compression it leaves heavy ringing , luckily thread on CN user marksmeddley seems to solved that writing a program to covert files to Adobe dng then run through his program so fingers crossed I’m hoping flats issue is solved ,my 200pds has autofocus fitted , flocked tube  bobs knobs , wide dovetail , primary fan and recently got a concentre to collimate , and a slip ring above top ota ring , D2 filter seems to work but obviously filters the yellow channel  which is currently on my frac t-ring I need to mount back in the 2” holder and screw back on my CC for use on the 200pds .

I have a lot of faith in ST and Ivo - I think the interface is excellent and it produces great results from pretty well default settings. But show it an offcentre circular heavy gradient and wipe doesn't know what to do. I sort of hack away at it as I go through the modules and usually end up with various 'clouds' around my galaxies (only an issue in BB) or I loose a lot of detail. 

Fan on the primary - have you managed to baffle it so that light doesn't get in around the mirror cell? I looked at a fan but having made a point of light proofing the rear end I didn't want that problem back! I have added an aperture reducer/mask to the primary to get rid of the mirror clip issue, likewise I put on a Moonlite focuser to hold the mono+EFW tight and remove the focuser tube obstruction. Now I am looking at an EAF, not often the temp variation in the UK causes too much of an issue, but I would like to be able to have it refocus between filters as I often have it set up to do a nebula for the 1st half of the night, then swap to a galaxy that doesn't require framing in the same way - in the middle will do - for the second half. I find the D2 works well with the DSLR, but pretty well kills the red channel, currently looking for a solution for the mono cam, no clip in for that!

Slip ring above the top ota ring? Is that to hold the tube shape around the aperture due to the weight of the focuser etc + guide or spotting scope? I have lots of offcentre vignetting which is a pain but down to the focuser tube diameter I believe. A dew solution for the secondary would be good, there seem to be plenty of DIY solutions, but none seem to be totally convincing and there are no commercial ones which is surprising for such a common OTA.

[bottletopburly]

I had thought about putting a black shower cap over back end and cutting hole out centre for fan , draw tube I need to look at as think that’s affecting  diffraction spike or primary clips are one or the other , autofocus is a godsend I use deepskydad units which have the ability to be used via hand controller too if you buy the hand controller and temp sensor which are extra if required, also works via WiFi too though I just use via APT ,slip ring is only so you can adjust ota with out slipping , never really suffer with dew on secondary since having a fan blowing .

[philhilo]

Both clips and drawtube are messing with the stars. Clips is a piece of cardboard painted black circle cut out, glued onto clips -  £1 for the cereal box for the cardboard. I have seen folks saw the end off the drawtube (scary), or replace focuser, £300, expensive.

[jager945]

12 hours ago, philhilo said:

I have a lot of faith in ST and Ivo - I think the interface is excellent and it produces great results from pretty well default settings. But show it an offcentre circular heavy gradient and wipe doesn't know what to do. I sort of hack away at it as I go through the modules and usually end up with various 'clouds' around my galaxies (only an issue in BB) or I loose a lot of detail. 

I have lots of offcentre vignetting which is a pain but down to the focuser tube diameter I believe. A dew solution for the secondary would be good, there seem to be plenty of DIY solutions, but none seem to be totally convincing and there are no commercial ones which is surprising for such a common OTA.

A "circular gradient" sounds like a flats issue; vignetting and unwanted light gradients are rather different things with different origins (uneven lighting vs unwanted added light) and have different algorithmic solutions (subtraction vs division).

Basic sample-setting algorithms for gradient modelling and removal as found in APP/PI/Siril are very crude tools that, while getting you results quickly, make it very easy to destroy faint detail (e.g. think faint IFN around M81/M82 etc.).

In essence, they ask you to guess what is "background" in various places. Unfortunately it is impossible to tell with 100% certainty what is background, as you can't see through the muck that is still there - worse, sometimes there is no true background. Conversely, Wipe asks you - if even necessary - to tell it what is definitely not background. For the rest, it relies on undulation frequency of the gradient (slow undulating), which is almost always easily distinguishable (by the algorithm) from actual detail or faint nebulosity (fast undulating). I've likened these different approaches to performing archaeology with a shovel vs doing it with a brush. One causes collateral damage, the other doesn't.

For your immediate vignetting problem, ST 1.7 has a dedicate vignetting stage now that might help somewhat, depending on how badly it is off-center. However, you should take flats as soon as practically possible. They are really not optional in AP, and attempting to just "live with vignetting" will severely hamper what you can achieve with your gear; nor algorithm, nor human can be 100% certain that what is in your image is true celestial detail otherwise.

That said, if you think this is something ST should be able to solve, please feel free to share a dataset with me - I'm always looking for ways to improve ST!

Clear skies!

[philhilo]

On 03/03/2021 at 05:30, jager945 said:

A "circular gradient" sounds like a flats issue; vignetting and unwanted light gradients are rather different things with different origins (uneven lighting vs unwanted added light) and have different algorithmic solutions (subtraction vs division).

Basic sample-setting algorithms for gradient modelling and removal as found in APP/PI/Siril are very crude tools that, while getting you results quickly, make it very easy to destroy faint detail (e.g. think faint IFN around M81/M82 etc.).

In essence, they ask you to guess what is "background" in various places. Unfortunately it is impossible to tell with 100% certainty what is background, as you can't see through the muck that is still there - worse, sometimes there is no true background. Conversely, Wipe asks you - if even necessary - to tell it what is definitely not background. For the rest, it relies on undulation frequency of the gradient (slow undulating), which is almost always easily distinguishable (by the algorithm) from actual detail or faint nebulosity (fast undulating). I've likened these different approaches to performing archaeology with a shovel vs doing it with a brush. One causes collateral damage, the other doesn't.

For your immediate vignetting problem, ST 1.7 has a dedicate vignetting stage now that might help somewhat, depending on how badly it is off-center. However, you should take flats as soon as practically possible. They are really not optional in AP, and attempting to just "live with vignetting" will severely hamper what you can achieve with your gear; nor algorithm, nor human can be 100% certain that what is in your image is true celestial detail otherwise.

That said, if you think this is something ST should be able to solve, please feel free to share a dataset with me - I'm always looking for ways to improve ST!

Clear skies!

Hi Ivo,

Apologies for the delay in responding (families, work, life etc) and a big thank you for responding, you don't get that with Photoshop! I do plan on getting a dataset over to you with my ideas, probably through the ST contact form rather than leave this hanging whilst I try to find the time to get something sorted.

Cheers.