ASI1600 and 130PDS - about to give up, need help

[vlaiv]

Just now, BrendanC said:

I'll see if I can do a longer animation maybe using PIPP and try to understand this,

You can do it in ImageJ (at least I did) - and yes, do whole range of luminance subs to see if you can find anything useful in how it behaves.

This is only 30 first subs (calibrated and normalized and binned x10 to increase SNR of individual sub).

[teoria_del_big_bang]

Following on from @vlaiv observation about the external light I was also running the subs though a similar animation and for sure on the luminance subs you start imaging at 10 PM on 1st June and maybe at this time of year with no astro darkness in UK that is really too early. Once it gets to 11 PM the subs seem to vastly improve. So if you stack all the subs with equal weighting maybe these earlier subs are doing you no favours, especially on such a dim target anyway.

Below are just the Luminance subs but all with same stretch so you see the actual difference between subs, it starts with the first exposure at 10 PM and runs though all subs twice ending up back at the first one.

 

 

Steve

 

[BrendanC]

Good point, thanks for this. I'm working on removing the bad subs. I still have no idea why I get gradients on everything else though.

Btw I started at 11pm, not 10pm.

Edited June 6, 2022 by BrendanC

[teoria_del_big_bang]

I do feel that this set of data may not be the best to give you help with as the target is so feint and to be fair no real astro darkness when they were taken, although I think you said no moon so maybe that's not a big issue, but the feint target means the image with a quite low integration time was never going to be great. Looking at other peoples images of this target (not one I have even tried) many are poor and many not as good as yours.
When I look at the actual background levels they do not seem to vary that much around the image even the stretched image looks bad I think it is just stretched so much that any small change in levels looks huge.

Now some of the other images you put in one reply with much brighter galaxies are a different matter and these may be better data sets to look at.
I am not so much experience as others like Vlaiv but having got that camera (different scopes mind) and so many others that have it unless it is faulty I cannot see how it could be the camera.
But like other I do feel for you and understand your frustration (I know that's not helping and that's what you need).

Steve

[david_taurus83]

To answer a few of the questions you asked me earlier.

DSLR: I went through a stage of having way too much gear as well as saving for a house purchase so had a quiet word with myself and decided to sell a lot of it but I wanted to keep one foot in the hobby and kept the Canon 6D. It's proven to be such a good camera I eventually decided to keep it as I've built my kit back up.

I have an Atik 460 which I bought second hand back in 2020 and just last week bought the new ASI533MM. This is setup on my main scope, an Esprit 100 but the 6D is currently setup on my Redcat 51 just waiting for those larger targets to get a bit higher in the sky.

My point about the flats being difficult to take on a newt. Putting a lightbox directly onto the OTA is 100% going to introduce extra light into the focuser. The only light that should reach the camera should be reflected from primary to secondary to sensor. You really should try and take your flats by pointing at an even light source at a distance away from the OTA. At very least set the light box at the end of the dew shield. This will help keep extra light out of the focuser.

If you really want to keep using the 130PDS then you need to acknowledge its shortcomings. Its an entry level scope, not a precision instrument. You should try and modify it to improve your results. Cut the tube down like you've said, flock the tube including the focuser. Put an aperture ring around the primary to tighten up your stars. I recently bought a 200P myself and I will be doing all those things to try and improve it! 

 

[barbulo]

Blink is the first process I run to discard subs with clouds, thick trails or overexposed. Then I move to Subframe Selector.

[Pitch Black Skies]

Brendan, you don't happen to image with your set up exactly like this do you?

[teoria_del_big_bang]

This is a long shot but is there anything in the processes you use for calibration that reads the fits headers.
Reason I ask is because I am trying to use WBPB script in PI to load files and it thinks the Dark Flats (which are actually darks for the flats)  are Flats and loading them up as flats.
I am sort of assuming that the software you are using is just manually loading files so not a problem for you the headers has the wrong image type, but just so you know if you do go to trying PI then this may be an issue.

Steve

[BrendanC]

Nice spot, thank you. That's probably something to do with me running the same plan to produce flats as for dark flats. When I import them into APP I apportion them manually so it's not an issue. 

[inFINNity Deck]

6 hours ago, BrendanC said:

 - how do I determine whether I have one of the older models that has this problem? Is it possible that Vlaiv's model doesn't have this issue?

Here they are, all three models ASI1600MM Cool, only the one with PRO on the housing does not suffer from this issue if I am right.

By the way, my version of NGC4236 shown above has an integration time of 11 hours, 7 hours of which was luminance and still I think I need to at least double that.

Nicolàs

[BrendanC]

14 hours ago, david_taurus83 said:

If you really want to keep using the 130PDS then you need to acknowledge its shortcomings. Its an entry level scope, not a precision instrument. You should try and modify it to improve your results. Cut the tube down like you've said, flock the tube including the focuser. Put an aperture ring around the primary to tighten up your stars. I recently bought a 200P myself and I will be doing all those things to try and improve it! 

 

Have done all those things! Not flocked the entire scope, just behind the secondary. Have recently chopped the tube and installed a baffle a while ago.

11 hours ago, inFINNity Deck said:

Here they are, all three models ASI1600MM Cool, only the one with PRO on the housing does not suffer from this issue if I am right.

 

So, I need to figure out a way of enabling >1 sec flats with a lightbox that I purchased specifically to help with flats, which at the dimmest setting produces flats of considerably less than a second. No idea how I'm going to do this. I could put bits of paper or a t-shirt or something between them but the whole reason I got the box was to get as pure light as possible, because I suspected using paper etc was giving me bad flats.

14 hours ago, david_taurus83 said:

My point about the flats being difficult to take on a newt. Putting a lightbox directly onto the OTA is 100% going to introduce extra light into the focuser. The only light that should reach the camera should be reflected from primary to secondary to sensor. You really should try and take your flats by pointing at an even light source at a distance away from the OTA. At very least set the light box at the end of the dew shield. This will help keep extra light out of the focuser.

 

I'm still very interested in what @vlaiv has to say about flats taken with this camera. 

 

13 hours ago, Pitch Black Skies said:

Brendan, you don't happen to image with your set up exactly like this do you?

 

Yes. I've tried putting the ASI1600 below the OTA but it doesn't work with the rings and auto focuser. Everything balances fine, and as far as I can tell I'm not getting any focuser slop. Why?

Edited June 6, 2022 by BrendanC

[Pitch Black Skies]

22 minutes ago, BrendanC said:

Yes. I've tried putting the ASI1600 below the OTA but it doesn't work with the rings and auto focuser. Everything balances fine, and as far as I can tell I'm not getting any focuser slop. Why?

No it's not that. I can see that the primary end is exposed. The 130P-DS's have serious light leaks at the primary end and I'm wondering if that could be the cause of your problems. If you look down the tube through the 50mm aperture on the dust cap while moving a torch around the primary you should see light beaming through the edges. A rubber swimming hat or a shower cap placed over the primary end completely rectifies it.

[vlaiv]

51 minutes ago, BrendanC said:

I'm still very interested in what @vlaiv has to say about flats taken with this camera. 

I haven't had issues with flats on this camera with even very short exposures (few milliseconds). My model is one of early models - not pro version.

Both of my scopes are well baffled - refractor and RC.

I can see how newtonian could be bit of a problem with respect to flat calibration since it is not well baffled design.

If you want to check linearity of your flats - there is simple way to do it.

Take flats with few different exposures and calibrate them one against another. They should produce flat even images regardless which exposure you used to calibrate another.

 

[ollypenrice]

19 hours ago, BrendanC said:

OK, so I'm back. Here's some food for thought...

Here's NGC 4236 using all my data, which is about 7 hours' worth. Slightly better maybe but still bad stars.

 

Here are some examples of what I've been up against - these are just taken into the StarTools Compose module, binned, wiped, and then given an AutoDev stretch to show what's going on in the data. Gradients a-plenty.

They were all taken in quite radically different sky conditions, different place in the garden, different methods of capturing flats and darks, different offsets and gains - but still, mad gradients and stars. I've been able to make something of them, but this is after judicious cropping and fighting the gradients, which I just feel I shouldn't be having to do with this camera. Regardless of how faint NGC 4236 is, one day I really will want to do a full star field, for example an open cluster, and I just won't be able to that at this rate.

So, state of play is:

• Bad stars - is this microlensing? Too much exposure? Too little? (btw I know the 'bitemarks' are the result of the focus tube impinging on the FOV which I'm also going to fix soon)
• Gradients - is this a light leak? Bad filters? Bad flats? Too little data? I don't quite see how more data will fix these gradients. 

@vlaiv Here are the averaged masters (plus stacks) that you asked for, all done at 32-bit in APP, 901MB in total, which hopefully you can download now (and anyone else who fancies a go): https://drive.google.com/drive/folders/1wt0HEs7_vwI--TeWzWcmMQO7MIKoXgQh?usp=sharing

You've already helped me establish that my library darks are ok, but that I might have a leak in my focuser tube. So, I've got the focuser tube wrapped in a snood, which also has tin foil lining it. However, I must admit I haven't since tested this using ImageJ as you recommended, which I probably should. I am sorely tempted to try, one night, actually wrapping the whole scope in foil, just to try and block out any extraneous light whatsoever, and see what difference it makes. I'm still very puzzled as to why this should be happening though, given that I was achieving 300s exposures with my DSLR, with perfectly flat wipes in StarTools. I should also mention that Ivo, the creator of StarTools, is very much of the opinion that my problem is with my flats, which is why I got the Lacerta box. However, given that you have short flats exposures then what do you make of this:

@inFINNity Deck - how do I determine whether I have one of the older models that has this problem? Is it possible that Vlaiv's model doesn't have this issue?

Other points:

Which one? And, out of interest, why did you go back to a DSLR after the 1600? And then back to mono?

Really don't want to change the scope too. I've invested 2+ years getting to know it, and it's a fab little performer. As I said above, I was achieving 300s exposures with the DSLR with none of these problems. The light box, when taking flats, is absolutely butted up against the OTA, with a retaining mask around it, there is no light leaking out from that at all.

Yes, and I do use one.

At the end of the day, my real worry is that, after all this, if I did then decide to go back to an OSC - not least because I also find the 4x time/storage/processing etc very fiddlesome with mono - and I continue to have these problems, I'll be totally at a loss. I find this all constantly gnawing at the back of my mind, like tooth ache, and it's not a good way to live currently. It's like those threads you read in the forums, and think 'Thank God that's not me' except, this time, it's me. 

The other big problem is that I'm doing this totally on my own, with nothing to compare anything with. I'm toying with the idea of contacting a local astro club to see if they could help. So, if anyone's reading this who lives in Oxfordshire or Bucks, and would like to help (and I'd be prepared to pay), then please feel free to message me.

OK, looking at the 5 greyscale images in this post, all but image 2 show very obvious inverse vignetting as caused, routinely, by over correcting flats. In image 2 something else is going on as well. 

I really can see no room for doubt about this but, to confirm it, you just need to make stacks without flats and see that the bright corners and dark centre will be transformed into dark corners and bright centre. So far this is all perfectly mainstream but, where it gets difficult, is in working out why the flats are over correcting. I operated a setup (not my own) on which we never got to the bottom of it. What's even odder is that the over-correcting-flats problem popped up one night despite absolutely no identifiable changes to any aspect of capture or pre-processing. I would certainly follow the suggestions of Nicolas regarding flat exposures. When I was struggling with this problem Harry Page, who knows what he's talking about, suggested that there might be a problem with RBI (Residual Bulk Image) when capturing flats in quick succession. It might be worth trying a multi-second delay between flat exposures. (Harry was talking about CCD so I don't know whether this is relevant to CMOS or not but it's a dead easy experiment.)

This is not an alternative set of comments to Vlaiv's. It's perfectly possible to have two problems at once. I would certainly check that there is no small equipment light coming on and off during your runs. We've just found that, after a change in our RASA cable routing, a tiny camera light was reflecting off our cable guides and creating a huge gradient. Obviously the front-camera RASA is particularly sensitive to this but I'm wondering if you have something which isn't lit when you check the system but which decides to light up once you're not looking!

What you should avoid at all costs is using the black point to clip out the gradient. Gradients must be adjusted out, not clipped out. To do this the software needs the full range of brightnesses both above and below the target background brightness. If you black clip you leave it nothing to work with on the side below the target.

Olly

[inFINNity Deck]

2 hours ago, BrendanC said:

So, I need to figure out a way of enabling >1 sec flats with a lightbox that I purchased specifically to help with flats, which at the dimmest setting produces flats of considerably less than a second. No idea how I'm going to do this. I could put bits of paper or a t-shirt or something between them but the whole reason I got the box was to get as pure light as possible, because I suspected using paper etc was giving me bad flats.

 

I always use a white foam cap on the Esprit 150ED. A friend of mine has darkened the flat-panel by applying car window tinting foil to it.

1 hour ago, vlaiv said:

I haven't had issues with flats on this camera with even very short exposures (few milliseconds). My model is one of early models - not pro version.

I would be interested receiving a few flats from that camera taken with exposures of 0.95s, 1.05s, 4.95s and 5.05s to see if that gradients shows. The graph I posted earlier was actual data of a non-Pro ASI1600MM.

If possible I would like to receive these from both Brendan and Vlad.

Nicolàs

[vlaiv]

6 minutes ago, inFINNity Deck said:

I would be interested receiving a few flats from that camera taken with exposures of 0.95s, 1.05s, 4.95s and 5.05s to see if that gradients shows. The graph I posted earlier was actual data of a non-Pro ASI1600MM.

If possible I would like to receive these from both Brendan and Vlad.

I'll see what I can do - my flat panel is extremely bright and I use millisecond exposures - like literally few milliseconds for luminance and 30-40ms for NB filters.

Not sure if I'll be able to significantly decrease its brightness (order of x1000) in order to shoot 5s exposures.

 

[vlaiv]

On 03/06/2022 at 15:28, inFINNity Deck said:

The ZWO ASI1600MM Cool comes in a normal and pro version. The normal one does not like flats taken with exposure times below 1s (USB2) or 5s (USB2). Those produce a sloped ADU level when compared to longer exposures:

So these gradients can simply be solved by taking care that the flats-exposures are longer than 1s (USB3) or 5s (USB2).

I did not properly pay attention to this post - but I have several issues with it.

This shows what appears to be influence of "global shutter" on flat exposure rather than non linearity of camera response.

X axis is in pixels and not exposure length so above graph does not represent non linearity of camera - but apparently some sort of shutter artifact.

Problem is that ASI1600 has a rolling shutter - not global. It is electronic rolling shutter and exposure length can't impact its operation (unlike with mechanical shutters - where you have to ensure that exposure is long enough versus shutter speed).

I simply can't believe that above graph is true - it's certainly not true for my camera.

Here are three examples of master flat files. First two are from RC 8" scope, third is from 80mm F/6 refractor. Third also suffers from slight OAG shadow in the top of the image (I did not pay full attention to completely avoid casting shadow)

All three are made with very short exposures. In fact - it takes much more to download file in SGP than what exposure is (for some reason dowload in SGP lasts for about second although camera is capable of higher FPS). I shoot 256 flat exposures and associated flat darks in about 10 minutes.
 

Never had issues with flat calibration of my images, and only gradients I had are from either passing high altitude clouds (which can easily be detected by doing frame animation) or linear LP gradients that are easily removed (and also change with the course of session as target is being tracked).

 

[vlaiv]

In addition, @BrendanC posted master flats - which show the same thing:

This is ~90ms master flat for luminance.

And Red channel at 0.34s

There is no significant tilt in flat exposures.

[inFINNity Deck]

I had to do quite a search for the original thread in which I discussed this matter with the owner of a ZWO ASI1600 Cool, Jac Brosens (jbrosens), on a Dutch forum, but here it is (if you open it in Chrome it will be automatically translated).

It appears that there have been different firmware versions causing different effect. The graph I showed before was taken with jbrosen's camera and had the tipping point between rolling and global shutter at 1 second. According to ZWO (see post 55 in that thread), the tipping point should be at 2 seconds for the ASI1600 and at 1 second for all other cameras. At the same time they told me that the Pro has no tipping point (whereas they first wrote that it was at 2 seconds), so there is some contradiction in their explanation, which is further discussed in post 56.

Post 55 shows that the Pro that I have has no tipping point. In post 57 the effect is shown by jbrosens in his own flats using the same approach as Vlad just did, but now clearly showing the effect. I was sent the original fits which I processed using my own software and resulted in the graph I posted earlier. The original FITS-files are still in my possession and can be downloaded from my server.

Nicolàs

 

Edited June 6, 2022 by inFINNity Deck

[vlaiv]

3 minutes ago, inFINNity Deck said:

t appears that there have been different firmware versions causing different effect.

This has been common theme with CMOS sensors and ASI (and probably other vendors as well) - results seem to be heavily impacted by driver / firmware versions.

I would not be surprised that this has actually been fixed in drivers in the mean time.

[teoria_del_big_bang]

2 hours ago, ollypenrice said:

OK, looking at the 5 greyscale images in this post, all but image 2 show very obvious inverse vignetting as caused, routinely, by over correcting flats. In image 2 something else is going on as well. 

I really can see no room for doubt about this but, to confirm it, you just need to make stacks without flats and see that the bright corners and dark centre will be transformed into dark corners and bright centre. So far this is all perfectly mainstream but, where it gets difficult, is in working out why the flats are over correcting

As I had all this data already open in PI I did what you suggested for the Lum anyway. A stack using uncalibrated data left and Right is after calibration with darks and flats.
Note: the uncalibrated was just the raw files so these do not have darks calibration applied either but  that should not affect the vignetting unless the darks had an issue which I do not think they do.

Steve

Edited June 6, 2022 by teoria_del_big_bang

[BrendanC]

3 hours ago, Pitch Black Skies said:

No it's not that. I can see that the primary end is exposed.

Ah, I see what you mean. I have a hat over the end of it now, very similar to how you're set up. Have had this arrangement for quite some time now.

 

2 hours ago, ollypenrice said:

you just need to make stacks without flats and see that the bright corners and dark centre will be transformed into dark corners and bright centre.

I think I've probably already tried that (I'm losing count of what I have and haven't tried), but I'll try it again.

 

1 hour ago, inFINNity Deck said:

I would be interested receiving a few flats from that camera taken with exposures of 0.95s, 1.05s, 4.95s and 5.05s to see if that gradients shows. The graph I posted earlier was actual data of a non-Pro ASI1600MM.

 

Like @vlaiv, I do not know how I can achieve this.

 

14 minutes ago, teoria_del_big_bang said:

As I had all this data already open in PI I did what you suggested for the Lum anyway. A stack using uncalibrated data left and Right is after calibration with darks and flats.

Thanks for this but I'm afraid I'm confused (again) - does this bear out what Olly's saying about over-correcting flats?

[BrendanC]

So, the state of play now is:

• The scope isn't up to the job
• The flats are over-correcting
• The flats need to be >1 sec

... in addition to the first set of bullet points I listed. I don't really feel I'm much closer to a solution.

Let's say I stick with the scope, and want to fix the flats:

• Am I right in thinking that I can reduce the gain/offset to zero to try and get longer flats, even if this is different from the subs?
• Also, if I have to resort to putting sheets of paper in front of the light panel to achieve this, will inconsistencies in the paper cause artefacts in the flats?

Let's say I stick with the scope, and want to change the camera:

• What do people think my chances would be of getting better results with an ASI533MC which is looking like an increasingly attractive alternative?

 

[ollypenrice]

One way to mitigate a stack which has been over-corrected by flats is to blend it with a stack without flats at whatever relative weighting gives the best result. You can do this in many programs but it's best to use one which gives a screen-only stretched view of the linear data. That way you can put one on top of the other and weight them while seeing what you'll get. I used to do this with our problem rig using AstroArt. It isn't a solution but it kept us in business.

Olly

[BrendanC]

Thanks Olly. Honestly, I wouldn't even know where to begin doing this, however. Plus, I don't know whether I want to, especially not for every shoot I do.

I've taken flats every which way, and stacked using Siril, DSS and APP, and still don't know why this is happening.