Strange issue with Light frames. Looks like hot pixels that move frame by frame. What could this possibly be?

[Xiga]

I was out the other night shooting some Lum on Vdb 152 with my Qhy268m. The rig captured around 7 hrs in total, but when i came to inspect the data, i noticed something very strange. 

I use APP for calibration, and have never, ever had any issues calibrating my data. I calibrate with Masters for Darks, Flats, Dark Flats, and a BPM. But when i inspected the calibrate subs, i noticed lots and lots of what look like hot pixels remaining. I think a small number of them are in fact hot pixels (i probably need to take new Darks and a new BPM, it's been 18 months since i last took some) but the majority can't be hot pixels as they are clearly moving frame by frame. What's even weirder, is that the number of them start out quite low, then as the sub count increases, they grow in number until there are absolutely loads of them, and then they start to decrease in number, such that by the end of the night there are not many at all. 

I'm totally stumped as to what this could be. The camera was only running at -5C and i've checked the FITS data for a number of subs and they all show -5 (or -4.9) so i don't think it's a cooler issue. 

I've attached a few raw subs below, if someone could kindly take a look. I've included a few from the early part of the session, a few from the middle, and a few from the end. I've also attached the calibration masters as well. 

Here's a screenshot  at 100% zoom of how Sub 60 looks like after calibration!

I haven't bothered stacking everything yet. I suspect that the dithering and rejection algorithm will clean things up a lot, but i'd still very much like to sort this out as clearly something isn't right. 

vdb 152_120sec_No_ISO_filter0_L_3_frame1.fit

vdb 152_120sec_No_ISO_filter0_L_3_frame2.fit

vdb 152_120sec_No_ISO_filter0_L_3_frame3.fit

vdb 152_120sec_No_ISO_filter0_L_3_frame60.fit

vdb 152_120sec_No_ISO_filter0_L_3_frame61.fit

vdb 152_120sec_No_ISO_filter0_L_3_frame62.fit

vdb 152_120sec_No_ISO_filter0_L_3_frame197.fit

vdb 152_120sec_No_ISO_filter0_L_3_frame198.fit

vdb 152_120sec_No_ISO_filter0_L_3_frame199.fit

MF-IG_56.0-E_4.0s-QHYCCD-Cameras2-Capture-6252x4176-.fits

BPM-QHYCCD-Cameras2-Capture-6252x4176.fits

MD-IG_56.0-E120.0s-QHYCCD-Cameras2-Capture-6252x4176.fits

 

vdb 152_120sec_No_ISO_filter0_L_3_frame63.fit

[ONIKKINEN]

Looked at some of the subs, and cant say anything other than it does look weird. Cant say i have ever seen a sudden increase and an equally sudden decrease in hot pixels with my RisingCam OSC version of the IMX571 camera.

If this is a specific QHY camera issue then try the QHY 268 thread here:

Lots of users of your camera in this forum, someone there could be better informed on how to help with your issue.

[Oddsocks]

Only looked at a random selection of the images and my impression of the noise pattern is that despite the FITS header reporting the sensor temperature is -4.9c the cooling is not actually running.

I suspect that cooling control has "locked up" due to a software or hardware glitch and the -4.9c reported in the FITS header is bogus, the noise pattern has "structure" that appears very similar to what you see when the sensor is uncooled and imaging at ambient temperatures.

The sensor temperature from my own QHY268M recorded in the FITS headers varies by ~ +/- 0.2c during a series but the temperature recorded in a random selection of yours all show the same -4.9c, which doesn't quite fit with my experience.

Possibly the cooling shut down for some reason (low supply voltage protection?) but the camera firmware/driver never reported that back to the acquisition/capture program which continued to report and record the last good temperature reading it received?

There could be a multitude of other causes, the above is just a guess based on the appearance of the images and quick evaluation of a random selection of the images using the Image Statistics module in PI, which showed that the minimum pixel value in each frame was increasing with each successive image, which you would expect as dark current increases with a warming sensor.

Interesting problem....

[Xiga]

Thanks for the response guys. Onikkinen that's a good idea, I'll post a link to this thread in the Qhy268 thread 👍

I think you might be on to something Oddsocks. I use AC power for everything so I don't think it was a power issue, but I agree it does look like the cooling wasn't working properly for whatever reason. I took a long break (over a year) from astro, and this was only my 2nd session getting back into it. I looked over the subs from the 1st session from a couple of weeks earlier, and those subs were also affected. I also checked some subs from a year ago and they looked fine. 

I'm not convinced just yet that the cooling is booked, so i think I'm going to re-install the firmware and see if that helps. I think it's still running the version from when the camera was first released. 

Forecast looks awful here for the next while, so I might also do a test of shooting some darks with the cooler off, followed by some at -5C, and see if they look as expected. 

Cheers guys.

[Xiga]

I looked at all 210 subs a bit more closely, and i've found a weird correlation between when the mount did it's meridian flip, and a decrease in noise. 

The mount was due to do it's meridian flip quite early, but because the guiding was going so well i just decided to let it continue on past the meridian. I've been having mount issues and i wasn't confident that guiding would continue as well on the other side, so i let it go on for maybe 90 mins past the meridian before i told it to flip. 

The first 99 subs were done on one side, and they all exhibit noise that suggests cooling wasn't fully working (despite SGP saying it was ). The mount then flipped, and then from subs 100 to about 115 there is a gradual but very noticeable decrease in noise with each passing sub, before it eventually levels off and the subs all look nice and clean as expected. So somehow, the meridian flip un-glitched the software and the cooling suddenly started working fully again, is my best guess so far! Note, i checked the FITS header for most of the subs, and the temp does fluctuate between -4.9 and -5.0 in all of them. I would have felt better if they all showed the same value, but they don't, which only confuses things even more, lol. 

ps - I've now taken some new darks with and without cooling. I'll post those next. 

[vlaiv]

5 minutes ago, Xiga said:

So somehow, the meridian flip un-glitched the software and the cooling suddenly started working fully again, is my best guess so far!

I don't think meridian flip un-glitched the software

I'm more inclined that meridian flip un-glitched dodgy power connector for 12V to camera cooling. Without it - peltier won't work and although software might think it is cooling the camera - without power it won't.

[Xiga]

3 hours ago, vlaiv said:

I don't think meridian flip un-glitched the software

I'm more inclined that meridian flip un-glitched dodgy power connector for 12V to camera cooling. Without it - peltier won't work and although software might think it is cooling the camera - without power it won't.

Hmm, now that you say it Vlaiv, that does sound like a more logical conclusion! 

I run everything from inside the house, via an extension cord plugged into an RCD (for extra protection) which goes to a weatherproof box, which houses all the power bricks. I do have a 12v extension cable for the camera, as the one it came with is nowhere near long enough, but it has a nice snug fit and doesn't seem loose of flimsy. I've never had any power issues ever since i moved to AC many years ago (best thing i ever did tbh) but i guess something must have come loose somewhere, somehow. I've opened up the weatherproof box and checked all the connections just to be sure, but nothing was loose. I guess i'll just have to keep an eye on the next few sessions and see if it happens again. 

FWIW on the new darks i took - on the first 20 (uncooled), they started out all the same, ie noisy, then weirdly on subs 8 to 14 they suddenly became a lot cleaner, then on subs 15 to 19 they returned to being noisy again, but then sub 20 was clean, similar to subs 8 to 14. Very strange. But on the cooled ones (subs 21 to 40) they all looked the same, so i'm happy that the cooling worked. I won't bother posting them, unless anyone wants to see them!