Corrupt Data..... Again!!

[Mr Green]

Afternoon All,

So after gathering almost 12 hours of data for my first image with my new scope, I have again come across an issue where almost all my collected data is corrupt and can't be stacked.

I thought that it was perhaps a corrupt USB when the issue first happened but now that I have replaced it with a new one, surely the issue lies elsewhere.

Below is is the message that I get in DSS and Pixinsight also wont open any of my files.

I've attached one of the FIT files from one of the sessions and would really appreciate it if anyone could take a look and let me know if there is an obvious issue I am not seeing.

This is now 2 out of the last 3 projects that I've lost so I really need to get to the bottom of it. (Faulty ASIAir perhaps)

Appreciate any help.

Light_NGC 281_300.0s_Bin1_20240918-205747_0001.fit

[carastro]

I tried to open it in Astroart but it say incompatible file format.  Is that exactly how it comes out of the camera?  

Ditto in Maxim

Edited Saturday at 14:36 by carastro

[Mr Green]

The File has literally just been copied from the folder it was saved to via the ASIAir.

The ASIAir automatically puts the Lights, Flats, Bias etc within there own files on the USB stick connected to the AIR.

[david_taurus83]

Have you tried shortening the file name? That extension path looks very long.

[david_taurus83]

Just now, Mr Green said:

The File has literally just been copied from the folder it was saved to via the ASIAir.

The ASIAir automatically puts the Lights, Flats, Bias etc within there own files on the USB stick connected to the AIR.

Ah I see, I think that’s the problem. Seen somebody else have a similar issue. Save to ASIAir memory when capturing and then transfer to the USB stick.

[Mr Green]

I imaged on 3 consecutive nights and only part of the first nights images are readable in DSS & Pixinsight.

I power 2 dew heaters, osc camera & EAF via the ASIAir, could this be the issue? 

 

[Mr Green]

6 minutes ago, david_taurus83 said:

Ah I see, I think that’s the problem. Seen somebody else have a similar issue. Save to ASIAir memory when capturing and then transfer to the USB stick.

I think this might be a good idea but still doesn't explain why the issue isn't persistent. I had no issues with how I was saving images until the last month or so.

[carastro]

I bet it is something to do with things being updated.  Hate updates they always screw up what was working well previously.  

 

Edited Saturday at 15:16 by carastro

[Mr Green]

I'm just devastated that I've lost 10 hours of data on my first image with new scope.

It makes me not want to put as much time into each image.

I'll keep my fingers crossed that someone has had the same issue and came up with a fix to access the files.

[John_D]

If you look at the fits file definition then it should have an ascii header starting:

SIMPLE  =                    T / file does conform to FITS standard             
BITPIX  =                   16 / number of bits per data pixel                  
NAXIS   =                    3 / number of data axes                            
NAXIS1  =                 1080 / length of data axis 1                          
NAXIS2  =                 1920 / length of data axis 2                          
NAXIS3  =                    3 / length of data axis 3       

followed by more of the same and then the binary data which will look like funny characters in a text editor. ( the above is taken from a fits file that I have ) 

If I open the file that you posted then there are no header lines like this, it just goes straight into binary data. That's probably the reason for the error message however I can't help with why it happens, sorry. Have you got a "good" fits file that you've saved previously? If so then can you check the header lines in that file.

[Elp]

Instead of potentially wasting another night, just run it during the day indoors, camera taking images constantly, mount sidereal tracking, guidecamera on and looping, all power draw and keep an eye on what it's doing.

Remove the usb stick, write images to the internal emmc memory.

Do not power dew heaters via the air, use another power source for them.

Issue could be one of the usb ports, try a different one for the usb drive.

Firstly do the test as you have been using it, then make one change at a time.

[Mr Green]

19 minutes ago, John_D said:

If you look at the fits file definition then it should have an ascii header starting:

SIMPLE  =                    T / file does conform to FITS standard             
BITPIX  =                   16 / number of bits per data pixel                  
NAXIS   =                    3 / number of data axes                            
NAXIS1  =                 1080 / length of data axis 1                          
NAXIS2  =                 1920 / length of data axis 2                          
NAXIS3  =                    3 / length of data axis 3       

followed by more of the same and then the binary data which will look like funny characters in a text editor. ( the above is taken from a fits file that I have ) 

If I open the file that you posted then there are no header lines like this, it just goes straight into binary data. That's probably the reason for the error message however I can't help with why it happens, sorry. Have you got a "good" fits file that you've saved previously? If so then can you check the header lines in that file.

I've attached one of the files that seems to be ok.

The naming convention seems to be the same to me.

Light_NGC 281_300.0s_Bin1_20240917-210808_0001.fit

[Mr Green]

8 minutes ago, Elp said:

Instead of potentially wasting another night, just run it during the day indoors, camera taking images constantly, mount sidereal tracking, guidecamera on and looping, all power draw and keep an eye on what it's doing.

Remove the usb stick, write images to the internal emmc memory.

Do not power dew heaters via the air, use another power source for them.

Issue could be one of the usb ports, try a different one for the usb drive.

Firstly do the test as you have been using it, then make one change at a time.

Definitely a logical thing to do and no forecast clear nights now for the next week so will have plenty of time for trial and error I guess.

I will have to look into power options for the dew heaters as powering via the ASIAir is the only option I currently have.

Just wish I didn't lose so much data. What I have managed to salvage doesn't have much to be desired.

[Elp]

My pro used to stop mid session randomly, it always used to be after I went inside, I'd go back to it and it'd be idle doing nothing. I found out it was the usb WiFi extender I was using at the time (needed with the pro), so I also don't use dew heaters with it as they draw more power than the entire setup combined. Low power and the air will restart without warning (actually there is a warning now, you'll see a low voltage icon on the top left of the screen next to the app and firmware version).

I know the mount can be finniky too, my am3 beeps constantly when using batteries (my ioptron doesn't complain), it's a low voltage warning which you can't turn off.

But I don't think these are the issue. File corruption usually has something to do with read/write issues. I know sometimes if I stop an autorun exposure when taking flats mid exposure, it can write the file but it'll be zero size.

Edited Saturday at 16:23 by Elp

[John_D]

9 minutes ago, Mr Green said:

I've attached one of the files that seems to be ok.

The naming convention seems to be the same to me.

Light_NGC 281_300.0s_Bin1_20240917-210808_0001.fit 49.78 MB · 0 downloads

The "good" file that you uploaded has ( unsurprisingly ) the right header lines. My first thought is that this isn't a hardware issue but some sort of software bug. Are there any support forums for the ASIAir that you can ask if anybody else has seen this problem?

[Mr Green]

5 minutes ago, John_D said:

The "good" file that you uploaded has ( unsurprisingly ) the right header lines. My first thought is that this isn't a hardware issue but some sort of software bug. Are there any support forums for the ASIAir that you can ask if anybody else has seen this problem?

I've looked online and seen a post with the same issue but that individual came to the conclusion that it was a faulty USB stick. (not the case in my instance).

I also looked on ZWO's support forum but am unable to upload the FIT file due to it's size.

Is it possible to amend the header lines in the files that are not working? I am willing to try anything.

[Elp]

Also try a different usb cable from your camera, the flat ones provided are no good.

[Mr Green]

5 minutes ago, Elp said:

Also try a different usb cable from your camera, the flat ones provided are no good.

I'll take a look at the usb cables, think all the ones I am using are those that came with all the ZWO products that I have.

[AstroKeith]

Your bad file, isnt corrupted data - it's just blank. No header and the binary data is 'null'

It's not a good rule to write to a USB stick straight off. The USB stick has firmware in it to manage storage and you may be getting conflicts.

[Elp]

11 minutes ago, Mr Green said:

I'll take a look at the usb cables, think all the ones I am using are those that came with all the ZWO products that I have.

I bought all new shortest (0.5M) ones and shielded from kennable. Never had an issue with them, even in freezing minus 5-10 temps.

[Mr Green]

Would there be any logic in moving the files from the USB stick to the ASIair and then plugging the ASIAir directly into my PC to then move the files across?

I'm not sure if this would even be possible but perhaps the issue with the file naming would be corrected some how.

This probably sounds like a stupid idea, just wondering if there's any way I can get the lost data back.

[Mr Green]

5 minutes ago, AstroKeith said:

Your bad file, isnt corrupted data - it's just blank. No header and the binary data is 'null'

It's not a good rule to write to a USB stick straight off. The USB stick has firmware in it to manage storage and you may be getting conflicts.

I think that now I have lost roughly 18 hours of data (6 rare clear nights), I'll not be using a USB stick with the ASIAir ever again.

[Elp]

If storage is an issue on the internal, and if possible to do it, just take less longer exposures.

[John_D]

34 minutes ago, Mr Green said:

I've looked online and seen a post with the same issue but that individual came to the conclusion that it was a faulty USB stick. (not the case in my instance).

I also looked on ZWO's support forum but am unable to upload the FIT file due to it's size.

Is it possible to amend the header lines in the files that are not working? I am willing to try anything.

You could post to the support forums that you're getting corrupted headers in your fits files and see if anybody comments. Alternatively you could upload the fits files to Google drive or something and post a link ( if that's allowed )

Copying the header lines from elsewhere is certainly possible but you'll probably get complaints from other parts of the software then.

@Elp's idea of running your rig indoors for a few minutes with the minimum of accessories is a good one. You can then check the fits files for corruption. If they're ok then increase the run time and/or add accessories until they do corrupt. Hopefully you should get a small enough corrupt file that you can upload to the ASIAir forums

[fwm891]

I had a problem a little while back (not with an air) but I'd set my USB transfer rate too high and data was getting scambled. Not the file names but the image content was being corrupted.

Just a thought