A global repository for data

[Datalord]

I've had a thought for some time about collaborations and data sharing. Lately Astrobackyard started sharing some of his data on his site and while it is a fun social experiment, it's not quite what I had in mind. Imagine this:

A site where you can upload you data. A submission on a target would have to include:

• Target name
• RA/DEC
• Scope details (FL, aperture, brand etc)
• Camera details ( model, pixel size, gain etc)
• lights, flats, darks and bias, with info on filters and binning etc
• Capture dates and times for all subs
• Location of captures
• Possibly a report on statistics
• A picture from the author showing their processing of the target.

Why?

1. Imagine having 300 hours of data on a target instead of 10. 😮
2. Reference for your own data to compare both subs etc.
3. With the glory of AI and machine learning some researchers could possibly make something extraordinary out of this.

Questions:

Would you share? If not, why?

Would you use someone else's data, possibly to merge, mosaic etc? If not, why?

Is it a good idea? bad?

Does it already exist?

Any suggestions or things I haven't considered?

[vlaiv]

Answers:

Yes, why not.

I would for variety of reasons. Best thing that I could think of in using loads of data would be to test out different algorithms in "real scenarios".

Good idea.

Not sure if it already exists.

Funding for such project. There would be a need for very efficient compression algorithms - this would reduce upload/download times for most people (except those with very fast internet - don't think they would care much ) but also storage requirements. I guess it could be a sort of "crowd" funding by people interested in using it - like subscription fee or similar. This however runs a risk of dying out if people loose interest - it would not be durable to sustain fluctuation in interest.

As for data, there is one thing that is probably very hard to do - instrument response. If you want to do a good job of combining different data, you can either go "blindly" about it, but better way is to know instrument response, and probably atmospheric response as well. It is actually not easy to combine different data if you are to care about such things. For example consider very simple scenario - two stars of different stellar class. One is hot blue star, other is cool red star. Now imagine that you have two sets of data that are "equal" in most things - scope used, resolution, position in the sky, etc ... except for camera response to wavelengths.

One camera is more sensitive in blue, other one in red. You have luminance data from both sources and you try to combine those two images by using weighted stacking. Weighted stacking will try to determine SNR in each sub. One of the ways to do it is to assume that "straight" stack will have the least noise and subtracting each sub from such stack will give you estimate of the noise in that sub. You use that noise in relation to noise from other subs to assign weights. You do weighted average and repeat the process (do couple of iterations until values of weights stabilize). But here you have a problem - blue star will have more signal than red star in one set of luminance data, while it will be reversed for other camera. Here we are not talking about the noise, but rather difference in signal. You have no way of knowing that signal is different, but you need to account for that, otherwise algorithm will conclude that deviation in signal is due to noise and it will calculate wrong weights.

Only way this could be done is to have wavelength response for each camera and also consult filtered datasets in order to derive Lum - Lum transform that will produce "equivalent" output from two different cameras.

Not sure if there is algorithm for such thing, but I'm happy to give it a go once we have a repository of data and different recordings of the same target

In any case - camera response is not easy thing to measure, and while we can rely on manufacturer data - I found that it might not be as reliable as we would hope. I was searching for QE curve for ASI1600 and there is one graph that is "shared" all over the net - supposedly from manufacturer (was not able to confirm this). There is another such graph created by Christian Buil using spectroscope. The two differ significantly, and I have no idea which one might be more accurate. On top of this we would need to add scope response - coatings used on optics have different transmission / reflection curves as well.

[Datalord]

Yeah, I've actually tried to combine some of my CMOS RASA pictures with my CCD RC and it is definitely not simple. However, I got far enough that I'm certain it can be done with enough tinkering. And if I can do it, computers can.

Data storage wise, I think only a Google Cloud Platform or Amazon S3 storage is viable for something like this. It's not outrageously expensive in raw storage. But you have a point about compression.

Cost and price wise, it has to be free. I think there has to be a login mechanism and some approval of accounts before submission, but this will only work if the premise is that the upload is free. Maybe there could be some throttling on downloads, but that would be a matter of success showing the way.

More to ponder.

[vlaiv]

At some point - someone will have to secure funds for storage and hosting. If it is a free service (which I support as idea), we would need to find some sort of sponsoring - even if at first we need to pay hosting ourselves.

Combining from different sources need not be too complicated, in fact it can be done quite easily if you don't mind sub optimal solution. If you skip weighted average and go for straight average - it can work without problem. Sigma clipping and other things will suffer from above mentioned problems - to what extent? I'm not sure.

[Datalord]

Anyone else with opinions? 

[Gina]

It doesn't interest me.  I'm only interested in processing my own images.

[kirkster501]

Respectfully there are numerous data pool resources already available so I think such an endeavour would be reinventing the wheel.

http://www.mistisoftware.com/astronomy/index_fits.htm

..for example.  You can even download Hubble data to process yourself.

http://hla.stsci.edu/hlaview.html

Personally, I am with Gina on this;  I cannot consider a picture as "mine" unless I captured it with my own hands and my own equipment and I processed it.  That's surely the whole point of astrophotography for amateurs.

[vlaiv]

3 minutes ago, kirkster501 said:

Respectfully there are numerous data pool resources already available so I think such an endeavour would be reinventing the wheel.

http://www.mistisoftware.com/astronomy/index_fits.htm

..for example.  You can even download Hubble data to process yourself.

http://hla.stsci.edu/hlaview.html

Personally, I am with Gina on this;  I cannot consider a picture as "mine" unless I captured it with my own hands and my own equipment and I processed it.  That's surely the whole point of astrophotography for amateurs.

Not sure if you understood the idea ...

There are repositories of data that people put up them selves (amateurs) or scientific institutions, but there is no repository of data that you can go and upload your data to and make it available to others unless you create your own repository and host it yourself.

Here we are not talking about "ownership" of the image, but rather accessibility to data that you can use in variety of ways - compose into images (if processing is your thing), do comparison between the sets, maybe do some "scientific" work in amateur domain ...

[kirkster501]

4 minutes ago, vlaiv said:

Not sure if you understood the idea ...

There are repositories of data that people put up them selves (amateurs) or scientific institutions, but there is no repository of data that you can go and upload your data to and make it available to others unless you create your own repository and host it yourself.

Here we are not talking about "ownership" of the image, but rather accessibility to data that you can use in variety of ways - compose into images (if processing is your thing), do comparison between the sets, maybe do some "scientific" work in amateur domain ...

Ok understood.  No interest to me so I'll bow out.  Good luck.

[Datalord]

3 minutes ago, kirkster501 said:

there are numerous data pool resources already available

Yes, from individuals. That's what got me started on this train of thought. These are all quite limited in exposure times and it's not in one repo with the right information tagged.

8 minutes ago, kirkster501 said:

 I cannot consider a picture as "mine" unless I captured it with my own hands and my own equipment and I processed it.

What I'm after is collaboration. We've seen a few on this site where people with different gear and focal lengths collaborate to make truly astounding pieces of work. Or the example of the 200 megapixel mosaic posted earlier this year.

Imagine having your own 10 hours that you can add another 25 hours to, maybe you saw some LRGB that you added Ha to or maybe you found long FL data to add to your own widefield. Use it and credit the other author(s) and make a wonder.

6 minutes ago, vlaiv said:

compose into images (if processing is your thing), do comparison between the sets, maybe do some "scientific" work in amateur domain ...

All of this. Personally I often wonder about the subs people get with their equipment. 

[glowingturnip]

why not try an impromptu experiment here ?  Pick a target, and ask people to give a dropbox link to their data if they're willing to play, then we could all play around trying to process it

edit: should probably ask for calibrated subs, rather than go overboard with data on flats, bias, darks

Edited June 21, 2019 by glowingturnip

[Datalord]

5 hours ago, glowingturnip said:

why not try an impromptu experiment here ?

Indeed, why not. I shall commence...

5 hours ago, glowingturnip said:

edit: should probably ask for calibrated subs, rather than go overboard with data on flats, bias, darks

Good point. My naive mind said to add all of it because I'm used to it, but that makes no sense. 

[vlaiv]

1 hour ago, Datalord said:

Indeed, why not. I shall commence...

Good point. My naive mind said to add all of it because I'm used to it, but that makes no sense. 

At some level it actually does make sense.

Take for example need to correctly model sensor that took the data. One component of it would be read noise - another dark current. Some pixels in CMOS sensors for example suffer FPN (fixed pattern noise) - these can be easily identified by examining dark/bias set.

Maybe whole set is not required, maybe just some statistics of the set - like master dark and standard deviation stack or similar.

[Datalord]

1 minute ago, vlaiv said:

like master dark and standard deviation stack or similar

Probably reasonable to have masters present to aide with troubleshooting, especially if a set has difficult gradients or something similar. Since the intent is going to be amateurs data, we have to assume it won't be perfect.

[tomato]

It’s an interesting concept, since acquiring APP, I’ve combined data old and new, different scope and camera set ups and recently CCD and CMOS data, all with reasonable success, so I am sure it would work. 

I wouldn’t use it to produce images, but would be happy to contribute data.

Unless we all got really clever, I can see the images being heavily cropped....