Debayering a7S II and a7S III for video

[kylemcdonald]

Hello all, I’m helping with a cultural heritage project that requires an extremely low light video camera. We’re trying to capture a well-documented but never-photographed luminous phenomena that happens on the ocean at night. By following the posts on this forum and others, I learned about mono full-spectrum conversions. But my journey has been rough, and I wanted to ask for guidance from the experts

I got lucky with my first debayered camera: I found a Sony a7S II on eBay that had already been modified by Monochrome Imaging. It worked great for a few months. Check out this test video I recorded showing glint from Jupiter on the Pacific, just off Venice Beach in Los Angeles.

But two weeks before we set sail, the camera stopped working. It would go into a restart-loop when I turned it on. One repair center told me it was a problem with the sensor, but Monochrome Imaging offered to take a look (they have been amazing) under the suspicion that it was the main board. We just found out it is not the main board.

Right after this glitch appeared, I contacted LDP LLC (MaxMax) in hopes of converting an a7S III to mono full-spectrum. LDP has also been amazing, they took the rush job, shot some beautiful test images, and shipped it to me. But when I unboxed and turned it on, the display briefly showed some blocky artifacts, and then only a black image. I was devastated. Now I am looking for another a7S III sensor in hopes that LDP can try conversion again, but I’m concerned that we don’t really know what caused the failure.

So I’m here to ask:

1. Does anyone have experience with successful a7S II or a7S III mono conversions?
2. Where might I be able to find a reasonably priced a7S III sensor? The best I've found so far is $1600 on AliExpress.
3. Is there anyone else besides Monochrome Imaging or LDP, preferably in the US, who have near 100% success rates? Or am I hoping for too much here? I know there are different processes (chemicals, scraping, laser) but I’ve had trouble figuring out who else is still operating.

I also wanted to share some code I worked on as part of this project. I made a tool for extracting the raw 4K high-bit-depth pre-bayer-mixed image from an a7S III. If you stream from the a7S III to a device that can capture the ProRes Raw encoded video, like a Ninja V, it’s possible to get 4K grayscale video at 30fps.

[vlaiv]

Out of interest, what are the requirements for your project?

What sort of resolution you hope to capture, what sort of FOV / focal lengths are you working with?

It might be much more feasible to just use astronomy mono camera as there is no need to debayer anything? Over USB 3.0 it can achieve pretty decent frame rate in high bit depth.

Take a look at this camera for example:

https://astronomy-imaging-camera.com/product/asi183mm-mono

With right sort of laptop (ssd, usb3.0, decent processor and memory) - it will be capable of

3840×2160 at 36.12fps

Only drawback is that it is 1" sensor - much smaller than APS-C and full frame - and that needs to be taken into account when choosing matching lens if FOV is important.

[kylemcdonald]

Thank you for the thoughts. I previously looked into these cameras and found that we were unlikely to get similar low light performance. But I would appreciate your feedback, I will outline my best understanding.

Our requirements are: 24-30fps, 1080p-4k, around 50mm (40 degree horizontal angle of view), needs to be able to record continuously for more than 4 hours, and the lowest noise/most light we can get. This flash of light lingers for around 1/2 second and is sometimes reported as being slightly colored but often more dim than bioluminescence. Which would put it at around 0.01 lux, near the limit of mesopic vision.

The a7S II and a7S III should be around 55-65% QE with a bayer filter (here is some evidence the a7S II has a better SNR than the a7S III). The camera you shared has 84% QE without a bayer filter. While removing the bayer filter should increase the amount of light we get in, it will also remove the microlens array, and total gains are still unknown. Maybe an extra half stop of light. The bigger difference is that a full-frame sensor has at least 6x the area of a 1" sensor. I'm unsure how to compare the read noise, which is reported 1.6e @ 30dB for the ASI183. I have seen read noise estimates for the a7S II around 1e in the higher ISOs (see here and here). So overall I think there should be no comparison, but mainly because of the sensor size. 

However, I am still new to all of this. If you think it's worth looking into I might just try to find an ASI183 dealer with a generous return policy, and try a side-by-side to check the noise characteristics. Thanks again.

Edited February 1, 2022 by kylemcdonald

[vlaiv]

Low light performance is not something that is intrinsic to camera - it is combination of factors and the way you utilize the camera itself.

If you fix your field of view and opt for say 1080p over 4K as it will have better low light performance - then it is the matter of having the most aperture at these settings. That combination will have best low light performance.

For example - using 24FPS over 30FPS will provide significant benefit as each exposure will be 25% longer (25% more signal just by choosing FPS).

1080p versus 4K will be even more crucial. 4K resolution has x4 more pixels than 1080p - that means same FOV is spread over x4 more pixels, or signal is spread over x4 pixels - each pixel gets only 1/4 of the light.

Important question is how this full frame sensor records video - in particular when you select 1080p video, is data binned internally or not? I guess it is compressed - which is not ideal, compression artifacts can cause issues with low level signal.

If you opt for any of astronomy cameras, then here is what I would advise for best low level performance:

- use binning so that you get 1080p output. For example ASI294mm (still cheaper than those Sony cameras) can bin x2 to get 9.26µm pixel size and still provide you with 1080p type resolution

- use raw format.

This is going to another problem for you to solve.

1920 * 1080 * 24 * 60 * 60 * 4 = 716636160000 bytes = 699840000 KB = 683437.5 MB = ~667.5 GB

You'll need about 670GB of data storage for 4h of video material (well not that bad - m.2 SSD with 1TB would be enough to store single session).

- get largest aperture lens that will provide you with wanted FOV

For example if you opt for ASI294mm - get fastest 25mm lens that you can afford.

I'm not sure what sort of budget do you have for this - but there are other, more serious sensors out there. If you get IMX455 sensor - you'll have effective pixel size of 15µm and still be able to get 1080p - but that is going to cost as much as 2-3 those consumer Sony cameras.

 

[kylemcdonald]

Thanks again for your advice! We have been running at 24 fps with 1/25" exposure to maximize our light. The 1080p vs 4k distinction is important for the a7S II, but not the a7S III. The a7S II runs better at 1080p because it can do 2x2 pixel binning in hardware before it converts to 8-bit for HDMI. But the a7S III can output ProRes Raw at 4K 12-bit which allows us to either bin or not in post. The data is not a major issue, we record the ProRes Raw with a Ninja V to 2TB SSD drives. We are using a Zhongyi 50mm f/0.95 lens, which we found to be slightly sharper fully open compared to the similarly priced Zenitar 50mm f/0.95 (can't really go larger unless we plan a heist on the Carl Zeiss museum for one of their 50mm f/0.7 lenses). I have looked into the IMX455 sensor but it doesn't meet our framerate requirement (for example, only 10 fps for the QHY600). Our budget is limited to <$5000. I like your ASI294mm proposal, but I am concerned that a full frame sensor still has 3.5x more area than a 4/3" sensor, and also that it won't meet our framerate requirement (only 19fps at full resolution). Do you know if the ASI cameras have increased framerates when there is pixel binning, or is the framerate only a function of the ROI and ADC bit depth?

[vlaiv]

5 minutes ago, kylemcdonald said:

Do you know if the ASI cameras have increased framerates when there is pixel binning, or is the framerate only a function of the ROI and ADC bit depth?

I think that IMX455 should be able to provide you with that frame rate when using binning. I'm not entirely sure and that is something that will need to be checked.

I have found this - which points to that possibility:

http://www.touptek.com/product/showproduct.php?lang=en&id=298

It looks like it is just the matter of controller / firmware for these sensors. As long as data is binned before sending it over USB connection - there should be FPS improvement.

Also - according to this document from Baader - it looks like bin x2 and bin x3 is available for IMX411 as well:

https://www.baader-planetarium.com/en/downloads/dl/file/id/1656/product/4620/technical_data_in_comparsion_between_the_qhy_600_models_and_the_zwo_asi_6200_mm_pro.pdf

Confirmed - here is ASI6200 manual:

https://astronomy-imaging-camera.com/manuals/ASI6200_Manual_EN_v1.4.pdf

You can bin x3 in hardware and then additionally bin x2 in software for example. That will give you 1596 x 1064 which you can adjust for 1080p

Just confirm that hardware bin x3 will give you expected FPS with someone having the camera and USB 3.0 port.

[kylemcdonald]

😱 This is incredibly helpful research. Thank you for this!

It looks like the QHY600-Lite is the winner against the ASI6200, due to the slightly lower read noise, higher quantum efficiency, larger image buffer, and higher framerate. I'm not sure why the QE is higher or read noise is lower, I thought these were a properties of the sensor?

And it looks like both have an SDK, which would allow me to use a low-power device like a Raspberry Pi or Jetson as an "HDMI converter": connect to the camera, buffer the images, process them on the GPU, and output over HDMI to an external recording device. This would all probably double our total wattage, which is currently around 25W (camera, gimbal, recorder). This will create a new bottleneck in terms of power requirements, but it would be worth it if the video is less noisy.

Here's one thing I'm still not totally sure about. I've seen measurements of 0.54e- read noise for the highest gain settings on the a7S, but the QHY600 gives 1.1e- read noise at the highest gain setting in the high-gain mode. How can we compare the "overall sensitivity" in a way that accounts for this? Let's say we have 5 photons per photosite (or binned photosite), and ignore dark current. Can we say the following: a7S III SNR = (5 x 65%) / 0.54 = 6 and QHY600 SNR = (5 x 87%) / 1.1 = 3.9

Does this imply the overall SNR of the a7S III should be around 50% higher? Or have I oversimplified things?

One way I can imagine that I may have oversimplified is that the QHY600 sensor may have a microlens array, while a debayered a7S III does not. This could make a big difference in photon count.

Since it looks like the framerate concern is workable, do you have any intuition about this remaining SNR question?

For some context, here is what a raw frame of video looks like from an a7S III (with bayer filter) at 4K f/0.95 1/25" (shot in high-gain mode), out on the ocean in good conditions (SQM-L was reading 21.4-22.2). (This was 2020-06-13 around Hawaiʻi looking towards the West, but I don't remember the exact time.)

[vlaiv]

7 hours ago, kylemcdonald said:

It looks like the QHY600-Lite is the winner against the ASI6200, due to the slightly lower read noise, higher quantum efficiency, larger image buffer, and higher framerate. I'm not sure why the QE is higher or read noise is lower, I thought these were a properties of the sensor?

I'm rather surprised that they have different specs.

Same sensor - pretty much the same specs. There could be tiny differences in some of these, I'll explain.

First - read noise. There is no single read noise value because gain is variable.

QHY lists read noise in 1e-3.7e range:

(source: https://www.qhyccd.com/qhy600m-c/ )

ZWO lists it at 1.2-3.5e

(source: https://astronomy-imaging-camera.com/product/asi6200mm-pro-mono )

Read noise with CMOS sensors consists out of two parts - pre amp noise and post amp noise. For this reason, read noise value is depending on gain applied, and it usually follows c+1/x curve. We can easily see this if we observe what happens with noise in this conversion between e and ADUs when reading out. Let X be pre amp noise and Y be post amp noise

read noise in ADU is = X * gain + Y

read noise in electrons is then (X * gain + Y) / gain = X + Y/gain

Higher the gain - lower the read noise and vice verse (1/x curve).

IMX455 has multiple read out modes. QHY model lets you select which mode you want to use, while ZWO combines them into one (it chooses mode based on your gain selection). In any case - here are read noise graphs for each:

QHY:

These actually look pretty much the same - it looks like ZWO chose #1 (blue line) mode. Crossover is at different place (ZWO uses db scale for the gain, so each ~61 in their gain e/ADU halves).

In reality - both will have 1.5e read noise.

Sony does not provide absolute QE, but rather gives relative QE chart. Each vendor then estimates peak absolute sensitivity with respect to that chart and their calibration source. Few percent of estimation error is quite possible.

Frame rate depends mostly on USB 3.0 speed and number of pixels that you are going to transfer.

QHY says it is 2.5 FPS

ZWO lists it at 3.19FPS:

Max speed of USB 3.0 port is 4.8Gbit/s and 9576 x 6388 x 16 = 978743808 bits

That in theory gives max of 5.26 FPS (above two divided), however, USB protocol uses up only some percentage on data transfer - and actual speed also depends on speed of rest of hardware, so 2.5-3.2 FPS is realistic.

Overall - I don't think there is any real difference between two cameras as far as performance is concerned.

8 hours ago, kylemcdonald said:

Here's one thing I'm still not totally sure about. I've seen measurements of 0.54e- read noise for the highest gain settings on the a7S, but the QHY600 gives 1.1e- read noise at the highest gain setting in the high-gain mode. How can we compare the "overall sensitivity" in a way that accounts for this? Let's say we have 5 photons per photosite (or binned photosite), and ignore dark current. Can we say the following: a7S III SNR = (5 x 65%) / 0.54 = 6 and QHY600 SNR = (5 x 87%) / 1.1 = 3.9

I'm rather skeptical of that result. I don't see enough in there to be able to make conclusion on it, but two key pieces of information that are missing is e/ADU measurement for each ISO value checked for read noise and read noise measurement methodology.

I've never seen that low read noise. Best planetary camera in use today has read noise of ~0.75e (small sensor - imx224).

In any case, if you want to compare SNR of two cameras, here is how it should work

1. take some surface and set number of photons emitted from unit area of that surface per unit time (flux)

2. calculate how much surface one pixel covers given lens focal length, binning and all of that

3. multiply with average QE - or if you know source spectrum and spectral response of camera - piecewise multiply the two and integrate over spectrum (you can also use peak QE as first approximation)

4. calculate any read noise due to binning - read noise raises as bin factor for CMOS sensors (bin x2 raises read noise x2 and bin x3 raises read noise x3 and so on)

SNR = signal / sqrt( signal + total_read_noise^2)

In your example if we calculate per pixel SNR for same size pixel and same lens and per pixel area signal is 5e then respective SNRs will be

a7S III SNR = (5 * 0.65) / sqrt( (5 * 0.65) + 0.54^2) = ~1.727

QHY SNR = (5*0.87) / sqrt( (5*0.87) + 1.1^2) = ~1.845

Just to explain above formula - total noise is square root of sum of squares of all individual noise sources. In above calculation, we used only shot noise and read noise. We assumed that exposure is short enough so that dark current is effectively zero and that there is no background illumination (no light pollution). Second assumption might not be valid one - you'll need to check that. In astronomy, even SQM 22 sky is much brighter than targets and must be taken into account - fainter parts of brighter galaxies go down to SQM27-28 for example.

Shot noise is equal to square root of signal.

8 hours ago, kylemcdonald said:

For some context, here is what a raw frame of video looks like from an a7S III (with bayer filter) at 4K f/0.95 1/25" (shot in high-gain mode), out on the ocean in good conditions (SQM-L was reading 21.4-22.2). (This was 2020-06-13 around Hawaiʻi looking towards the West, but I don't remember the exact time.)

Out of interest, do you have single such frame in full resolution in raw format - say fits?

Did you calibrate that frame? It might be worth checking if you need to do bias removal and will it improve SNR.

[ONIKKINEN]

2 hours ago, vlaiv said:

I've never seen that low read noise. Best planetary camera in use today has read noise of ~0.75e (small sensor - imx224).

This is my Rising Cam ATR3CMOS26000KPA, or more commonly known as the Rising Cam IMX571 colour camera. Measured with sharpcap pro sensor analysis.

I dont know what ToupTek (manufacturer for RisingCam products) does differently with their electronics compared to ZWO or QHY offerings but this is how it is with this particular camera. Could be an option for OP if the absolute lowest read noise is a must have with a decent sized sensor, this one being APS-C. Framerates are low, like with all IMX571 sensors so shooting in 8-bit mode might be needed.

Of course if shooting in 8-bit mode you lose the low read noise in low gain values, so not sure if worth it. Below is the same analysis but in 8-bit mode.

[vlaiv]

3 hours ago, ONIKKINEN said:

This is my Rising Cam ATR3CMOS26000KPA, or more commonly known as the Rising Cam IMX571 colour camera. Measured with sharpcap pro sensor analysis.

How does sharpcap measure e/ADU displayed in the table?

[ONIKKINEN]

33 minutes ago, vlaiv said:

How does sharpcap measure e/ADU displayed in the table?

No clue, could be magic for all i know.

The measurement process itself goes like this: Constant illumination on the sensor, i used a flat panel on my telescope. Then sharpcap chooses a small area on the sensor to do the measurements on and goes through all the gain values while taking exposures and measuring something? Really dont know how this works.

The Rising Cam aliexpress store also reports similar values: https://www.aliexpress.com/item/4001359313736.html?spm=a2g0o.productlist.0.0.6f047164JGhOx6&algo_pvid=88c7fc7f-59b2-4b58-9bdc-a75b08237944&algo_exp_id=88c7fc7f-59b2-4b58-9bdc-a75b08237944-0

[vlaiv]

4 minutes ago, ONIKKINEN said:

e measurement process itself goes like this: Constant illumination on the sensor, i used a flat panel on my telescope.

well - this part is important.

That is how you measure e/ADU.

There is relationship between noise and electron counts - noise is square root of electron count. This however does not hold for ADUs

Say you have e/ADU of 0.25 and you receive 100e per pixel - then standard deviation of such frame will be 10e (this is simplification if there is no read noise or dark current and so on, but these can be accounted for in real measurement).

If e/ADU is 0.25 - then you will record 400 ADU but standard deviation will be 40ADU (10 / 0.25)  instead of expected sqrt(400) = 20. Signal linearly depends on gain - but noise in quadratic fashion.

Square root of ratio of the two is e/ADU so 40 = 2x20 -> 1/2^2 = 0.25

From measured standard deviation and measured average ADU - you can get e/ADU. This is best done on several different exposure lengths and one must account for read noise and dark current (need to be subtracted properly), but that is the principle.

I was just wondering if sharpcap did actual measurement or relied on driver reported value.

[ONIKKINEN]

2 minutes ago, vlaiv said:

I was just wondering if sharpcap did actual measurement or relied on driver reported value.

The process takes around 90 minutes with several steps in between sometimes covering the camera for dark exposures and other times re-introducing the light panel so i am pretty sure it does go through various exposure lengths and does some calibration of its own too.

[kylemcdonald]

Thank you both for all your insight. I'm learning a lot 🙏 

vlaiv, that image was not calibrated. When we initially recorded these I zoomed in on a sequence of a few consecutive frames and felt like the noise looked non-fixed, so I (naively) assumed that bias/dark calibration might not make a big difference. Also, watching videos at full resolution does not seem to reveal any strong fixed noise patterns. But at your prompting, I just read an article about the topic and I think I'm going to return to this. I still have the camera and recorder, so I will do some short cap-on recordings and build a dark and bias frame.

Regarding the low read noise, I have also seen this same number from PhotonsToPhotos: 0.54e to 0.57e in the 102400 to 409600 ISO range.

What's interesting to me about the equations you just shared is that somewhere around 2.3e input signal they converge at a SNR 1.1. Lower than that the a7S would have a slight advantage with the low read noise, higher than that the QHY600 has the advantage due to the QE. So this is really a head-to-head match.

What's unclear to me is if the QE of the a7S accounts for the bayer filter or not. In other words, if we remove the bayer filter and shoot the same photons at an a7S II or III, will our QE measurement increase? Because if that is the case, and the QE without a bayer filter is closer to 80%, then that convergence point is instead around 11e.

It sounds like my next steps are to keep pushing on the a7S somehow. Because we are operating with video in very low light, we are primarily in this read noise-constrained zone. If I can't get anywhere, then the QHY600 or ASI6200 both seem to be a good alternatives—assuming I can get the framerate up, and I can find one in stock It's just going to take a bit more programming and experimentation with the power.

Thanks again for all the help 🙏

[kylemcdonald]

I got a reply directly from QHY indicating that the framerate of the QHY600 only increases with ROI, and not with binning. So unless the customer service rep is confused, that rules out their hardware.

I am waiting for a reply from ZWO on their forum.

[chrisdelroy]

On 01/02/2022 at 14:38, kylemcdonald said:

Hello all, I’m helping with a cultural heritage project that requires an extremely low light video camera. We’re trying to capture a well-documented but never-photographed luminous phenomena that happens on the ocean at night. By following the posts on this forum and others, I learned about mono full-spectrum conversions. But my journey has been rough, and I wanted to ask for guidance from the experts

I got lucky with my first debayered camera: I found a Sony a7S II on eBay that had already been modified by Monochrome Imaging. It worked great for a few months. Check out this test video I recorded showing glint from Jupiter on the Pacific, just off Venice Beach in Los Angeles.

But two weeks before we set sail, the camera stopped working. It would go into a restart-loop when I turned it on. One repair center told me it was a problem with the sensor, but Monochrome Imaging offered to take a look (they have been amazing) under the suspicion that it was the main board. We just found out it is not the main board.

Right after this glitch appeared, I contacted LDP LLC (MaxMax) in hopes of converting an a7S III to mono full-spectrum. LDP has also been amazing, they took the rush job, shot some beautiful test images, and shipped it to me. But when I unboxed and turned it on, the display briefly showed some blocky artifacts, and then only a black image. I was devastated. Now I am looking for another a7S III sensor in hopes that LDP can try conversion again, but I’m concerned that we don’t really know what caused the failure.

So I’m here to ask:

1. Does anyone have experience with successful a7S II or a7S III mono conversions?
2. Where might I be able to find a reasonably priced a7S III sensor? The best I've found so far is $1600 on AliExpress.
3. Is there anyone else besides Monochrome Imaging or LDP, preferably in the US, who have near 100% success rates? Or am I hoping for too much here? I know there are different processes (chemicals, scraping, laser) but I’ve had trouble figuring out who else is still operating.

I also wanted to share some code I worked on as part of this project. I made a tool for extracting the raw 4K high-bit-depth pre-bayer-mixed image from an a7S III. If you stream from the a7S III to a device that can capture the ProRes Raw encoded video, like a Ninja V, it’s possible to get 4K grayscale video at 30fps.

Hi Kyle, 

 

I have a sony a7riii currently being converted by Monochrome imaging and really would love to know if there is a way to get the video footage out of it in pre debayered form to get low light astro video that's better than my a1 converted digitally. Would love to know what you were able to do in this regards and how you did the sii footage. Thanks!