How the Samsung Galaxy S20 Ultra takes great photos with Nona Binning

The smartphone camera megapixel wars slowed to a halt for several years but picked up steam again in 2019. The arrival in 2018 of the Huawei P20 Pro (review) and the Huawei Mate 20 Pro (review) with huge 40MP Quad Bayer sensors proved that high megapixel sensors weren’t necessarily a detriment to low light photography. Thanks to nifty 4-in-1 pixel binning and a powerful Night Mode, Huawei’s 2018 flagship smartphones were able to comfortably be in the top-tier in low-light photography, matching and in some cases even beating the best that Google’s HDR+ and Night Sight had to offer. In 2019, taking inspiration from Huawei, high megapixel Quad Bayer sensors arrived in the mainstream with most major smartphone vendors using some form of the 48MP Sony IMX586, 48MP Samsung GM1 and GM2, the 64MP Samsung ISOCELL GW1, and finally, the latest 64MP Sony IMX686. Xiaomi took things up a notch with the launch of the Mi Note 10, which featured a 108MP Samsung ISOCELL HMX primary sensor. The newly launched Samsung Galaxy S20 Ultra is the second phone on the market to use a 108MP primary camera (not counting the Mi Mix Alpha concept phone), but its workflow is different. Let’s get into the details.

The Galaxy S20 Ultra is the flagship phone in Samsung’s Galaxy S20 series, and it has the specifications and pricing to match. While the regular Galaxy S20 and the Galaxy S20+ use a new 12MP primary sensor with 1.8-micron pixel size, the Galaxy S20 Ultra moves up to a 108MP primary sensor. This isn’t the same sensor as the Mi Note 10, though. The Galaxy S20 Ultra is using an ISOCELL HM1 sensor, which is a slight improvement over the ISOCELL HMX used in the Mi Note 10. It’s capable of 8K@24fps video recording, for one. More interestingly, it steps away from the 4-in-1 pixel binning used in the Mi Note 10 to a new 9-in-1 pixel binning that produces 12MP photos. Samsung calls this nona binning, and it’s the first phone to use a 3×3 pixel binning rather than a 2×2 pixel binning. The Galaxy S20 Ultra can dynamically shift between a high-resolution 108MP mode and a nona pixel binned 12MP mode.

To understand why this is beneficial, let’s take a look at how pixel binning works.

Pixel binning on high megapixel Quad Bayer sensors

Via: AnandTech

Before 2018, essentially every smartphone camera had a Bayer color filter array. The Huawei P20 Pro and the Huawei Mate 20 Pro were the first phones to use Quad Bayer sensors. Simply put, a Quad Bayer sensor has less color resolution than a sensor with a standard Bayer layout.  On the IMX586, for example, the physical color filters on the camera sensor only have an effective resolution of 12MP. The ISP of such sensors is able to achieve a virtual 48MP Bayer result out of the sensor by re-arranging the subpixels among the logical pixels. It should be clear that this approach isn’t as good as using a standard Bayer filter. What is the specific difference? According to AnandTech, the 48MP IMX586 has closer to 27MP of spatial resolution as it’s only able to increase spatial resolution half-way.

Why use Quad Bayer sensors then? They make sense because up until now, they are the only option on the market if a device maker wants to incorporate a high megapixel sensor. Their negatives are mitigated by using 4-in-1 pixel binning, which combines four pixels into one pixel at the sensor level to reduce noise, improve dynamic range, and improve per-pixel detail. Combining multiple pixels into one isn’t a new idea; it was done with the Nokia 808 PureView and the Nokia Lumia 1020 back in 2012 and 2013. Before, the idea was that multiple noisy pixels will be combined into one clean “super” pixel. The advent of Quad Bayer sensors led to the popularization of pixel binning.

The theory behind pixel binning is this. Huawei’s custom 40MP sensor on its flagships has a 1.0-micron pixel size. The mainstream 48MP and 64MP sensors have an even smaller pixel size at 0.8 micron. Even the 108MP ISOCELL sensors have the same 0.8-micron pixel size because the pixel size is kept constant when the resolution is increased simultaneously with the sensor size. When the pixel size is lowered, low light image quality is adversely affected. Low light photos are highly affected by a pixel’s sensitivity to light, and therefore, the rule of thumb is to go with a bigger pixel size, with flagship 12MP sensors settling at around a 1.4-micron pixel size.

The Quad Bayer sensors theoretically circumvent this limitation by going with pixel binning. Huawei’s 4-in-1 pixel binning results in 10MP photos by default at an “effective pixel size” of 2.0 micron. The 48MP and 64MP sensors use 4-in-1 pixel binning for 12MP and 16MP photos by default at an effective pixel size of 1.6 micron. The Xiaomi Mi Note 10 uses 4-in-1 pixel binning for 27MP photos by default at an effective pixel size of 1.6 micron. This lets them achieve competitive results in low light. Of course, smartphone photography is a complicated field and image processing is an equally important factor, but all other factors being constant, an effective pixel size of 1.6 micron should be better than a pixel size of 1.4 micron. Some device makers were more successful than others in the implementation of such high megapixel Quad Bayer sensors. Samsung, Huawei, and Apple opted to stay with their custom sensors, while Google continued to use the relatively old Sony IMX363 in the Google Pixel 4, making up for it with computational photography in the form of improved HDR+ and Night Sight.

Thus, we now have a market where even lower mid-range phones have 48MP and 64MP Quad Bayer sensors, as they essentially have few real disadvantages over traditional 12MP or 16MP sensors. Using 4-in-1 pixel binning, the final photos will be at 12MP resolution in most lighting conditions anyway, especially in low light. The benefit of the high resolution comes in daylight, where some phones do offer 48MP modes to take advantage of the increased spatial resolution.

The Galaxy S20 Ultra’s “nona binning”

The Galaxy S20 Ultra is the logical progression of the pixel binning methodology. Up until now, phones using high megapixel sensors have used 2×2 binning to combine four pixels into one. This lets the effective pixel size be doubled from 0.8 micron to 1.6 micron, for example. With a 108MP sensor, though, there is potential for binning of even more pixels at once. 4-in-1 pixel binning will result in a 27MP pixel binned mode, which is still arguably too large for general photos. Also, 27MP photos at an effective 1.6 micron pixel size are good, but what if you could have 12MP photos at an effective pixel size of 2.4 micron?

Samsung achieves an effective pixel size of 2.4 micron by using 3×3 pixel binning, combining nine pixels into a single pixel at the sensor level. “Nona binning” is nothing but the company’s marketing term for 3×3 binning. This will combine nine relatively noisy pixels into one big and clean superpixel, which would theoretically further enhance low light sensitivity.  A 2.4 micron effective pixel size is unheard of in smartphone cameras. The only parallel is OmniVison’s 48MP OV48C image sensor, which launched at CES. It outputs standard Bayer photos and has a 1.2 micron pixel size that uses near-pixel binning to achieve 12MP photos with a 2.4 micron pixel size. While it has great specifications, it hasn’t made its way to any shipping phone yet.

Samsung’s approach, therefore, is theoretically the best of both worlds. The phone will dynamically shift between a high-resolution 108MP mode and a 12MP nona pixel binned mode, according to the company. In daylight, it should be able to take ultra-high-resolution 108MP photos to capture far-off details and for more effective cropping, if the image processing is up to scratch. In low light, it will dynamically shift to a 12MP mode, using 9-in-1 pixel binning. Low light photography is affected far more by the pixel size than resolution. On a theoretical level, 3×3 pixel binning in low light makes much more sense than 2×2 binning, even as the resolution is dropped from 27MP (2×2 binning) to 12MP (3×3 binning).

Thus, the 108MP sensor of the Galaxy S20 Ultra doesn’t have any weaknesses on paper. In fact, it should turn out to be a new image quality champion in daylight as well as in low light. While we reserve judgment until we have tested the phone ourselves, Samsung has made the right decisions on paper. I personally have no doubt that if the execution is carried out well, more vendors using 108MP sensors will migrate to 9-in-1 pixel binning from 4-in-1 binning to achieve better photos in low light. The Galaxy S20 Ultra is definitely an interesting phone, and it makes us hopeful that smartphone camera performance and image quality will take another step forward this year.

Samsung Galaxy S20 Forums ||| Samsung Galaxy S20+ Forums ||| Samsung Galaxy S20 Ultra Forums

The post How the Samsung Galaxy S20 Ultra takes great photos with Nona Binning appeared first on xda-developers.

from xda-developers https://ift.tt/2He10wv
via IFTTT