Image too big for processing when converting large (1.3 GB) DNG file to PNG using python

1.1k views Asked by At

I need to convert a DNG file to PNG using python. I found a post here how to convert DNG: Opencv Python open dng format

The code I tried:

#open dng and convert
import rawpy
import imageio
import os

os.chdir(r'C:\Path\to\dir')
path = r'path\to\file'

with rawpy.imread(path) as raw:
    rgb = raw.postprocess()

rgb_img.save('image.png')

However, it spits out the following errors:

line 13, in <module>
    rgb = raw.postprocess()
  File "rawpy\_rawpy.pyx", line 887, in rawpy._rawpy.RawPy.postprocess
  File "rawpy\_rawpy.pyx", line 790, in rawpy._rawpy.RawPy.dcraw_process
  File "rawpy\_rawpy.pyx", line 438, in rawpy._rawpy.RawPy.ensure_unpack
  File "rawpy\_rawpy.pyx", line 432, in rawpy._rawpy.RawPy.unpack
  File "rawpy\_rawpy.pyx", line 936, in rawpy._rawpy.RawPy.handle_error
rawpy._rawpy.LibRawTooBigError: b'Image too big for processing'

Is there either an alternative to convert the DNG files, or a way to bypass this error?

Info from exiftool:

ExifTool Version Number         : 11.88

File Name                       : DSCF0001.DNG

Directory                       : .

File Size                       : 1313 MB

File Modification Date/Time     : 2022:08:24 12:06:31+01:00

File Access Date/Time           : 2022:08:25 08:17:15+01:00

File Inode Change Date/Time     : 2022:08:24 15:19:04+01:00

File Permissions                : rwxrwxrwx

File Type                       : DNG

File Type Extension             : dng

MIME Type                       : image/x-adobe-dng

Exif Byte Order                 : Little-endian (Intel, II)

Make                            : FUJIFILM

Camera Model Name               : GFX 100

Preview Image Start             : 115208860

Orientation                     : Horizontal (normal)

Rows Per Strip                  : 3000

Preview Image Length            : 3011337

Software                        : FUJIFILM Pixel Shift Combiner 1.2.0.2 (Real Color + High Resolution mode)

Modify Date                     : 2022:08:24 12:06:29

Artist                          : 

Subfile Type                    : Full-resolution image

Image Width                     : 23296

Image Height                    : 17472

Bits Per Sample                 : 16 16 16

Compression                     : JPEG

Photometric Interpretation      : Linear Raw

Samples Per Pixel               : 3

Planar Configuration            : Chunky

Tile Width                      : 128

Tile Length                     : 96

Tile Offsets                    : (Binary data 341026 bytes, use -b option to extract)

Tile Byte Counts                : (Binary data 198743 bytes, use -b option to extract)

Black Level                     : 256 255 256

White Level                     : 65535 65535 65535

Default Scale                   : 1 1

Default Crop Origin             : 16 12

Default Crop Size               : 23264 17448

Anti Alias Strength             : 1

Best Quality Scale              : 1

Opcode List 3                   : WarpRectilinear, FixVignetteRadial

Rating                          : 0

Copyright                       : 

Exposure Time                   : 1/125

F Number                        : 8.0

Exposure Program                : Manual

ISO                             : 100

Sensitivity Type                : Standard Output Sensitivity

Standard Output Sensitivity     : 100

Exif Version                    : 0230

Date/Time Original              : 2019:03:10 00:44:16

Create Date                     : 2019:03:10 00:44:16

Shutter Speed Value             : 1/125

Aperture Value                  : 8.0

Brightness Value                : 8.57

Exposure Compensation           : 0

Max Aperture Value              : 2.0

Metering Mode                   : Multi-segment

Light Source                    : Unknown

Flash                           : No Flash

Focal Length                    : 110.0 mm

Version                         : 0130

Internal Serial Number          : 

Quality                         : NORMAL

White Balance                   : Auto

Saturation                      : 0 (normal)

White Balance Fine Tune         : Red +0, Blue +0

Noise Reduction                 : 0 (normal)

Fuji Flash Mode                 : Manual

Flash Exposure Comp             : 0

Focus Mode                      : Manual

AF Mode                         : No

Focus Pixel                     : 2001 1501

AF-S Priority                   : Release

AF-C Priority                   : Release

Focus Mode 2                    : AF-M

AF Area Mode                    : Single Point

AF Area Point Size              : n/a

AF Area Zone Size               : n/a

AF-C Setting                    : Set 1 (multi-purpose)

AF-C Tracking Sensitivity       : 2

AF-C Speed Tracking Sensitivity : 0

AF-C Zone Area Switching        : Auto

Slow Sync                       : Off

Picture Mode                    : Manual

Exposure Count                  : 1

Shadow Tone                     : 0 (normal)

Highlight Tone                  : 0 (normal)

Lens Modulation Optimizer       : On

Grain Effect                    : Off

Color Chrome Effect             : Off

Crop Mode                       : n/a

Color Chrome FX Blue            : Off

Shutter Type                    : Electronic

Auto Bracketing                 : Unknown (6)

Sequence Number                 : 1

Drive Mode                      : Single

Drive Speed                     : n/a

Blur Warning                    : None

Focus Warning                   : Good

Exposure Warning                : Good

Dynamic Range                   : Standard

Film Mode                       : F0/Standard (Provia)

Dynamic Range Setting           : Manual

Development Dynamic Range       : 100

Min Focal Length                : 110

Max Focal Length                : 110

Max Aperture At Min Focal       : 2

Max Aperture At Max Focal       : 2

Image Stabilization             : Sensor-shift; Off; 0

Image Generation                : Original Image

Image Count                     : 34

Flicker Reduction               : Off (0x0002)

Faces Detected                  : 0

Num Face Elements               : 0

Color Space                     : Uncalibrated

Focal Plane X Resolution        : 5320

Focal Plane Y Resolution        : 5320

Focal Plane Resolution Unit     : cm

File Source                     : Digital Camera

Scene Type                      : Directly photographed

Custom Rendered                 : Normal

Exposure Mode                   : Auto

Focal Length In 35mm Format     : 87 mm

Scene Capture Type              : Standard

Sharpness                       : Unknown (3)

Subject Distance Range          : Unknown (48)

Serial Number                   : 

Lens Info                       : 110mm f/2

Lens Make                       : FUJIFILM

Lens Model                      : GF110mmF2 R LM WR

Lens Serial Number              : 

DNG Version                     : 1.4.0.0

DNG Backward Version            : 1.1.0.0

Unique Camera Model             : FUJIFILM GFX 100

Color Matrix 1                  : 1.7191 -1.1 0.1278 -0.3574 1.1733 0.2076 -0.0002 0.0497 0.654

Color Matrix 2                  : 1.6212 -0.8423 -0.1583 -0.4336 1.2583 0.1937 -0.0195 0.0726 0.6199

Analog Balance                  : 1 1 1

As Shot Neutral                 : 0.5644 1 0.5153

Baseline Exposure               : -0.01

Baseline Noise                  : 1

Baseline Sharpness              : 1.33

Linear Response Limit           : 1

Camera Serial Number            : 

DNG Lens Info                   : 110mm f/2

Shadow Scale                    : 1

DNG Private Data                : (Binary data 114927728 bytes, use -b option to extract)

Calibration Illuminant 1        : Standard Light A

Calibration Illuminant 2        : D65

Aperture                        : 8.0

Image Size                      : 23296x17472

Megapixels                      : 407.0

Preview Image                   : (Binary data 3011337 bytes, use -b option to extract)

Scale Factor To 35 mm Equivalent: 0.8

Shutter Speed                   : 1/125

Circle Of Confusion             : 0.038 mm

Field Of View                   : 23.4 deg

Focal Length                    : 110.0 mm (35 mm equivalent: 87.0 mm)

Hyperfocal Distance             : 39.81 m

Light Value                     : 13.0
1

There are 1 answers

5
Mark Setchell On BEST ANSWER

There seems to be a limit of 2GB for the fully expanded in memory image. I don't mean the space your DNG requires on disk, I mean the following number:

ImageHeight * ImageWidth * NumberOfChannels * BytesPerSample

So it would be useful if you used exiftool to tell us those parameters, by clicking edit under your question and adding the output from:

exiftool YOURIMAGE.DNG

The limit is enforced in the variable imgdata.params.max_raw_memory_mb which is set to 2048 in the underlying libraw C code. I am not immediately sure how you could set that through the Python interface.


You may have some success converting it to a PNG or a TIFF with ImageMagick as follows:

magick YOURIMAGE.DNG converted.png

If that works, you could use wand which is a Python binding to Imagemgaick.


You could try with ufraw maybe, along the lines of:

ufraw-batch --out-type=png --out-depth=16 --output=result.png YOURIMAGE.dng