Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

malloc issue into create tiff image from raw image #1829

Closed
laurechr opened this issue Sep 10, 2019 · 5 comments
Closed

malloc issue into create tiff image from raw image #1829

laurechr opened this issue Sep 10, 2019 · 5 comments
Labels
awaiting_feedback Awaiting feedback from reporter

Comments

@laurechr
Copy link

Expected behavior and actual behavior.

i use gdal to transform image in roipac format to gtiff or png format:
the size of images are WIDTH 2825 FILE_LENGTH 2155 on float64
i launch my program on hpc clusteur (in this case with 8 core)
this program use a subroutine that i describe below which transform image from RoiPac format to png format
I show the function :
def mk_tiff_file(lfilesrc, dirsrc, fileima, dirdst, typeima, bandtiff,
sizeima=None):
""" transform a raw file on tiff format
:lfilesrc: list of src file
:dirsrc: the directory of src file
:fileima: the name of dst file, we add the extension .tiff
:dirdst: the directory of ima file
:typeima: type from def_ima={'InW_radar','InW_geo','InU_radar',
'InU_geo','DTs_radar','DTs_geo'}
:bands: list of the bands we use to transform
exemples
band=['phase']
or
band=['phase','ampli']
:sizeima: return the ima size if the initial value is
different of None
"""

# Open dataset (image)
filesrc = lfilesrc[0]
dsIn = gdal.Open(os.path.join(dirsrc, filesrc), gdal.GA_ReadOnly)
colsIn = dsIn.RasterXSize
rowsIn = dsIn.RasterYSize
size = colsIn*rowsIn
projection = dsIn.GetProjection()
geotransform = dsIn.GetGeoTransform()
driverdst = gdal.GetDriverByName('GTiff')
filedst = fileima + ".tiff"
nbband = len(bandtiff)

logger.info("create tiff file %s type %s  nb bands: %d",
            filedst, classima, nbband)
dsOut = driverdst.Create(os.path.join(dirdst, filedst), colsIn, rowsIn,
                         nbband, gdal.GDT_Float32,
                         options=['PROFILE=GeoTIFF', ])
if geotransform is not None and geotransform != (0, 1, 0, 0, 0, 1):
    dsOut.SetGeoTransform(geotransform)
if projection is not None and projection != '':
    dsOut.SetProjection(projection)
bandout = []
for i in range(nbband):
    index = i+1
    # logger.info("index: %d %d", i, index)
    bandout.append(dsOut.GetRasterBand(index))

for i in range(nbband):
    if bandtiff[i] == 'phase':
        dataPhase = np.ascontiguousarray(np.array(dsIn.GetRasterBand(2).ReadAsArray()))
    elif bandtiff[i] == 'ampli':
        dataAmpli = np.ascontiguousarray(np.array(dsIn.GetRasterBand(1).ReadAsArray()))
        
# write bands
for i in range(nbband):
    if bandtiff[i] == 'phase':
        bandout[i].WriteArray(dataPhase)
    if bandtiff[i] == 'ampli':
        bandout[i].WriteArray(dataAmpli)
    bandout[i].FlushCache()
    
del dsIn
del dsOut
del dataAmpli
del dataPhase
del bandout
if sizeima is not None:
    return(size)

issue

in the log file:
2019-09-10 12:03:22,603 - main - INFO - mk_image_file : 20160330-20160423_mdel_8rlks.unw
2019-09-10 12:03:22,903 - main - INFO - mk_image_file tiff: CNES_20150429-20150710_mdel_8rlks.unw.tiff
2019-09-10 12:03:22,905 - main - INFO - mk_image_file png: CNES_20150429-20150710_mdel_8rlks.unw.png
2019-09-10 12:03:22,914 - main - INFO - mk_image_file : 20150429-20150827_mdel_8rlks.unw
2019-09-10 12:03:23,087 - main - INFO - mk_image_file tiff: CNES_20150827-20150920_mdel_8rlks.unw.tiff
2019-09-10 12:03:23,088 - main - INFO - mk_image_file png: CNES_20150827-20150920_mdel_8rlks.unw.png
2019-09-10 12:03:23,250 - main - INFO - mk_image_file tiff: CNES_20141007-20150111_mdel_8rlks.unw.tiff
2019-09-10 12:03:23,256 - main - INFO - mk_image_file png: CNES_20141007-20150111_mdel_8rlks.unw.png
2019-09-10 12:03:23,260 - main - INFO - mk_image_file : 20141007-20150204_mdel_8rlks.unw
2019-09-10 12:03:23,313 - main - INFO - mk_image_file : 20150827-20151014_mdel_8rlks.unw
2019-09-10 12:03:23,437 - main - INFO - mk_image_file tiff: CNES_20150312-20150405_mdel_8rlks.unw.tiff
2019-09-10 12:03:23,439 - main - INFO - mk_image_file png: CNES_20150312-20150405_mdel_8rlks.unw.png
2019-09-10 12:03:23,446 - main - INFO - mk_image_file : 20150312-20150429_mdel_8rlks.unw
2019-09-10 12:03:24,002 - main - INFO - mk_image_file tiff: CNES_20141218-20150111_mdel_8rlks.unw.tiff
2019-09-10 12:03:24,002 - main - INFO - mk_image_file png: CNES_20141218-20150111_mdel_8rlks.unw.png
2019-09-10 12:03:24,011 - main - INFO - mk_image_file : 20141218-20150204_mdel_8rlks.unw
2019-09-10 12:03:24,127 - main - INFO - mk_image_file tiff: CNES_20160423-20160728_mdel_8rlks.unw.tiff
2019-09-10 12:03:24,129 - main - INFO - mk_image_file png: CNES_20160423-20160728_mdel_8rlks.unw.png
2019-09-10 12:03:24,133 - main - INFO - mk_image_file : 20160423-20160821_mdel_8rlks.unw
2019-09-10 12:03:24,213 - main - INFO - mk_image_file tiff: CNES_20160211-20160330_mdel_8rlks.unw.tiff
2019-09-10 12:03:24,214 - main - INFO - mk_image_file png: CNES_20160211-20160330_mdel_8rlks.unw.png
2019-09-10 12:03:24,221 - main - INFO - mk_image_file : 20160211-20160423_mdel_8rlks.unw
2019-09-10 12:03:24,225 - main - INFO - mk_image_file tiff: CNES_20151201-20151225_mdel_8rlks.unw.tiff
2019-09-10 12:03:24,225 - main - INFO - mk_image_file png: CNES_20151201-20151225_mdel_8rlks.unw.png
2019-09-10 12:03:24,261 - main - INFO - mk_image_file tiff: CNES_20151225-20160306_mdel_8rlks.unw.tiff
2019-09-10 12:03:24,262 - main - INFO - mk_image_file png: CNES_20151225-20160306_mdel_8rlks.unw.png
2019-09-10 12:03:24,264 - main - INFO - mk_image_file : 20151225-20161201_mdel_8rlks.unw
2019-09-10 12:03:24,342 - main - INFO - Starting PoolWorker-13
2019-09-10 12:03:24,344 - main - INFO - mk_image_file : 20160517-20160821_mdel_8rlks.unw
2019-09-10 12:03:27,623 - main - INFO - mk_image_file tiff: CNES_20150920-20160118_mdel_8rlks.unw.tiff
2019-09-10 12:03:27,625 - main - INFO - mk_image_file png: CNES_20150920-20160118_mdel_8rlks.unw.png
2019-09-10 12:03:27,634 - main - INFO - mk_image_file : 20151014-20151201_mdel_8rlks.unw

there are some messages in log file but there is no crashes

malloc_consolidate(): invalid chunk size
free(): invalid next size (fast)
free(): invalid pointer
double free or corruption (!prev)
corrupted size vs. prev_size

and the program is stop

Steps to reproduce the problem.

try this function in lot of images

Operating system

Linux luke 3.16.0-5-amd64 #1 SMP Debian 3.16.51-3+deb8u1 (2018-01-08) x86_64
python 2.7

GDAL version and provenance

gdal-2.2.4
@laurechr
Copy link
Author

i precise the problem shows after we transform 251 image from 290 initials images (35 Mo size of one image)

@rouault rouault added the awaiting_feedback Awaiting feedback from reporter label Sep 12, 2019
@rouault
Copy link
Member

rouault commented Sep 12, 2019 •
edited
Loading

I'm afraid we can't do anything with this issue as it. You should determine if it is a memory saturation issue (monitor memory consumption), or if the issue arises on a particular image. And if so, please provide it.

@laurechr
Copy link
Author

laurechr commented Sep 26, 2019 via email

@rouault
Copy link
Member

rouault commented Sep 26, 2019

can you give some help to solve the malloc issue

I'm afraid it is not possible with just what you provided. You need to make sure you don't have too many processes running on the same machine. Try to strip down your code to the minimum amount sufficient to reproduce the error(s).

@rouault
Copy link
Member

rouault commented Nov 3, 2019

Closing this issue, as it is not actionable on our side

@rouault rouault closed this as completed Nov 3, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
awaiting_feedback Awaiting feedback from reporter
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rouault @laurechr