Out of memory trying to load custom level

[vi]

03-29 00:52:24.342  9734  9796 I erwylo.beatgam: Alloc concurrent copying GC freed 8(31KB) AllocSpace objects, 0(0B) LOS objects, 41% free, 67MB/115MB, paused 25us total 9.235ms
03-29 00:52:24.342  9734  9796 I erwylo.beatgam: Forcing collection of SoftReferences for 128MB allocation
03-29 00:52:24.342  9734  9796 I erwylo.beatgam: Starting a blocking GC Alloc
03-29 00:52:24.350  9734  9796 I erwylo.beatgam: Alloc concurrent copying GC freed 5(16KB) AllocSpace objects, 0(0B) LOS objects, 41% free, 67MB/115MB, paused 17us total 7.674ms
03-29 00:52:24.350  9734  9796 W erwylo.beatgam: Throwing OutOfMemoryError "Failed to allocate a 134217744 byte allocation with 50331648 free bytes and 124MB until OOM, target footprint 120694216, growth limit 201326592" (VmSize 13863200 kB)
03-29 00:52:24.351  9734  9796 E AndroidRuntime: FATAL EXCEPTION: Thread-3
03-29 00:52:24.351  9734  9796 E AndroidRuntime: Process: com.serwylo.beatgame, PID: 9734
03-29 00:52:24.351  9734  9796 E AndroidRuntime: java.lang.OutOfMemoryError: Failed to allocate a 134217744 byte allocation with 50331648 free bytes and 124MB until OOM, target footprint 120694216, growth limit 201326592
03-29 00:52:24.351  9734  9796 E AndroidRuntime:        at java.util.Arrays.copyOf(Arrays.java:3161)
03-29 00:52:24.351  9734  9796 E AndroidRuntime:        at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
03-29 00:52:24.351  9734  9796 E AndroidRuntime:        at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
03-29 00:52:24.351  9734  9796 E AndroidRuntime:        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
03-29 00:52:24.351  9734  9796 E AndroidRuntime:        at com.serwylo.beatgame.audio.fft.FFTKt.readPcm(FFT.kt:225)
03-29 00:52:24.351  9734  9796 E AndroidRuntime:        at com.serwylo.beatgame.audio.fft.FFTKt.calculateMp3FFTWithValues(FFT.kt:39)
03-29 00:52:24.351  9734  9796 E AndroidRuntime:        at com.serwylo.beatgame.audio.AudioIOKt.loadFromDisk(AudioIO.kt:46)
03-29 00:52:24.351  9734  9796 E AndroidRuntime:        at com.serwylo.beatgame.audio.AudioIOKt.loadWorldFromMp3(AudioIO.kt:31)
03-29 00:52:24.351  9734  9796 E AndroidRuntime:        at com.serwylo.beatgame.screens.LoadingScreen$startLoading$1.run(LoadingScreen.kt:103)
03-29 00:52:24.351  9734  9796 E AndroidRuntime:        at java.lang.Thread.run(Thread.java:923)
03-29 00:52:24.353  1423 11544 W ActivityTaskManager:   Force finishing activity com.serwylo.beatgame/.AndroidLauncher```

[vi]

If I trim the file to 300 first seconds, it goes further in loading the level:

03-29 01:06:28.411 10249 10452 W erwylo.beatgam: "Thread-7" prio=5 tid=25 Runnable
03-29 01:06:28.411 10249 10452 W erwylo.beatgam:   | group="main" sCount=0 dsCount=0 flags=2 obj=0x12c41ab0 self=0x7d3c6c6c00
03-29 01:06:28.411 10249 10452 W erwylo.beatgam:   | sysTid=10452 nice=0 cgrp=default sched=0/0 handle=0x7bbb301cc0
03-29 01:06:28.411 10249 10452 W erwylo.beatgam:   | state=R schedstat=( 21122716253 7299690 888 ) utm=1946 stm=165 core=6 HZ=100
03-29 01:06:28.411 10249 10452 W erwylo.beatgam:   | stack=0x7bbb1fe000-0x7bbb200000 stackSize=1043KB
03-29 01:06:28.411 10249 10452 W erwylo.beatgam:   | held mutexes= "mutator lock"(shared held)
03-29 01:06:28.411 10249 10452 W erwylo.beatgam:   at org.apache.commons.math3.transform.TransformUtils.createComplexArray(TransformUtils.java:138)
03-29 01:06:28.411 10249 10452 W erwylo.beatgam:   at org.apache.commons.math3.transform.FastFourierTransformer.transform(FastFourierTransformer.java:376)
03-29 01:06:28.411 10249 10452 W erwylo.beatgam:   at com.serwylo.beatgame.audio.fft.FFTKt.calculateFFTWindow(FFT.kt:181)
03-29 01:06:28.411 10249 10452 W erwylo.beatgam:   at com.serwylo.beatgame.audio.fft.FFTKt.calculateMp3FFTWithValues(FFT.kt:46)
03-29 01:06:28.411 10249 10452 W erwylo.beatgam:   at com.serwylo.beatgame.audio.AudioIOKt.loadFromDisk(AudioIO.kt:46)
03-29 01:06:28.411 10249 10452 W erwylo.beatgam:   at com.serwylo.beatgame.audio.AudioIOKt.loadWorldFromMp3(AudioIO.kt:31)
03-29 01:06:28.411 10249 10452 W erwylo.beatgam:   at com.serwylo.beatgame.screens.LoadingScreen$startLoading$1.run(LoadingScreen.kt:103)
03-29 01:06:28.411 10249 10452 W erwylo.beatgam:   at java.lang.Thread.run(Thread.java:923)

It starts working at around 220 seconds of audio (240 seconds still fail).

Device memory:

crosshatch:/ # free -m
                total        used        free      shared     buffers
Mem:             3579        3473         105          15           3
-/+ buffers/cache:           3470         109
Swap:            2047        1567         480

[pserwylo]

Ah, thanks so much for the extremely detailed report.

I really need to change the implementation of level generation so that it streams the entire thing.

At present, it actually needs to process the entire level in one go before then deciding how to extract obstacles. I think this could realistically be improved to instead analyse the first 10 seconds of audio, find extract any obstacles, throw away the source information, then move onto the next 10 seconds (with a little bit of magic to make sure that any obstacle spanning the gap between each 10 second block is managed appropriately).

Making this change would certainly address this issue.

[pserwylo]

In fact, I think by adding a few unit tests, and perhaps a GUI tool to view the output of such processes, that would also help with not only improvements to this part of the game, but also it will help when trying to investigate better level generation strategies (i.e. #43).