Directory

Too much memory usage on build · Issue #58902 · WordPress/gutenberg · GitHub
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Too much memory usage on build #58902

Open
alexstine opened this issue Feb 9, 2024 · 9 comments
Open

Too much memory usage on build #58902

alexstine opened this issue Feb 9, 2024 · 9 comments
Labels
[Priority] High Used to indicate top priority items that need quick attention [Type] Bug An existing feature does not function as intended [Type] Build Tooling Issues or PRs related to build tooling

Comments

@alexstine
Copy link
Contributor

Description

I spun up my new WSL VM this afternoon and discovered the npm run build script is using an insane amount of memory. I received a Node out of memory heap error and had to increase using this environment variable.

export NODE_OPTIONS=--max_old_space_size=6000

It looks like at peak build, it nearly uses 3GB of memory which is absolutely insane. It fails right at the TypeScript build.

Step-by-step reproduction instructions

  1. Spin up a WSL2 Ubuntu 20.04 VM.
  2. Clone the repo.
  3. Run npm i && npm run build.
  4. See error.

Screenshots, screen recording, code snippet

No response

Environment info

N/A.

Please confirm that you have searched existing issues in the repo.

Yes

Please confirm that you have tested with all plugins deactivated except Gutenberg.

Yes

@alexstine alexstine added [Type] Bug An existing feature does not function as intended [Priority] High Used to indicate top priority items that need quick attention labels Feb 9, 2024
@t-hamano
Copy link
Contributor

Thanks for the report.

I have been running the Gutenberg project on WSL2 for over two years, but I have never encountered such an error, nor do I remember operating max_old_space_size.

It looks like at peak build, it nearly uses 3GB of memory which is absolutely insane.

How can I test to measure this memory?

@gziolo gziolo added the [Type] Build Tooling Issues or PRs related to build tooling label Feb 10, 2024
@alexstine
Copy link
Contributor Author

@t-hamano If you do the following, you should be able to see memory usage.

  1. Start a screen session with screen -S build.
  2. Run npm run build.
  3. Press Control+A and then D to detach.
  4. Run free -g to output the memory in GB while the build is running. Run the command multiple times.

Part of me wonders if this is only an issue on the initial build now considering cache is likely used on each build after. This might not be reproducible without starting fresh.

Thanks.

@t-hamano
Copy link
Contributor

I performed the following steps.

  1. Cloning the gutenberg repository again.
  2. Run npm i.
  3. Start a screen session with screen -S build.
  4. Run npm run build.
  5. Press Control+A and then D to detach.
  6. Run free -g to output the memory in GB while the build is running. Run the command multiple times.

The average value of the cell where the used column and mem row intersect was 3, and increased to a maximum of 5. Is this measured correctly?

total used free shared buff/cache available
Mem: 31 3 24 0 3 28
Swap: 8 0 8

@alexstine
Copy link
Contributor Author

Yes, that looks correct. I don't ever remember our TypeScript taking that much memory before. Are you using NVM to manage Node?

@t-hamano
Copy link
Contributor

Are you using NVM to manage Node?

I'm currently using volta.

@alexstine
Copy link
Contributor Author

I'll leave this open in case anyone else has this problem with NVM, but it seems like I might be the only one. So strange. All in all, the Windows filesystem doesn't perform very well with Node so I guess this should come at no shock to anyone. Mac/Linux handles it much better.

@simonhammes
Copy link
Contributor

I am using NVM on Debian Trixie and just got an OOM while running npm run dev.

This was the last command:

node ./bin/packages/validate-typescript-version.js && tsc --build && node ./bin/packages/check-build-type-declaration-files.js

Running export NODE_OPTIONS=--max_old_space_size=6000 before npm run dev fixes the issue.

@alexstine
Copy link
Contributor Author

Yes, I am thinking we should either document this or fix the out of memory error.

@benridane
Copy link
Contributor

benridane commented May 26, 2024

It appears that the memory-consuming area when tsc --build is run is in packages/components.

212:~/wp1/gutenberg/packages/components$ ../../node_modules/.bin/tsc --build --extendedDiagnostics

<--- Last few GCs --->

[185248:0x62aa000]    45794 ms: Scavenge 972.5 (999.2) -> 969.2 (999.2) MB, pooled: 5 MB, 3.27 / 0.00 ms  (average mu = 0.436, current mu = 0.068) allocation failure;
[185248:0x62aa000]    46331 ms: Mark-Compact 972.5 (999.8) -> 969.1 (1008.5) MB, pooled: 0 MB, 528.37 / 0.01 ms  (average mu = 0.333, current mu = 0.208) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0xe18f84 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
 2: 0x12102f0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 3: 0x12105c7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 4: 0x14400d5  [node]
 5: 0x1459949 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 6: 0x142e018 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 7: 0x142ef45 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 8: 0x1407b8e v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
 9: 0x1868ffc v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
10: 0x1f1e576  [node]
中止 (コアダンプ)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Priority] High Used to indicate top priority items that need quick attention [Type] Bug An existing feature does not function as intended [Type] Build Tooling Issues or PRs related to build tooling
Projects
None yet
Development

No branches or pull requests

5 participants