Massive Event Loop Lag caused by VALIDATE_CREDENTIALS and readFileSync

[nik-suri]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution
• This is an issue with version 2.x of the SDK

Describe the bug
I have an application running on EC2 that consumes events from SQS, processes them, and publishes new events to another queue on SQS. When running this application using pm2 I noticed very large event loop lags (up to 500ms). After profiling and analyzing the flamegraph, I saw that the lag is caused by attempting to turn sqs.sendMessageBatch into a promise, using sqs.sendMessageBatch.promise(). These calls end up performing costly operations which appear to all be synchronous (including readFileSync, emitting events, and handleRequest).

I also see repeated calls to VALIDATE_CREDENTIALS, which leads to costly hashing operations. I have attached an IAM role to EC2 which gives me access to my other AWS services. After going through your sdk code I thought that directly passing in environment variables for my ACCESS/SECRET keys might help resolve the issue. However, this did not work either.

I have done some digging and found these two related issues:
#3926 -- running in EC2 results in repeated filesystem reads for getting the credentials.
aws/aws-sdk-go#2972 -- getting credentials from the EC2 role results in slowdowns.

Is the issue in the browser/Node.js?
Node.js

If on Node.js, are you running this on AWS Lambda?
No

Details of the browser/Node.js version
v17.2.0

SDK version number
2.1048.0

To Reproduce (observed behavior)

1. Create SQS queue
2. Begin publishing events to SQS queue
3. View large event loop lag in pm2

Expected behavior
Publishing events to SQS using promises should result in async behavior, and should definitely not result in the synchronous bottlenecks seen below.

Screenshots

Additional context
Running on EC2. Have attempted to prevent repeated calls to VALIDATE_CREDENTIALS by removing the IAM role from my EC2 instance and passing in my ACCESS/SECRET keys as environment variables.

[dreamspider42]

Was there any action on this? We've been seeing lots of readFileSync as well and seem to be setting all the credentials explicitly.

[dreamspider42]

Wanted to leave an update it seems like this can be prevented if you are using aws-sdk version 2.683 or later along with either setting the environment variables: AWS_ENABLE_ENDPOINT_DISCOVERY or AWS_ENDPOINT_DISCOVERY_ENABLED to a value (either true or false). If it is undefined in the service config or the env var it attempts to look for a configuration file. Since it will return false in the event that file is not found, you can save yourself some unnecessary IO by setting it explicitly to false.

You can see the related code that triggers this process here:

aws-sdk-js/lib/discover_endpoint.js

Line 273 in 5244129

function resolveEndpointDiscoveryConfig(request) {

[ajredniwja]

@nik-suri apologies the issue fell out of queue, @dreamspider42 comment holds true. Will track the issue with #3926

[aBurmeseDev]

Similar issue was reported here and our team member responded in this comment. Closing as duplicate.