Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

integration tests failing #592

Closed
geemus opened this issue Apr 21, 2023 · 16 comments · Fixed by #593
Closed

integration tests failing #592

geemus opened this issue Apr 21, 2023 · 16 comments · Fixed by #593

Comments

@geemus
Copy link
Member

geemus commented Apr 21, 2023

@Temikus - do you have any ideas on what might be happening with the integration tests? Seems to be failing when trying to bundle and reference the local stuff. It's not obvious to me that my recent merge impacted it, so I wasn't sure what else may have changed. You can see an example of the failure here: https://github.com/fog/fog-google/actions/runs/4767247086. Thanks!

@Temikus
Copy link
Member

Temikus commented Apr 22, 2023

Bundler fails for some reason:

Using fog-google 1.19.0 from source at `.`
Error: The process '/opt/hostedtoolcache/Ruby/2.7.8/x64/bin/bundle' failed with exit code null
##[debug]Node Action run completed with exit code 1
##[debug]Finishing: Set up Ruby

I suspect the new Ruby 2.7.8 has some caveats. This is a local runner, so I should be able to debug a bit.

@Temikus
Copy link
Member

Temikus commented Apr 22, 2023

Cannot repro on the runner either 🤔

Using fog-google 1.19.0 from source at `.`
Bundle complete! 14 Gemfile dependencies, 78 gems now installed.
Bundled gems are installed into `./vendor/bundle`

@Temikus
Copy link
Member

Temikus commented Apr 22, 2023

Lemme try running from master to exclude any weirdness.

@Temikus
Copy link
Member

Temikus commented Apr 22, 2023

The only difference with unit that passed is:

        env:
          # Needs to be set up for self-hosted runners, see:
          # https://github.com/ruby/setup-ruby#using-self-hosted-runners
          # Image used in runners: summerwind/actions-runner
          ImageOS: ubuntu20

Maybe actions-runner no longer needs it? Let's try it out I guess 🤔

@Temikus
Copy link
Member

Temikus commented Apr 22, 2023

That didn't work either - kinda bizarre 🤔 I'll dig a bit more this evening.

@Temikus Temikus reopened this Apr 22, 2023
@Temikus
Copy link
Member

Temikus commented Apr 22, 2023

Now it's running fine in #594 🙃 I think there's something weird with our runners 🤔

@geemus
Copy link
Member Author

geemus commented Apr 24, 2023

Weird. Maybe just some kind of intermittent issue...

@Temikus
Copy link
Member

Temikus commented Apr 25, 2023

I'm gonna try and add some debugging now and see how it goes on the main branch again.

@Temikus
Copy link
Member

Temikus commented Apr 25, 2023

Pushed a new debug workflow in, I'll see what it spits out 🤔

@Temikus
Copy link
Member

Temikus commented Apr 25, 2023

Ok, so something looks to be killing the runners:

Installing parser 3.2.2.1
/runner/_work/_temp/b9ca9963-76ef-47c5-b939-eab94c68eea3.sh: line 1:   354 Killed                  bundle install --verbose
Error: Process completed with exit code 137.

Looks like we're OOM'ing:
Screenshot 2023-04-25 at 6 28 55 pm

Memory cgroup out of memory: Killed process 2996984 (bundle) total-vm:777476kB, anon-rss:325672kB, file-rss:4792kB, shmem-rss:0kB, UID:1000 pgtables:860kB oom_score_adj:937

@Temikus
Copy link
Member

Temikus commented Apr 25, 2023

Resized everything, will push a change soon as soon as the cluster rescales.

@Temikus
Copy link
Member

Temikus commented Apr 26, 2023

Tests are looking much better - I will wait for current queue to finish and then restore caching and see if that was the reason for ruby/setup-ruby failing.

@Temikus
Copy link
Member

Temikus commented Apr 26, 2023

Aight, tests are all green, trying with the ruby/setup-ruby 👍

@geemus
Copy link
Member Author

geemus commented Apr 26, 2023

@Temikus Awesome, thanks again for digging in and getting to the bottom of this.

@Temikus
Copy link
Member

Temikus commented May 2, 2023

No worries, everything's looking good now ( except the usual Google API jitter :/ ) - I'll merge some fixes I made along the way soon 👍

@Temikus Temikus closed this as completed May 2, 2023
@geemus
Copy link
Member Author

geemus commented May 3, 2023

Sounds good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants