Improve Github repository scan to include remote dangling commits

[mota-lhd]

Please review the Community Note before submitting

Description

Sometimes we also have some dangling commits on branches (merged or not) and these dangling commits contain sensitive information.

Preferred Solution

We want to

• identify dangling commits on Github repository branches
• run detectors on Github dangling commits

Additional Context

References

[rgmz]

Related to #1918?

[mota-lhd]

Related to #1918?

part of it yes ; the dangling commits parts is not in #1918. This happens when for example somebody forces push on a remote branch.

[rgmz]

I didn't think it was possible to access dangling commits from GitHub (e.g., GitHub retains "removed" commits but if you run git clone and then git reflog you will not see them.) Do you have an example?

[mota-lhd]

yes, Github does not have an API to list remote dangling commits but there are some ways to get them.

• get all commits in the repository through GraphQL and filter out commits that are related to pull-requests.
• get all force-push events within a PR and try to fetch commits from that force-push commit.

long-running jobs, I agree, but worth it to unearth some juicy secrets :)

[MickaelFontes]

Hello @rgmz 👋 !

I have an example here (https://github.com/MickaelFontes/dangling-finder/blob/2e5c6e94cd5c4835a5e775c51715d45a57f0c5c2/dangling_finder/listing.py#L49-L78) to query for dangling commits created by a force push in a pull request.
(It takes some time to get the full list of force-pushed references on a big repo ~8 minutes or a repo with ~ 20k PR)

That's at least one straightforward way to find some dangling commits, since we can't directly ask for a list of all dangling commits.

There must be some other effective ways to find other dangling commits, but I'm still looking for them 😅