RARE NOT RANDOM: USING TOKEN EFFICIENCY FOR SECRETS SCANNING

Summary from the TLDR InfoSec newsletter

Gitleaks has traditionally built its secrets scanner using a combination of regexes, entropy, and rule-based filters. This post details how the team adapted the scanner to use token efficiency, which tests how frequently a piece of text appears in a model's training data by dividing it into tokens, instead of entropy, to boost performance. Using this method and a few other tweaks, the author created a tool called BetterLeaks, which outperforms other scanners, such as CredSweeper, on secret detection in the CredData dataset.