linguist

Linguist This library is used on GitHub.com to detect blob languages, ignore binary or vendored files, suppress generated files in diffs, and generate language breakdown graphs. Documentation How Linguist works Change Linguist's behaviour with overrides Troubleshooting Contributing guidelines Installation Install the gem: gem install github-linguist Dependencies Linguist is a Ruby library so you will need a recent version of Ruby installed. There are known problems with the macOS/Xcode supplied version of Ruby that causes problems installing some of the dependencies. Accordingly, we highly recommend you install a version of Ruby using Homebrew, rbenv, rvm, ruby-build, asdf or other packaging system, before attempting to install Linguist and the dependencies. Linguist uses charlock_holmes for character encoding and rugged for libgit2 bindings for Ruby. These components have their own dependencies. charlock_holmes cmake pkg-config ICU zlib rugged libcurl OpenSSL You may need to install missing dependencies before you can install Linguist. For example, on macOS with Homebrew: brew install cmake pkg-config icu4c On Ubuntu: sudo apt-get install build-essential cmake pkg-config libicu-dev zlib1g-dev libcurl4-openssl-dev libssl-dev ruby-dev Usage Application usage Linguist can be used in your application as follows: require 'rugged' require 'linguist' repo = Rugged::Repository.new('.') project = Linguist::Repository.new(repo, repo.head.target_id) project.language #=> "Ruby" project.languages #=> { "Ruby" => 119387 } Command line usage The github-linguist executable operates in two distinct modes: Git Repository mode - Analyzes an entire Git repository (when given a directory path or no path) Single file mode - Analyzes a specific file (when given a file path) Git Repository A repository's languages stats can be assessed from the command line using the github-linguist executable. Without any options, github-linguist will output the language breakdown by percentage and file size. cd /path-to-repository github-linguist You can try running github-linguist on the root directory in this repository itself: $ github-linguist 66.84% 264519 Ruby 24.68% 97685 C 6.57% 25999 Go 1.29% 5098 Lex 0.32% 1257 Shell 0.31% 1212 Dockerfile Additional options --rev REV The --rev REV flag will change the git revision being analyzed to any gitrevisions(1) compatible revision you specify. This is useful to analyze the makeup of a repo as of a certain tag, or in a certain branch. For example, here is the popular Jekyll open source project. $ github-linguist jekyll 70.64% 709959 Ruby 23.04% 231555 Gherkin 3.80% 38178 JavaScript 1.19% 11943 HTML 0.79% 7900 Shell 0.23% 2279 Dockerfile 0.13% 1344 Earthly 0.10% 1019 CSS 0.06% 606 SCSS 0.02% 234 CoffeeScript 0.01% 90 Hack And here is Jekyll's published website, from the gh-pages branch inside their repository. $ github-linguist jekyll --rev origin/gh-pages 100.00% 2568354 HTML --breakdown The --breakdown or -b flag will additionally show the breakdown of files by language. You can try running github-linguist on the root directory in this repository itself: $ github-linguist --breakdown 66.84% 264519 Ruby 24.68% 97685 C 6.57% 25999 Go 1.29% 5098 Lex 0.32% 1257 Shell 0.31% 1212 Dockerfile Ruby: Gemfile Rakefile bin/git-linguist bin/github-linguist ext/linguist/extconf.rb github-linguist.gemspec lib/linguist.rb … --strategies The --strategies or -s flag will show the language detection strategy used for each file. This is useful for understanding how Linguist determined the language of specific files. Note that unless the --json flag is specified, this flag will set the --breakdown flag implicitly. You can try running github-linguist on the root directory in this repository itself with the strategies flag: $ github-linguist --breakdown --strategies 66.84% 264519 Ruby 24.68% 97685 C 6.57% 25999 Go 1.29% 5098 Lex 0.32% 1257 Shell 0.31% 1212 Dockerfile Ruby: Gemfile [Filename] Rakefile [Filename] bin/git-linguist [Extension] bin/github-linguist [Extension] lib/linguist.rb [Extension] … If a file's language is affected by .gitattributes, the strategy will show the original detection method along with a note indicating whether the gitattributes setting changed the result or confirmed it. For instance, if you had the following .gitattributes overrides in your repo: *.ts linguist-language=JavaScript *.js linguist-language=JavaScript the output of Linguist would be something like this: 100.00% 217 JavaScript JavaScript: demo.ts [Heuristics (overridden by .gitattributes)] demo.js [Extension (confirmed by .gitattributes)] --json The --json or -j flag output the data into JSON format. $ github-linguist --json {"Dockerfile":{"size":1212,"percentage":"0.31"},"Ruby":{"size":264519,"percentage":"66.84"},"C":{"size":97685,"percentage":"24.68"},"Lex":{"size":5098,"percentage":"1.29"},"Shell":{"size":1257,"percentage":"0.32"},"Go":{"size":25999,"percentage":"6.57"}} This option can be used in conjunction with --breakdown to get a full list of files along with the size and percentage data. $ github-linguist --breakdown --json {"Dockerfile":{"size":1212,"percentage":"0.31","files":["Dockerfile","tools/grammars/Dockerfile"]},"Ruby":{"size":264519,"percentage":"66.84","files":["Gemfile","Rakefile","bin/git-linguist","bin/github-linguist","ext/linguist/extconf.rb","github-linguist.gemspec","lib/linguist.rb",...]}} NB. The --strategies flag has no effect, when the --json flag is present. Single file Alternatively you can find stats for a single file using the github-linguist executable. You can try running github-linguist on files in this repository itself: $ github-linguist grammars.yml grammars.yml: 884 lines (884 sloc) type: Text