Feature file statistics

A suite of feature files grows one scenario at a time, so nobody knows what it adds up to. gurkencheck stats counts it: how many test cases really run, how much of the step vocabulary is shared, and where the same sentence has been written two ways.

Running it#

The linter tells you what is wrong with your feature files. The same files answer a different question — how much is there, and how much of it is the same thing written twice — and stats is the command that asks it. There is nothing to configure.

npx gurkencheck stats features

With no paths given it searches the current directory, the same way the linter does. The word stats has to come first, before any option. Written anywhere else it is read as a path rather than as a command, so gurkencheck --format json stats reports that there is no such file — or quietly lints a directory, if one of that name happens to exist.

The report goes to stdout, so gurkencheck stats > report.txt works. It exits 0 whatever it finds. Statistics describe a suite rather than judge it, and nothing here should ever be the reason a build goes red.

The report#

This is the whole of it for a small shop's worth of feature files. Every list in it is the head of a longer one, cut to the number of entries --top allows and marked with how many were left out; this run asked for three, to keep it short.

9 files, 9 features, 38 test cases

Inventory
  Features              9
  Rules                 2
  Backgrounds           8
  Scenarios            23
  Scenario Outlines     5
  Examples tables       5   15 rows
  Steps               104   as written
  Data tables           0
  Doc strings           0

Scenarios
  Test cases           38   one per Scenario, one per Examples row
  Steps per scenario   min 2   median 3   p90 4   max 4   mean 3.3   Background steps excluded

  Longest
    4  Adding the first item         features/basket.feature:7
    4  Checking out with no address  features/checkout.feature:15
    4  Choosing next day delivery    features/delivery.feature:4
    … and 25 more scenarios

Steps
  Written          104   Background steps included
  Distinct          63   61% of all steps - lower is more reuse
  Written once      43   68% of distinct steps
  Words per step   min 3   median 6   p90 7   max 10   mean 5.7
  Keywords         Given 39 (38%)   When 29 (28%)   Then 36 (35%)   And and But resolved to what they follow

  Most used
    8  i am signed in as ""         features/basket.feature:5
    6  i have 0 items in my basket  features/checkout.feature:6
    4  i check out                  features/checkout.feature:12
    … and 17 more steps

  Written once
    i add "" to my wishlist            features/wishlist.feature:8
    i add <> of "" to my basket        features/basket.feature:26
    i am asked for a delivery address  features/checkout.feature:18
    … and 40 more steps

  Nearly the same (6 groups)
    8  i am signed in as ""  features/basket.feature:5
    1  i'm signed in as ""   features/wishlist.feature:5

    6  i have 0 items in my basket  features/checkout.feature:6
    2  i have 0 item in my basket   features/basket.feature:14

    4  i have a delivery address on file   features/checkout.feature:11
    1  i have no delivery address on file  features/checkout.feature:16
    … and 3 more groups

Tags
  Written              13
  Distinct             10
  Written once          7   a tag written once is often a typo of one written often
  Untagged scenarios    0   0% of scenarios, counting inherited tags

  Most used
    2  @account
    2  @checkout
    2  @slow

  Written once
    @basket, @delivery, @payment
    … and 4 more tags

The sections below walk it in the order it prints, because several of the numbers count something slightly different from what their name suggests.

Test cases, not scenarios#

These files hold 23 Scenarios and 5 Scenario Outlines, and the suite runs 38 tests. An Outline is not one test. It is one per row of its Examples table, and the five tables here carry 15 rows between them. That is the figure that predicts how long a run takes, and it is almost always higher than the one people carry in their heads.

An Outline that has no Examples table yet counts as one, which is how max-scenarios-per-file counts it as well, so the two never disagree about the size of a suite.

Two numbers called steps#

The report prints 104 steps written and a median of 3 steps per scenario, and the two do not reach each other: 104 steps across 28 scenarios would be 3.7 apiece. They count different things. The 104 is every step in the files, Background steps included — 11 of them here — because every one of them still needs a step definition behind it. The steps per scenario leave Backgrounds out, because a Background is written once and read many times, and charging it to every scenario would make a tidy suite look long-winded.

The summary line is four real numbers and an average: the min, median, p90 and max are each a step count that some scenario in your files actually has, and the mean is not. A mean of 3.3 on its own would hide a forty-step scenario; the Longest list underneath names it and says where it is.

What makes two steps the same step#

Distinct is the number worth watching, and what it means depends entirely on when two steps count as one. I have 3 items in my basket and I have 17 items in my basket are one step behind one step definition, and counting them as two would make every measure of reuse meaningless.

So before two steps are compared, the parts a step definition would capture are replaced by a marker. A number becomes 0, a double quoted string becomes "", and a Scenario Outline placeholder becomes <>. Case, repeated spaces, and a full stop or an exclamation mark at the end are ignored. The keyword is left out altogether, so a Given and an And of the same sentence are one step.

Single quotes are left exactly as they are. the user's basket is 'empty' has three of them, and a rule that paired them up would eat half the sentence. Cucumber's own expressions quote with " in any case.

Each kind of argument keeps a marker of its own, so the delivery cost is 0 and the delivery cost is <> stay apart. That understates reuse a little in a suite full of Outlines, and it is the trade worth making: the normalised text is what the report shows you, and a 0 in a step that has no number in it would be a lie about the file. The next section puts the two back together.

A low share of distinct steps means the team shares a vocabulary. A high one means everybody invents their own phrasing, and the step definitions rot.

Steps written two ways#

This is the part of the report you can act on the same afternoon. Each group is one behaviour that is costing you more than one step definition.

  Nearly the same (6 groups)
    8  i am signed in as ""  features/basket.feature:5
    1  i'm signed in as ""   features/wishlist.feature:5

    6  i have 0 items in my basket  features/checkout.feature:6
    2  i have 0 item in my basket   features/basket.feature:14

    4  i have a delivery address on file   features/checkout.feature:11
    1  i have no delivery address on file  features/checkout.feature:16

    2  my basket has 0 item   features/basket.feature:10
    1  my basket has 0 items  features/basket.feature:16

    2  the delivery cost is 0   features/delivery.feature:8
    1  the delivery cost is <>  features/delivery.feature:20

    1  the basket total is 0   features/basket.feature:11
    1  the basket total is <>  features/basket.feature:27

Two steps are grouped when at most three single character edits turn one into the other, and when those edits are no more than about a seventh of the longer of the two. Steps shorter than eight characters are left out, because at that length almost everything looks like everything. Three edits is deliberately tight: further apart than that and a step is a different sentence rather than the same one spelled two ways.

A group may be a chain. Where one step is close to a second and the second is close to a third, all three are reported together even though the first and the third are not close to each other, so a phrasing that drifted over three years arrives as one group rather than two.

Nothing on the command line changes how close two steps have to be. A script can: collectStatistics takes a similarity setting.

Tags nobody agreed on#

Seven of the ten tags here are written exactly once. Some of those are deliberate. The rest are what a tag looks like when it was typed from memory: @wishlst sits in that list a few characters from @wishlist, and a run filtered on either of them quietly misses the other's scenarios.

Untagged scenarios counts the ones carrying no tag of their own, none on their Examples tables, and none inherited from their Feature or their Rule, and so reachable by no tag expression at all. A tag on a Feature covers every scenario inside it, which is why that figure is zero here and often is.

Keeping the report beside a build#

The text report is for reading. The other two formats are for keeping and for showing to somebody else.

npx gurkencheck stats features --format json > stats.json

--format json writes the whole dataset with none of the lists cut short: every distinct step, every scenario, every group, and every file that could not be read. It is indented rather than packed onto one line, so two runs of it can be read side by side and a build can keep a record of what the suite looked like when it went out. --top does not apply to it.

--format md writes the same report as Markdown tables, for pasting into a pull request where the person who has to agree to the work is not going to run the command themselves.

Options#

Option	What it does
`-f, --format`	Output format: `text` (the default), `json` or `md`. These are the stats command's own formats; the linter's do not apply here.
`-i, --ignore`	Comma separated globs to skip. Replaces `.gurkencheckignore` rather than adding to it.
`-l, --language`	The dialect to read files in when they carry no `# language:` header. This command does not read `.gurkencheckrc`, so a `language` key there does not reach it.
`--top`	How many entries each list in the text and Markdown reports shows. Ten by default, and it changes nothing under `--format json`, which never cuts a list short.
`-h, --help`	Show the options.

Everything the command can refuse is refused before a file is read: a format that does not exist, a --top that is not a whole number, a language code that is not a dialect, an argument that is neither a feature file, a directory nor a glob. Any of those exits 2. A path that is well formed but matches nothing is not one of them — the report says No feature files found. and exits 0 — so a directory renamed out from under a build script goes unnoticed rather than failing it. Once a report has been produced it exits 0, whatever the numbers say.

What is not counted#

A file the parser refuses is listed at the end of the report and counted nowhere in it. Half a broken file is worse than none of it, because every number above it would quietly be wrong. That list is wider than files with a syntax error in them: it also holds the ones breaking a rule Gherkin itself enforces — two Features in one file, a second Background, a tag on a Background — which are the 5 always-on rules.

Nothing on this page has an opinion. There is no threshold to cross, no figure that turns the report red, and no advice about what a good reuse number looks like — a target invented here would be quoted back as a standard. The report says lower is more reuse and stops there. What to do about it is yours, and holding a team to it afterwards is what the rules are for.