SolvedBehat 3.4.0 breaks the ability to run individual tests

Update + quick fix

This is a long standing issue which is affecting many users. It has been solved but the PR is not yet accepted. If you are affected by this you can solve it by adding the following lines to your composer.json:

"require-dev": {
    "behat/gherkin": "^4.6.0",
    "cweagans/composer-patches": "^1.6.7",
},
"patches": {
    "behat/gherkin": {
        "Correctly filter features, now that the base path is correctly set": "https://github.com/Behat/Gherkin/compare/v4.6.0...a7d84eb6ba245ea268a304312d14660788bf6c29.patch"
    }
}

Note that this problem is not caused directly by Behat but it is due to a confusion between absolute and relative paths in the Behat/Gherkin library. We are keeping this issue open in the Behat queue so that it is easier to find for people who encounter this problem when using Behat.

If possible please vote for + review + approve the fix at Behat/Gherkin#129 - hopefully once it is approved it can be merged.

Original summary

This was originally reported at Behat/Gherkin#128 but it appears to be a bug in Behat and not in Gherkin.

In #1056 a change was introduced to make it possible to run individual tests from inside a different folder than where the tests are located. Unfortunately this breaks the more common use case of running the tests from inside the actual folder where they are located.

To replicate this, create a folder structure in which the tests are located in a subfolder of the root of the project, e.g in the tests/ folder:

$ composer require behat/behat-3.4.0
$ mkdir tests && cd tests/
$ ../vendor/bin/behat --init
# Create a test
$ vim features/homepage.feature
# Make sure the gherkin cache is cleared, this might "hide" the problem
$ rm -rf /tmp/behat_*
# Try to run a single test
$ ../vendor/bin/behat features/homepage.feature -vvv

Result:

$ ../vendor/bin/behat features/homepage.feature -vvv


No scenarios
No steps
0m0.73s (9.22Mb)

                                                                                                                                            
  [Behat\Testwork\Tester\Exception\WrongPathsException]                                                                                     
  No specifications found at path(s) `features/homepage.feature`. This might be because of incorrect paths configuration in your `suites`.  
                                                                                                                                            

Exception trace:
 () at /home/pieter/v/joinup-dev/vendor/behat/behat/src/Behat/Testwork/Tester/Cli/ExerciseController.php:111
 Behat\Testwork\Tester\Cli\ExerciseController->execute() at /home/pieter/v/joinup-dev/vendor/behat/behat/src/Behat/Testwork/Cli/Command.php:63
 Behat\Testwork\Cli\Command->execute() at /home/pieter/v/joinup-dev/vendor/symfony/console/Command/Command.php:264
 Symfony\Component\Console\Command\Command->run() at /home/pieter/v/joinup-dev/vendor/symfony/console/Application.php:841
 Symfony\Component\Console\Application->doRunCommand() at /home/pieter/v/joinup-dev/vendor/symfony/console/Application.php:206
 Symfony\Component\Console\Application->doRun() at /home/pieter/v/joinup-dev/vendor/behat/behat/src/Behat/Testwork/Cli/Application.php:124
 Behat\Testwork\Cli\Application->doRun() at /home/pieter/v/joinup-dev/vendor/symfony/console/Application.php:125
 Symfony\Component\Console\Application->run() at /home/pieter/v/joinup-dev/vendor/behat/behat/bin/behat:34

Using git bisect I have pinpointed the introduction of this bug to 08802b7

44 Answers

✔️Accepted Answer

On a side note, I was able to run tests again from PhpStorm after in the Run/Debug configuration I set the Custom working directory option (in the default configuration for Behat) to the directory in which behat.yml lives (it needs to align with whatever in Behat is %paths.base%).

Other Answers:

Revert to 3.3 for now, I'll try to look into the bug ASAP.

We also can't run a single feature anymore since the update to Behat 3.4. We start a single feature with bin/behat feature/natarch-continuous/ArchiveLandingPage.feature - I seem to have found the culprit within the Behat\Behat\Gherkin\Specification\Locator\FilesystemFeatureLocator::locateSpecifications() - That method instantiates the LazyFeatureIterator with the $locator = 'features/natarch-continuous/ArchiveLandingPage.feature' and $filters = array(new PathsFilter($suiteLocators));
If I set the $filters = []; Then the single feature is run. I can also run bin/behat features/natarch-continuous/ and it finds and runs the features inside the directory.

This is our behat.yml configuration.

default:
  suites:
    natarch_features:
      paths: [ features ]
      contexts:
        - Drupal\DrupalExtension\Context\DrupalContext
        - Drupal\DrupalExtension\Context\MessageContext
        - Drupal\DrupalExtension\Context\DrushContext
        - Drupal\DrupalExtension\Context\MarkupContext
        - rdx\behatvars\BehatVariablesContext
        - FeatureContext
        - OtapDrushContext
        - SeleniumContext
        - CustomSoapContext
        - DebugContext:
           screen_shot_path: project/behat/screenshots
  formatters:
    pretty:
    junit:
      output_path: %paths.base%/build
  extensions:
    Behat\MinkExtension:
      goutte:
        guzzle_parameters:
          verify: false
      javascript_session: selenium2
      selenium2:
        wd_host: http://natarch.dev:8643/wd/hub
#        wd_host: http://<your-local-ip-address>:4444/wd/hub
      base_url: http://natarch.dev
#      default_session: selenium2
#      browser_name: 'firefox'
      files_path: %paths.base%/features/dummy
    Drupal\DrupalExtension:
      subcontexts:
        paths:
          - %paths.base%/features/bootstrap
      blackbox: ~
      api_driver: 'drupal'
      drush_driver: 'drush'
      drush:
        root: '/var/www/natarch'
      drupal:
        drupal_root: '/var/www/natarch'
      text:
        username_field: "Gebruikersnaam"
        password_field: "Wachtwoord"
        log_in: "Inloggen"
        log_out: "Uitloggen"
      region_map:
        breadcrumb: ".region--breadcrumb"
        content: ".region-content, .layout-content"
        footer: ".region--footer"
        header: ".region--header"
        highlighted: ".region--highlighted"
        main_links: ".region--main-links"
        primary_menu: ".region--primary-menu"
        secondary_menu: "#block-natarch-account-menu"
        sidebar_first: ".layout-sidebar-first"
        site_menu_first: ".region--site-menu-first"
        site_menu_second: ".region--site-menu-second"
        title_image: ".region--title-image"
      selectors:
        message_selector: ".messages"
        error_message_selector: ".messages.messages--error"
        success_message_selector: ".messages.messages--status"
    Behat\SoapExtension:
          # An associative array as second argument for \SoapClient::__soapCall().
          options: {}
          # An associative array as parameters for \DOMXpath::registerNamespace().
          namespaces: {}
    rdx\behatvars\BehatVariablesExtension: ~
testserver:
  extensions:
    Behat\MinkExtension:
      selenium2:
        wd_host: http://miauwmiauwmiauw.test.rad.lan:8643/wd/hub
      base_url: https://test.miauwmiauwmiauw.nl
    Drupal\DrupalExtension:
      drush:
        root: '/var/www/miauwmiauwmiauw-web'
      drupal:
        drupal_root: '/var/www/miauwmiauwmiauw-web'
testzap:
  extensions:
    Behat\MinkExtension:
      selenium2:
        #wd_host: http://miauwmiauwmiauw.test.rad.lan:8643/wd/hub
        wd_host: http://miauwmiauwmiauw:8643/wd/hub
        capabilities:
          proxy:
            proxyType: 'manual'
            httpProxy: 'localhost:6667'

Can we remove $filters = array(new PathsFilter($suiteLocators)); from the FilesystemFeatureLocator?

More Issues: