Skip to main content

MySQL 8.0: New mysql-test-run option to minimize skipped tests in regression test runs

Anybody who has run the MySQL MTR test suite must have seen a statement like “x tests were skipped, y by the test itself. ” at the end of test runs. Why are some tests being skipped? Are the skipped tests affecting test coverage? If you are interested to know, read on …

Background

The MTR test suite is run regularly on multiple platforms and with various values for MySQL system variables. Not all MTR tests can run on all environments making it necessary to skip tests in some cases. MTR supports this with the command called skip. It is used to check if the prerequisites needed to run a test are satisfied . If not satisfied, the test is skipped with an informative message defined by the test author. For example, tests that can run only on windows have to be skipped on other platforms.
However, wrong usage of ‘skip’ can lead to a situation where some tests are never run in the regression test environment. This leads to reduced test coverage and hence bug escape. An example is “Bug#80288 missing innodb_numa_interleave” which escaped our regression tests, but was caught by community.  There was an error in the build script due to which server was built without NUMA support, even though the machine had NUMA libraries installed.

Solution

To resolve this, we have introduced a new MTR option called ‘no-skip‘. When a test or a test-suite is run with no-skip option, it ignores all the skip commands used within the test and forces the test to run . In case of a missing prerequisite, the test will fail and will be noticed in the regression test environment. This is a better behavior than silently skipping tests. Note that by default this variable is OFF.
However, during the course of this work-log we discovered that skip cannot be avoided in all situations. For example a test that can only run on Windows has to be skipped on other platforms. To handle these special cases we have introduced a file ‘excludenoskip.list’. It will contain a list of tests which should continue to skip even if MTR is run with –no-skip option.
To help understand what the excludenoskip.list does, refer the listed example below.

Other Improvements

This project not only added –no-skip option, but also identified scenarios where skip could be avoided,
  • Tests modified to set required mysqld system variables(WL#9027 )– We identified few tests that used to skip if a mysqld system variable is not set to a required value. Tests have the ability to set system variables to any value and hence this skip was unnecessary . It is good to have the mysqld option set in the test itself because this will always allow the test to run. If instead, a skip check is used, the test will be skipped if the option is not passed while running the test. For instance – Test main.flush2 needs this option replay_log_info_repository to be set to ‘FILE’ for execution. Instead of creating a dependency check (eg- include/not_relay_log_info_table.inc), this dynamic server variable was set within the test.
  • rpl and binlog test suites split based on required GTID_MODE setting( WL#9249, WL#9394 ): The rpl and binlog suites had 3 types of tests. Tests that can run only with GTID_MODE ON, tests that can run only with GTID_MODE OFF and tests that are GTID_MODE agnostic. In order to run all tests one had to run the suites with both GTID_MODE ON and OFF. The default value of the variable is OFF and hence tests that need ON were not run if one does not explicitly pass GTID_MODE=ON in MTR command line. We improved this situation by splitting the tests into rpl, rpl_gtid and rpl_nogtid, binlog, binlog_gtid and binlog_nogtid depending on whether they need gtid ON or OFF.
  • Redundant skips removed (WL#9170 ) – Some skips were redundant because of changes in the product. For ex- In MySQL 8.0.0, we can no longer compile out charsets from the server. This implies charsets will always be available and we no longer need to check for its existence. All checks for charsets were removed from tests.


Conclusion

Running tests with –no-skip option will certainly reduce the number of skipped tests. It will also ensure that there are no tests getting skipped accidentally.
The no-skip option is now used for all MySQL internal testing. We recommend community to use it for improving test coverage.
In case of any questions, please drop a comment below and we would be glad to address them.
Thank you for using MySQL!









Comments

Post a Comment

Popular posts from this blog

Design Tools to Help You Create Your Next Project- Part 3

Coolors Coolors   is a super fast color scheme generator. You can explore thousands of pre-existing color schemes (each one features five colors). Or, you can generate your own in a matter of minutes. Once you go to the “generate” page, hit the space bar to start with a different color scheme, and then you can adjust each color’s hue, saturation, and brightness accordingly. Web Gradients Web Gradients   is a collection of almost 200 background gradients, created by the  itmeo  team. You can use each of these content backdrops for any part of your website. You’ll find a .PNG version of each gradient, as well as easy-to-copy CSS3 crossbrowser code. Bonus: there are even curated packs for  Sketch  &  Photoshop . Color Hunt On  Color Hunt , browse through countless palettes, comprised of four colors each. You can browse and sort through the list based on what’s hot and popular, or just pick “random” and see what comes u...
Post a Comment
Read more

WordPress 4.8 Release Candidate 2 on June 1, 2017

The second release candidate for WordPress 4.8 is now available. To test WordPress 4.8, you can use the  WordPress Beta Tester  plugin or you can  download the release candidate here  (zip). We’ve made  a handful of changes  since releasing RC 1 last week. For more details about what’s new in version 4.8, check out the  Beta 1 ,  Beta 2 , and  RC1  blog posts. Think you’ve found a bug?  Please post to the  Alpha/Beta support forum . If any known issues come up, you’ll be able to  find them here . Happy testing!                                                                                                                   ...
Post a Comment
Read more

#PHP 5.6 is now in security fix only mode, which runs until the end of 2018. Plan your upgrades to PHP 7!- The 5th Annual China PHP Conference

Supported Versions Each release branch of PHP is fully supported for two years from its initial stable release. During this period, bugs and security issues that have been reported are fixed and are released in regular point releases. After this two year period of active support, each branch is then supported for an additional year for critical security issues only. Releases during this period are made on an as-needed basis: there may be multiple point releases, or none, depending on the number of reports. Once the three years of support are completed, the branch reaches its end of life and is no longer supported. Currently Supported Versions Branch Initial Release Active Support Until Security Support Until 5.6   * 28 Aug 2014 2 years, 8 months ago 19 Jan 2017 4 months ago 31 Dec 2018 in 1 year, 7 months 7.0 3 Dec 2015 1 year, 5 months ago 3 Dec 2017 in 6 months 3 Dec 2018 in 1 year, 6 months 7.1 1 Dec 2016 5 months ago 1 Dec 2018 in 1 year, 6 months 1 Dec 2019 in ...
Post a Comment
Read more