mongodb/buildscripts/smoke

new smoke module README

CURRENTLY IN ACTIVE DEVELOPMENT

This directory provides a POC implementation of a new test runner.  Features include:

    - Test metadata and test tagging
    - Pluggable and isolated test APIs...
    - ...for different test types
    - Simple JSON/YAML (re)configuration

RUNNING:

For command line options invoke:

    $ ./buildscripts/resmoke.py --help

The smoke test runner is completely configured by a JSON/YAML configuration - this configuration can either be loaded from file(s) or built at the command line using --set/--unset/--push manipulations to "nested.path.specifiers".  For basic testing this isn't necessary however, configuration files have already been provided and are available using special command line options, as shown below.

Some simple examples:

Run a MongoDB test suite:

    $ ./buildscripts/resmoke.py --jscore
    $ ./buildscripts/resmoke.py --disk
    $ ./buildscripts/resmoke.py --sharding
    $ ./buildscripts/resmoke.py --replicasets
    (more to come)

To run selected files inside a MongoDB test suite:

    $ ./buildscripts/resmoke.py --jscore jstests/core/count.js
    $ ./buildscripts/resmoke.py --disk jstests/disk/b*.js
    $ ./buildscripts/resmoke.py --sharding jstests/sharding/addshard1.js jstests/sharding/addshard2.js

To run a suite with authentication:

    $ ./buildscripts/resmoke.py --jscore --auth
    $ ./buildscripts/resmoke.py --sharding --auth_shell

NOTE: You may need to change permissions for the jstests/libs/authTestsKey.

To run the core suite with master/slave replication (small oplog):

    $ ./buildscripts/resmoke.py --jscore --master_slave
    $ ./buildscripts/resmoke.py --jscore --master_slave --auth

By default, the output of smoke testing goes to files.  This can be changed, however:

    $ ./buildscripts/resmoke.py --disk --log_console
    $ ./buildscripts/resmoke.py --jscore --log_suppress

Sometimes we may want to set custom options while running the standard suites:

    $ ./buildscripts/resmoke.py --jscore \
        --set "executor.fixtures.mongodb_server.mongod_options.noprealloc" ""

... or change the dbpath of the mongod fixture:

    $ ./buildscripts/resmoke.py --jscore \
        --set "executor.fixtures.mongodb_server.mongod_options.dbpath" "/data/db/mypath"

... or change the executables used by the mongod fixture and the shell:

    $ ./buildscripts/resmoke.py --jscore \
        --set "executor.fixtures.mongodb_server.mongod_executable" "mongod-2.6" \
        --set "executor.testers.js_test.shell_executable" "mongo-2.6"

... or change verbosity of the mongod fixture:

    $ ./buildscripts/resmoke.py --jscore \
        --set "executor.fixtures.mongodb_server.mongod_options.verbose" 2

... or change the value of a server parameter:

    $ ./buildscripts/resmoke.py --jscore \
        --set "executor.fixtures.mongodb_server.mongod_options.set_parameters.enableLocalhostAuthBypass" "false"

... or set some auth parameters:

    $ ./buildscripts/resmoke.py --jscore --auth \
        --set "executor.fixtures.mongodb_server.mongod_options.keyFile" "myKey" \
        --set "executor.fixtures.mongodb_server.mongod_options.setParameters.enableLocalhostAuthBypass" false \
        --set "executor.fixtures.shell_globals.TestData.keyFile" "myKey"

This can quickly get wordy, with lots of parameters.  However, if this is a configuration you plan on testing repeatedly:
    
    $ mkdir -p ~/.smoke_config
    $ ./buildscripts/resmoke.py [all your options and args here] --dump-options > ~/.smoke_config/my_auth.yaml
    $ ./buildscripts/resmoke.py --my_auth

Note that you can also pipe config file data *into* resmoke.py if you'd like to toy with custom config processing.

As you can see, "special" options to resmoke.py are actually just .json/.yaml option files.  The "smoke_config" module provides access to the default suite .json/.yaml files, and you can add/override to these option files in your local user ".smoke_config" directory.  Equivalently you can use the '--config-file' option to load a file not in the special directories.

Also, considering running individual files in a suite:

    $ ./buildscripts/resmoke.py --jscore jstests/core/count.js

This is just shorthand for overriding the "tests.roots" option with the specified files:

    $ ./buildscripts/resmoke.py --jscore --set "tests.roots" "jstests/core/count.js"

TEST METADATA:

Test metadata comes from two sources - embedded in the test files themselves and in a special "test_metadata.json" sibling file on the same path as the test file (by default).  For jstests, the "test_metadata.json" file isn't really necessary to manage - but for executable tests that are not inspectable (unittests, dbtests) an external metadata file is needed.

For jstests things are generally simpler.  Tags can be added to jstests and will be parsed (by default) when resmoke.py builds the test database from the test roots.  These tags have the following form, at the beginning of a jstest file:

    /**
     * @tags : [ mytagA, mytagB ]
     */

Note that the tags array must be well-formed YAML.

These tags are then available for test filtering:

    $ ./buildscripts/resmoke.py --jscore --set suite.include_tags '^mytagA$'
    $ ./buildscripts/resmoke.py --disk --set suite.exclude_tags '^assumes_memmapped$'

NOTE: smoke.py has also been instrumented to manage basic jstest tags, with the following syntax:

    $ ./buildscripts/smoke.py jsCore --include-tags '^mytagA$'

TEST METADATA BUILDING:

For automated / repeated testing, sometimes it isn't desirable to scan every test file for changes to test metadata.  The "tests.extract_metadata" option controls this behavior.  An example script to extract metadata in one shot (which can then be used for many test runs without further extraction) is available at:

    $ ./buildscripts/resmoke_build_metadata.py --jscore

Note that the example script uses the same kind of options as the resmoke.py script.

INTEGRATION WITH OTHER TOOLS:

To use test database, suite extraction, and suite execution functionality in other tools (like SConscript), import the "smoke" module.  This provides:

- smoke.tests: test discovery, metadata load/save
- smoke.suite: test filtering by tags
- smoke.executor: test execution with custom fixtures and logging