mongodb/docs/evergreen-testing/multiversion.md

Multiversion Testing

Table of contents

• Multiversion Testing
  • Table of contents
  • Terminology and overview
    • Introduction
    • Latest vs last-lts vs last-continuous
    • Old vs new
    • Explicit and Implicit multiversion suites
    • Version combinations
  • Working with multiversion tasks in Evergreen
    • Exclude tests from multiversion testing
    • Multiversion task generation

Terminology and overview

Introduction

Some tests test specific upgrade/downgrade behavior expected between different versions of MongoDB. Several versions of MongoDB are spun up during those test runs.

• Multiversion suites - resmoke suites that are running tests with several versions of MongoDB.

• Multiversion tasks - Evergreen tasks that are running multiversion suites. Multiversion tasks in most cases include multiversion or downgrade in their names.

Latest vs last-lts vs last-continuous

For some of the versions we are using such generic names as latest, last-lts and last-continuous.

• latest - the current version. In Evergreen, the version that was compiled in the current build.

• last-lts - the latest LTS (Long Term Support) Major release version. In Evergreen, the version that was downloaded from the last LTS release branch project.

• last-continuous - the latest Rapid release version. In Evergreen, the version that was downloaded from the Rapid release branch project.

Old vs new

Many multiversion tasks are running tests against latest/last-lts or latest/last-continuous versions. In such context we refer to last-lts and last-continuous versions as the old version and to latest as a new version.

A new version is compiled in the same way as for non-multiversion tasks. The old versions of compiled binaries are downloaded from the old branch projects with db-contrib-tool. db-contrib-tool searches for the latest available compiled binaries on the old branch projects in Evergreen.

Explicit and Implicit multiversion suites

Multiversion suites can be explicit and implicit.

• Explicit - JS tests are aware of the binary versions they are running, e.g. multiversion.yml. The version of binaries is explicitly set in JS tests, e.g. jstests/multiVersion/genericSetFCVUsage/major_version_upgrade.js:

const versions = [
  {
    binVersion: "4.4",
    featureCompatibilityVersion: "4.4",
    testCollection: "four_four",
  },
  {
    binVersion: "5.0",
    featureCompatibilityVersion: "5.0",
    testCollection: "five_zero",
  },
  {
    binVersion: "6.0",
    featureCompatibilityVersion: "6.0",
    testCollection: "six_zero",
  },
  {
    binVersion: "last-lts",
    featureCompatibilityVersion: lastLTSFCV,
    testCollection: "last_lts",
  },
  {
    binVersion: "last-continuous",
    featureCompatibilityVersion: lastContinuousFCV,
    testCollection: "last_continuous",
  },
  {
    binVersion: "latest",
    featureCompatibilityVersion: latestFCV,
    testCollection: "latest",
  },
];

• Implicit - JS tests know nothing about the binary versions they are running, e.g. retryable_writes_downgrade.yml. Most of the implicit multiversion suites are using matrix suites, e.g. replica_sets_last_lts:

$ python buildscripts/resmoke.py suiteconfig --suite=replica_sets_last_lts

executor:
  config:
    shell_options:
      global_vars:
        TestData:
          useRandomBinVersionsWithinReplicaSet: last-lts
      nodb: ''
selector:
  exclude_files:
  - jstests/replsets/initial_sync_rename_collection.js
  - jstests/replsets/initial_sync_drop_collection.js
  - jstests/replsets/apply_prepare_txn_write_conflict_robustness.js
  - jstests/replsets/invalidate_sessions_on_stepdown.js
  - jstests/replsets/initial_sync_fails_unclean_restart.js
  exclude_with_any_tags:
  - multiversion_incompatible
  - backport_required_multiversion
  - replica_sets_multiversion_backport_required_multiversion
  - disabled_for_fcv_6_1_upgrade
  roots:
  - jstests/replsets/*.js
test_kind: js_test

In implicit multiversion suites the version of binaries is defined on the resmoke fixture level.

The example of replica set fixture configuration override:

fixture:
  num_nodes: 3
  old_bin_version: last_lts
  mixed_bin_versions: new_new_old

The example of sharded cluster fixture configuration override:

fixture:
  num_shards: 2
  num_rs_nodes_per_shard: 2
  old_bin_version: last_lts
  mixed_bin_versions: new_old_old_new

The example of shell fixture configuration override:

value:
  executor:
    config:
      shell_options:
        global_vars:
          TestData:
            useRandomBinVersionsWithinReplicaSet: "last-lts"

Version combinations

In implicit multiversion suites the same set of tests may run in similar suites that are using various mixed version combinations. Those version combinations depend on the type of resmoke fixture the suite is running with. These are the recommended version combinations to test against based on the suite fixtures:

• Replica set fixture combinations:

  • last-lts new-new-old (i.e. suite runs the replica set fixture that spins up the latest and the last-lts versions in a 3-node replica set where the 1st node is the latest, 2nd - latest, 3rd - last-lts, etc.)
  • last-lts new-old-new
  • last-lts old-new-new
  • last-continuous new-new-old
  • last-continuous new-old-new
  • last-continuous old-new-new
  • Ex: change_streams uses a ReplicaSetFixture so the corresponding multiversion suites are
    • change_streams_last_continuous_new_new_old
    • change_streams_last_continuous_new_old_new
    • change_streams_last_continuous_old_new_new
    • change_streams_last_lts_new_new_old
    • change_streams_last_lts_new_old_new
    • change_streams_last_lts_old_new_new

• Sharded cluster fixture combinations:

  • last-lts new-old-old-new (i.e. suite runs the sharded cluster fixture that spins up the latest and the last-lts versions in a sharded cluster that consists of 2 shards with 2-node replica sets per shard where the 1st node of the 1st shard is the latest, 2nd node of 1st shard - last-lts, 1st node of 2nd shard - last-lts, 2nd node of 2nd shard - latest, etc.)
  • last-continuous new-old-old-new
  • Ex: change_streams_downgrade uses a ShardedClusterFixture so the corresponding multiversion suites are
    • change_streams_downgrade_last_continuous_new_old_old_new
    • change_streams_downgrade_last_lts_new_old_old_new

• Shell fixture combinations:

  • last-lts (i.e. suite runs the shell fixture that spins up last-lts as the old versions, etc.)
  • last-continuous
  • Ex: initial_sync_fuzzer uses a Shell Fixture, so the corresponding multiversion suites are
    • initial_sync_fuzzer_last_lts
    • initial_sync_fuzzer_last_continuous

If last-lts and last-continuous versions happen to be the same, or last-continuous is EOL, we skip last-continuous and run multiversion suites with only last-lts combinations in Evergreen.

Working with multiversion tasks in Evergreen

Multiversion task generation

Please refer to mongo-task-generator documentation for generating multiversion tasks in Evergreen.

Exclude tests from multiversion testing

Sometimes tests are not designed to run in multiversion suites. To avoid implicit multiversion suites to pick up the test multiversion_incompatible tag can be added to the test, e.g. jstests/concurrency/fsm_workloads/drop_database_sharded_setFCV.js.

// @tags: [multiversion_incompatible]

There is additional requires_fcv_XX set of tags that can be used to disable tests from running in multiversion where XX is the version number, e.g. requires_fcv_61 stands for version 6.1 in jstests/serverless/change_collection_expired_document_remover.js.

// @tags: [requires_fcv_61]

Tests with requires_fcv_XX tags are excluded from multiversion tasks that may run the versions below the specified FCV version, e.g. when the latest version is 6.2, last-continuous is 6.1 and last-lts is 6.0, tests tagged with requires_fcv_61 will NOT run in multiversion tasks that run latest with last-lts, but will run in multiversion tasks that run lastest with last-continuous.

In addition to disabling multiversion tests based on FCV, there is no need to run in-development featureFlagXYZ tests (featureFlags that have default: false) because these tests will most likely fail on older versions that have not implemented this feature. For multiversion tasks, we pass the --runNoFeatureFlagTests flag to avoid these failures on all feature flag variants.

For more info on FCV, take a look at FCV_AND_FEATURE_FLAG_README.md.

Another common case could be that the changes on master branch are breaking multiversion tests, but with those changes backported to the older branches the multiversion tests should work. In order to temporarily disable the test from running in multiversion it can be added to the etc/backports_required_for_multiversion_tests.yml. Please follow the instructions described in the file.