diff --git a/docs/best-practices.md b/docs/best-practices.md index 180f815ca..a3e5f2629 100644 --- a/docs/best-practices.md +++ b/docs/best-practices.md @@ -84,19 +84,19 @@ Use the _Foo.camelCase_ naming convention, which is similar to _Classes_ and _cl Use an upper case first letter for: -- File names. Example: _StorefrontCreateCustomerTest.xml_ -- Test name attributes. Example: ``. -- Data entity names. Example: ``. -- Page name. Example: ``. -- Section name. Example: `
`. -- Action group name. Example: ``. +- File names. Example: _StorefrontCreateCustomerTest.xml_ +- Test name attributes. Example: ``. +- Data entity names. Example: ``. +- Page name. Example: ``. +- Section name. Example: `
`. +- Action group name. Example: ``. #### Lower case Use a lower case first letter for: -- Data keys. Example: ``. -- Element names. Examples: ``. +- Data keys. Example: ``. +- Element names. Examples: ``. ## Page object @@ -134,9 +134,9 @@ Define these three elements and reference them by name in the tests. 1. Keep your tests short and granular for target testing, easier reviews, and easier merge conflict resolution. It also helps you to identify the cause of test failure. 1. Use comments to keep tests readable and maintainable: - - Keep the inline `` and [``] tags up to date. + - Keep the inline `` and [``] tags up to date. It helps to inform the reader of what you are testing and to yield a more descriptive Allure report. - - Explain in comments unclear or tricky test steps. + - Explain in comments unclear or tricky test steps. 1. Refer to [sections] instead of writing selectors. ## Test step merging order diff --git a/docs/credentials.md b/docs/credentials.md index 101711ee6..3065454b0 100644 --- a/docs/credentials.md +++ b/docs/credentials.md @@ -148,8 +148,8 @@ Credentials can be used in actions: [`fillField`][], [`magentoCLI`][], and [`cre Define the value as a reference to the corresponding key in the credentials file or vault such as `{{_CREDS.my_data_key}}`: -- `_CREDS` is an environment constant pointing to the `.credentials` file -- `my_data_key` is a key in the the `.credentials` file or vault that contains the value to be used in a test step +- `_CREDS` is an environment constant pointing to the `.credentials` file +- `my_data_key` is a key in the the `.credentials` file or vault that contains the value to be used in a test step For example, reference secret data in the [`fillField`][] action with the `userInput` attribute. diff --git a/docs/data.md b/docs/data.md index 4c618ae45..2a1b845d4 100644 --- a/docs/data.md +++ b/docs/data.md @@ -17,8 +17,8 @@ userInput="{{SimpleSubCategory.name}}" In this example: -* `SimpleSubCategory` is an entity name. -* `name` is a `` key of the entity. The corresponding value will be assigned to `userInput` as a result. +* `SimpleSubCategory` is an entity name. +* `name` is a `` key of the entity. The corresponding value will be assigned to `userInput` as a result. ### Environmental data @@ -28,8 +28,8 @@ userInput="{{_ENV.MAGENTO_ADMIN_USERNAME}}" In this example: -* `_ENV` is a reference to the `dev/tests/acceptance/.env` file, where basic environment variables are set. -* `MAGENTO_ADMIN_USERNAME` is a name of an environment variable. +* `_ENV` is a reference to the `dev/tests/acceptance/.env` file, where basic environment variables are set. +* `MAGENTO_ADMIN_USERNAME` is a name of an environment variable. The corresponding value will be assigned to `userInput` as a result. ### Sensitive data @@ -40,10 +40,10 @@ userInput="{{_CREDS.my_secret_token}}" In this example: -* `_CREDS` is a constant to reference to the `dev/tests/acceptance/.credentials` file, where sensitive data and secrets are stored for use in a test. -* `MY_SECRET_TOKEN` is the name of a key in the credentials variable. +* `_CREDS` is a constant to reference to the `dev/tests/acceptance/.credentials` file, where sensitive data and secrets are stored for use in a test. +* `MY_SECRET_TOKEN` is the name of a key in the credentials variable. The corresponding value of the credential will be assigned to `userInput` as a result. -* The decrypted values are only available in the `.credentials` file in which they are stored. +* The decrypted values are only available in the `.credentials` file in which they are stored. Learn more in [Credentials][]. @@ -59,8 +59,8 @@ userInput="$createCustomer.email$" In this example: -* `createCustomer` is a step key of the corresponding test step that creates an entity. -* `email` is a data key of the entity. +* `createCustomer` is a step key of the corresponding test step that creates an entity. +* `email` is a data key of the entity. The corresponding value will be assigned to `userInput` as a result.
@@ -118,10 +118,10 @@ The format of `` is: The following conventions apply to MFTF ``: -* A `` file may contain multiple data entities. -* Camel case is used for `` elements. The name represents the `` type. For example, a file with customer data is `CustomerData.xml`. A file for simple product would be `SimpleProductData.xml`. -* Camel case is used for the entity name. -* The file name must have the suffix `Data.xml`. +* A `` file may contain multiple data entities. +* Camel case is used for `` elements. The name represents the `` type. For example, a file with customer data is `CustomerData.xml`. A file for simple product would be `SimpleProductData.xml`. +* Camel case is used for the entity name. +* The file name must have the suffix `Data.xml`. ## Example @@ -152,16 +152,16 @@ All entities that have the same name will be merged during test generation. Both `_defaultCategory` sets three data fields: -* `name` defines the category name as `simpleCategory` with a unique suffix. Example: `simpleCategory598742365`. -* `name_lwr` defines the category name in lowercase format with a unique suffix. Example: `simplecategory697543215`. -* `is_active` sets the enable category to `true`. +* `name` defines the category name as `simpleCategory` with a unique suffix. Example: `simpleCategory598742365`. +* `name_lwr` defines the category name in lowercase format with a unique suffix. Example: `simplecategory697543215`. +* `is_active` sets the enable category to `true`. `SimpleSubCategory` sets four data fields: -* `name` that defines the category name with a unique suffix. Example: `SimpleSubCategory458712365`. -* `name_lwr` that defines the category name in lowercase format with a unique suffix. Example: `simplesubcategory753698741`. -* `is_active` sets the enable category to `true`. -* `include_in_menu` that sets the include in the menu to `true`. +* `name` that defines the category name with a unique suffix. Example: `SimpleSubCategory458712365`. +* `name_lwr` that defines the category name in lowercase format with a unique suffix. Example: `simplesubcategory753698741`. +* `is_active` sets the enable category to `true`. +* `include_in_menu` that sets the include in the menu to `true`. The following is an example of a call in test: diff --git a/docs/debugging.md b/docs/debugging.md index 4b839ff6e..be17e952a 100644 --- a/docs/debugging.md +++ b/docs/debugging.md @@ -2,15 +2,15 @@ Debugging within the Magento Functional Testing Framework is helpful in identifying test bugs by allowing you to pause execution so that you may: -- Examine the page. -- Check returned data and other variables being used during run-time. +- Examine the page. +- Check returned data and other variables being used during run-time. This is straightforward to do once you create a basic Debug Configuration. ## Prerequisites -- [Xdebug][] -- PHPUnit configured for use in [PHPStorm][] +- [Xdebug][] +- PHPUnit configured for use in [PHPStorm][] ## Creating Debug Configuration with PHPStorm @@ -27,8 +27,8 @@ If you get a warning `Path to Codeception for local machine is not configured.`: The easiest method of tagging a test for debugging is the following: -- In your Debug configuration, locate `Test Runner options:` and set `--group testDebug`. -- When you want to debug a test you are working on, simply add `` to the annotations. Be sure to remove this after done debugging. +- In your Debug configuration, locate `Test Runner options:` and set `--group testDebug`. +- When you want to debug a test you are working on, simply add `` to the annotations. Be sure to remove this after done debugging. Your Debug Configuration should now be able to run your test and pause execution on any breakpoints you have set in the generated `.php` file under the `_generated` folder. diff --git a/docs/extending.md b/docs/extending.md index 666121986..91dc97c5d 100644 --- a/docs/extending.md +++ b/docs/extending.md @@ -5,9 +5,9 @@ For example, only one or two parameters (for example, URL) might vary between te To avoid copy-pasting and to save some time the Magento Functional Testing Framework (MFTF) enables you to extend test components such as [test], [data], and [action group]. You can create or update any component of the parent body in your new test/action group/entity. -* A test starting with `` creates a test `SampleTest` that takes body of existing test `ParentTest` and adds to it the body of `SampleTest`. -* An action group starting with `` creates an action group based on the `ParentActionGroup`, but with the changes specified in `SampleActionGroup`. -* An entity starting with `` creates an entity `SampleEntity` that is equivalent to merging the `SampleEntity` with the `ParentEntity`. +* A test starting with `` creates a test `SampleTest` that takes body of existing test `ParentTest` and adds to it the body of `SampleTest`. +* An action group starting with `` creates an action group based on the `ParentActionGroup`, but with the changes specified in `SampleActionGroup`. +* An entity starting with `` creates an entity `SampleEntity` that is equivalent to merging the `SampleEntity` with the `ParentEntity`. Specify needed variations for a parent object and produce a copy of the original that incorporates the specified changes (the "delta"). @@ -71,8 +71,8 @@ __Use case__: Create two similar tests with different `url` (`"{{AdminCategoryPa __Use case__: Create two similar tests where the second test contains two additional steps: -* `checkOption` before `click` (`stepKey="clickLogin"`) -* `seeInCurrentUrl` after `click` in the `LogInAsAdminTest` test (in the `.../Backend/Test/LogInAsAdminTest.xml` file) +* `checkOption` before `click` (`stepKey="clickLogin"`) +* `seeInCurrentUrl` after `click` in the `LogInAsAdminTest` test (in the `.../Backend/Test/LogInAsAdminTest.xml` file) > Tests with "extends": @@ -119,8 +119,8 @@ __Use case__: Create two similar tests where the second test contains two additi __Use case__: Create two similar tests where the second one contains two additional actions in the `before` hook: -* `checkOption` before `click` (`stepKey="clickLogin"`) -* `seeInCurrentUrl` after `click` in the `LogInAsAdminTest` test (in the `.../Backend/Test/LogInAsAdminTest.xml` file) +* `checkOption` before `click` (`stepKey="clickLogin"`) +* `seeInCurrentUrl` after `click` in the `LogInAsAdminTest` test (in the `.../Backend/Test/LogInAsAdminTest.xml` file) > Tests with "extends": @@ -366,4 +366,4 @@ __Use case__: Create an entity named `DivPanelGreen`, which is similar to the `D [test]: ./test.md [data]: ./data.md [action group]: ./test/action-groups.md -[actions]: ./test/actions.md \ No newline at end of file +[actions]: ./test/actions.md diff --git a/docs/getting-started.md b/docs/getting-started.md index 450dbc2a6..626cca31b 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -10,10 +10,10 @@ The latest Magento 2.2 release supports MFTF 2.3.8. Make sure that you have the following software installed and configured on your development environment: -- [PHP version supported by the Magento instance under test][php] -- [Composer 1.3 or later][composer] -- [Java 1.8 or later][java] -- [Selenium Server Standalone 3.1 or later][selenium server] and [ChromeDriver 2.33 or later][chrome driver] or other webdriver in the same directory +- [PHP version supported by the Magento instance under test][php] +- [Composer 1.3 or later][composer] +- [Java 1.8 or later][java] +- [Selenium Server Standalone 3.1 or later][selenium server] and [ChromeDriver 2.33 or later][chrome driver] or other webdriver in the same directory
[PhpStorm] supports [Codeception test execution][], which is helpful when debugging. @@ -147,16 +147,16 @@ vim dev/tests/acceptance/.env Specify the following parameters, which are required to launch tests: -- `MAGENTO_BASE_URL` must contain a domain name of the Magento instance that will be tested. +- `MAGENTO_BASE_URL` must contain a domain name of the Magento instance that will be tested. Example: `MAGENTO_BASE_URL=http://magento.test` -- `MAGENTO_BACKEND_NAME` must contain the relative path for the Admin area. +- `MAGENTO_BACKEND_NAME` must contain the relative path for the Admin area. Example: `MAGENTO_BACKEND_NAME=admin` -- `MAGENTO_ADMIN_USERNAME` must contain the username required for authorization in the Admin area. +- `MAGENTO_ADMIN_USERNAME` must contain the username required for authorization in the Admin area. Example: `MAGENTO_ADMIN_USERNAME=admin` -- `MAGENTO_ADMIN_PASSWORD` must contain the user password required for authorization in the Admin area. +- `MAGENTO_ADMIN_PASSWORD` must contain the user password required for authorization in the Admin area. Example: `MAGENTO_ADMIN_PASSWORD=123123q`
@@ -222,8 +222,8 @@ During testing, the MFTF generates test reports in CLI. You can generate visual representations of the report data using [Allure Framework][]. To view the reports in GUI: -- [Install Allure][] -- Run the tool to serve the artifacts in `dev/tests/acceptance/tests/_output/allure-results/`: +- [Install Allure][] +- Run the tool to serve the artifacts in `dev/tests/acceptance/tests/_output/allure-results/`: ```bash allure serve dev/tests/acceptance/tests/_output/allure-results/ diff --git a/docs/img/catalogCategoryRepository-operations.png b/docs/img/catalogCategoryRepository-operations.png index 5db7b7008..e6fa806bb 100644 Binary files a/docs/img/catalogCategoryRepository-operations.png and b/docs/img/catalogCategoryRepository-operations.png differ diff --git a/docs/img/issue.png b/docs/img/issue.png index 8a2941f4c..1dcf78ce2 100644 Binary files a/docs/img/issue.png and b/docs/img/issue.png differ diff --git a/docs/img/mftf-fork.gif b/docs/img/mftf-fork.gif index 650687b06..6ea138cce 100644 Binary files a/docs/img/mftf-fork.gif and b/docs/img/mftf-fork.gif differ diff --git a/docs/img/pull-request.png b/docs/img/pull-request.png index 66bff3f63..c492350d8 100644 Binary files a/docs/img/pull-request.png and b/docs/img/pull-request.png differ diff --git a/docs/img/switching-the-base.png b/docs/img/switching-the-base.png index 5b6653d36..412ab1624 100644 Binary files a/docs/img/switching-the-base.png and b/docs/img/switching-the-base.png differ diff --git a/docs/img/trouble-chrome232.png b/docs/img/trouble-chrome232.png index 4af49146c..1dc44f6a4 100644 Binary files a/docs/img/trouble-chrome232.png and b/docs/img/trouble-chrome232.png differ diff --git a/docs/introduction.md b/docs/introduction.md index 1c7bfd329..61c1eaef3 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -5,11 +5,11 @@ The Magento Functional Testing Framework (MFTF) aims to replace the [Functional Testing Framework] in future releases. MFTF improves: -- **Traceability** for clear logging and reporting capabilities. -- **Modularity** to run tests based on installed modules and extensions. -- **Customizability** for existing tests. -- **Readability** using clear and declarative XML test steps. -- **Maintainability** based on simple test creation and overall structure. +- **Traceability** for clear logging and reporting capabilities. +- **Modularity** to run tests based on installed modules and extensions. +- **Customizability** for existing tests. +- **Readability** using clear and declarative XML test steps. +- **Maintainability** based on simple test creation and overall structure. Because MFTF tests are written in XML, you no longer need to learn PHP to write tests. @@ -27,17 +27,17 @@ This MFTF guide is intended for Magento developers and software engineers, such The purpose of MFTF is to: -- Facilitate functional testing and minimize the effort it takes to perform regression testing. -- Make it easier to support the extension and customization of tests via XML merging. +- Facilitate functional testing and minimize the effort it takes to perform regression testing. +- Make it easier to support the extension and customization of tests via XML merging. ## Scope MFTF will enable you to: -- Test user interactions with web applications in testing. -- Write functional tests located in `/app/code///Test/Mftf/`. -- Cover basic functionality using out-of-the-box tests. You can test extended functionality using custom tests. -- Automate regression testing. +- Test user interactions with web applications in testing. +- Write functional tests located in `/app/code///Test/Mftf/`. +- Cover basic functionality using out-of-the-box tests. You can test extended functionality using custom tests. +- Automate regression testing. ## Use cases @@ -49,8 +49,8 @@ As a software engineer, perform regression testing before release to ensure that There are two options to find out your MFTF version: -- using the MFTF CLI -- using the Composer CLI +- using the MFTF CLI +- using the Composer CLI ### MFTF CLI @@ -91,18 +91,18 @@ codeception.dist.yml // Codeception configuration (generated while ru ## MFTF output -- Generated PHP Codeception tests -- Codeception results and console logs -- Screenshots and HTML failure report -- Allure formatted XML results -- Allure report dashboard of results +- Generated PHP Codeception tests +- Codeception results and console logs +- Screenshots and HTML failure report +- Allure formatted XML results +- Allure report dashboard of results ## MFTF tests The MFTF supports two different locations for storing the tests and test artifacts: -- `/app/code///Test/Mftf/` is the directory to create new tests. -- `/vendor///Test/Mftf/` is the directory with the out of the box tests (fetched by the Composer). +- `/app/code///Test/Mftf/` is the directory to create new tests. +- `/vendor///Test/Mftf/` is the directory with the out of the box tests (fetched by the Composer). All tests and test data from these locations are merged in the order indicated in the above list. @@ -133,4 +133,4 @@ Follow the [MFTF project] and [contribute on Github]. [Functional Testing Framework]: https://devdocs.magento.com/guides/v2.3/mtf/mtf_introduction.html [MFTF project]: https://github.com/magento/magento2-functional-testing-framework [Find your MFTF version]: #find-your-mftf-version -[MFTF Test Migration]: https://github.com/magento/magento-functional-tests-migration \ No newline at end of file +[MFTF Test Migration]: https://github.com/magento/magento-functional-tests-migration diff --git a/docs/merging.md b/docs/merging.md index 32ea966c3..99df18464 100644 --- a/docs/merging.md +++ b/docs/merging.md @@ -2,11 +2,11 @@ The MFTF allows you to merge test components defined in XML files, such as: -- [``][] -- [``][] -- [``][] -- [``][] -- `` +- [``][] +- [``][] +- [``][] +- [``][] +- `` You can create, delete, or update the component. It is useful for supporting rapid test creation for extensions and customizations. @@ -18,11 +18,11 @@ Your update (XML node with changes) must have the same attribute `name` as its b For example: -- All tests with `` will be merged into one. -- All sections with `
` will be merged into one. -- All data entities with `` will be merged into one. -- All action groups with `` will be merged into one. +- All tests with `` will be merged into one. +- All sections with `
` will be merged into one. +- All data entities with `` will be merged into one. +- All action groups with `` will be merged into one. Although a file name does not influence merging, we recommend using the same file names in merging updates. This makes it easier to search later on. diff --git a/docs/metadata.md b/docs/metadata.md index 91ecbe2e9..795c8a038 100644 --- a/docs/metadata.md +++ b/docs/metadata.md @@ -4,9 +4,9 @@ In this topic we talk about handling entities that you need in your tests (such Using data handling actions like [`createData`], [`deleteData`], [`updateData`], and [`getData`], you are able to create, delete, update, and read entities for your tests. The framework enables you to send HTTP requests with these statically defined data entities: -- [Sending a REST API request][rest request] -- [Handling a REST API response][rest response] -- [Sending an HTML form encoded in URL][html form] +- [Sending a REST API request][rest request] +- [Handling a REST API response][rest response] +- [Sending an HTML form encoded in URL][html form] You have probably noticed that some modules in acceptance functional tests contain a directory, which is called `Metadata`. @@ -25,16 +25,16 @@ This directory contains XML files with metadata required to create a valid reque A metadata file contains a list of operations with different types (defined in `type`). Each [operation] includes: -- The set of adjustments for processing a request in [attributes][operation], and in some cases, a response (see `successRegex`, `returnRegex` and `returnIndex` in [reference details][operation]). -- The type of body content encoding in [contentType]. -- The body of the request represented as a tree of objects, arrays, and fields. +- The set of adjustments for processing a request in [attributes][operation], and in some cases, a response (see `successRegex`, `returnRegex` and `returnIndex` in [reference details][operation]). +- The type of body content encoding in [contentType]. +- The body of the request represented as a tree of objects, arrays, and fields. When a test step requires handling the specified data entity, the MFTF performs the following steps: -- Reads input data (``) and the type (the `type` attribute) of the specified [entity]. -- Searches the metadata operation for the `dataType` that matches the entity's `type`. For example, `` matches ``) and the type (the `type` attribute) of the specified [entity]. +- Searches the metadata operation for the `dataType` that matches the entity's `type`. For example, `` matches ``: -- `name="CreateCategory"` defines a descriptive name of the operation, which is used for merging if needed. -- `dataType="category"` defines a relation with data entities with input data for a Category (``). -- `auth="adminOauth"` defines OAuth authorization, which is required for the Admin area. -- `url="/V1/categories"` defines a routing URL to the corresponding service class. +- `name="CreateCategory"` defines a descriptive name of the operation, which is used for merging if needed. +- `dataType="category"` defines a relation with data entities with input data for a Category (``). +- `auth="adminOauth"` defines OAuth authorization, which is required for the Admin area. +- `url="/V1/categories"` defines a routing URL to the corresponding service class. (The request will be sent to `https://example.com/rest/V1/categories` if `MAGENTO_BASE_URL=https://example.com/` and `MAGENTO_BACKEND_NAME=admin` are set in the _acceptance/.env_ configuration file.) -- `method="POST"` defines a POST method of the HTTP request. +- `method="POST"` defines a POST method of the HTTP request. `application/json` defines a content type of the REST API request, which is set as `application/json` here. @@ -333,8 +333,8 @@ The following example of response in JSON demonstrates how to reference data on For cases when REST API is not applicable, you may use [HTML forms] (when all object parameters are encoded in a URL as `key=name` attributes). There are two different attributes to split access to different areas: -- `auth="adminFormKey"` is used for objects in an Admin area. -- `auth="customerFormKey"` is used for objects in a storefront. +- `auth="adminFormKey"` is used for objects in an Admin area. +- `auth="customerFormKey"` is used for objects in a storefront. You are able to create assurances with `successRegex`, and, optionally, return values with `returnRegex`. You can also use `returnIndex` when `returnRegex` matches multiple values. @@ -366,13 +366,13 @@ If the returned HTML code contains the `messages-message-success` string, it is The operation enables you to assign the following form fields: -- `group/group_id` -- `group/name` -- `group/code` -- `group/root_category_id` -- `group/website_id` -- `store_action` -- `store_type` +- `group/group_id` +- `group/name` +- `group/code` +- `group/root_category_id` +- `group/website_id` +- `store_action` +- `store_type` ### Create an object in storefront {#create-object-as-customerFormKey} @@ -396,9 +396,9 @@ If the returned HTML code contains a string that matches the regular expression The operation assigns three form fields: -- `product` -- `customer_email` -- `customer_password` +- `product` +- `customer_email` +- `customer_password` ## Reference @@ -422,24 +422,24 @@ Root element that points to the corresponding XML Schema. | `removeBackend` | boolean | optional | Removes backend name from requested URL. Applicable when `auth="adminFormKey"`. | | `filename` | string | optional | | -- \*`url` - full URL is a concatenation of _ENV.baseUrl_ + `/rest/` + _url_. +- \*`url` - full URL is a concatenation of _ENV.baseUrl_ + `/rest/` + _url_. To reuse data of a required entity or returned response use a field key wrapped in curly braces such as `{sku}`. When the data to reuse is of a different type, declare also the type of data such as `{product.sku}`. Example: `"/V1/products/{product.sku}/media/{id}"`. -- \*\*`auth` - available values: +- \*\*`auth` - available values: - - `adminOath` is used for REST API persistence in the Admin area with [OAuth-based authentication][oauth]. - - `adminFormKey` is used for HTML form persistence in the Admin area. - - `customerFormKey` is used for HTML form persistence in the Customer area. - - `anonymous` is used for REST API persistence without authorization. + - `adminOath` is used for REST API persistence in the Admin area with [OAuth-based authentication][oauth]. + - `adminFormKey` is used for HTML form persistence in the Admin area. + - `customerFormKey` is used for HTML form persistence in the Customer area. + - `anonymous` is used for REST API persistence without authorization. ### contentType {#contentType-tag} Sets one of the following operation types: -- `application/json` is used for REST API operations. -- `application/x-www-form-urlencoded` is used for HTML form operations. +- `application/json` is used for REST API operations. +- `application/x-www-form-urlencoded` is used for HTML form operations. ### object {#object-tag} @@ -497,7 +497,7 @@ Data entity with the corresponding assignment: ``` -- Resulted JSON request: +- Resulted JSON request: ```json "tax_rate_ids": diff --git a/docs/mftf-tests.md b/docs/mftf-tests.md index 3c4043d5b..b9d94baa9 100644 --- a/docs/mftf-tests.md +++ b/docs/mftf-tests.md @@ -28,4 +28,4 @@ This page lists all those tests so that developers can have a good sense of what {: .mftf-dl} {% endfor %} {% endfor %} -{% endfor %} \ No newline at end of file +{% endfor %} diff --git a/docs/page.md b/docs/page.md index 52107e525..705a94294 100644 --- a/docs/page.md +++ b/docs/page.md @@ -13,9 +13,9 @@ Two types of pages are available: -- Page with `url` declared as a constant string or [explicit page] - In a test it is called in a format like `{{NameOfPage.url}}`, where `NameOfPage` is a value of `name` in the corresponding page declaration `*.xml` file. -- Page with `url` declared as a string with one or more variables or [parameterized page] -- In a test it is called using a format like `{{NameOfPage.url(var1, var2, ...)}}`, where `var1, var2` etc. are parameters that will be substituted in the `url` of the corresponding `` declaration. +- Page with `url` declared as a constant string or [explicit page] - In a test it is called in a format like `{{NameOfPage.url}}`, where `NameOfPage` is a value of `name` in the corresponding page declaration `*.xml` file. +- Page with `url` declared as a string with one or more variables or [parameterized page] +- In a test it is called using a format like `{{NameOfPage.url(var1, var2, ...)}}`, where `var1, var2` etc. are parameters that will be substituted in the `url` of the corresponding `` declaration. The following diagram shows the XML structure of an MFTF page: @@ -41,10 +41,10 @@ The format of `` is: The following conventions apply to MFTF pages: -- `` name is the same as the file name. -- `` name must be alphanumeric. -- `*Page.xml` is stored in the _Page_ directory of a module. -- The name format is `{Admin|Storefront}{PageDescription}Page.xml`. +- `` name is the same as the file name. +- `` name must be alphanumeric. +- `*Page.xml` is stored in the _Page_ directory of a module. +- The name format is `{Admin|Storefront}{PageDescription}Page.xml`. The `.url` attribute is required when using the page for [actions] that require the URL argument. @@ -77,10 +77,10 @@ The corresponding web page is generated by the Magento Catalog module and is cal The `AdminCategoryPage` declares four [sections][section]: -- `AdminCategorySidebarActionSection` - located in the `Catalog/Section/AdminCategorySidebarActionSection.xml` file -- `AdminCategorySidebarTreeSection` - located in the `Catalog/Section/AdminCategorySidebarTreeSection.xml` file -- `AdminCategoryBasicFieldSection` - located in the `Catalog/Section/AdminCategoryBasicFieldSection.xml` file -- `AdminCategorySEOSection` - located in the `Catalog/Section/AdminCategorySEOSection.xml` file +- `AdminCategorySidebarActionSection` - located in the `Catalog/Section/AdminCategorySidebarActionSection.xml` file +- `AdminCategorySidebarTreeSection` - located in the `Catalog/Section/AdminCategorySidebarTreeSection.xml` file +- `AdminCategoryBasicFieldSection` - located in the `Catalog/Section/AdminCategoryBasicFieldSection.xml` file +- `AdminCategorySEOSection` - located in the `Catalog/Section/AdminCategorySEOSection.xml` file The following is an example of a call in test: diff --git a/docs/reporting.md b/docs/reporting.md index 26c7f8fc9..59a8617ba 100644 --- a/docs/reporting.md +++ b/docs/reporting.md @@ -2,16 +2,16 @@ The Magento Functional Testing Framework provides two types of reporting: -- Inline reporting that you can view in the terminal as you run [`mftf`][mftf] or [`codecept`][codecept] CLI commands. -- HTML reports that you can view using the [Allure Framework][] after a test run completes. +- Inline reporting that you can view in the terminal as you run [`mftf`][mftf] or [`codecept`][codecept] CLI commands. +- HTML reports that you can view using the [Allure Framework][] after a test run completes. When you run a test, MFTF copies all reporting artifacts to the `dev/tests/acceptance/tests/_output` subdirectory in the Magento root directory. The directory contains: -- `allure-results/` that is a directory generated and served by the Allure Framework. -- `failed` that is a text file containing relative paths to failed tests after the last test run. +- `allure-results/` that is a directory generated and served by the Allure Framework. +- `failed` that is a text file containing relative paths to failed tests after the last test run. The paths are relative to `dev/tests/acceptance/`. -- `.html` and `.png` files that are screenshots of fails in HTML and PNG formats. +- `.html` and `.png` files that are screenshots of fails in HTML and PNG formats. To cleanup the `_output/` directory, remove them manually. The `mftf` tool logs output continuously to the `dev/tests/acceptance/mftf.log` file. @@ -22,11 +22,11 @@ MFTF reports about its progress during test run when you run the `mftf` CLI tool The report can contain three main parts: -- Pre-run checks: - - Environment check, such as PHP warnings, etc. - - XML test validation like deprecation warnings such as missing required components in XML tests. -- Codeception report which is the progress report for each test. -- Total results of the test run such as number of tests, assertions, and failures. +- Pre-run checks: + - Environment check, such as PHP warnings, etc. + - XML test validation like deprecation warnings such as missing required components in XML tests. +- Codeception report which is the progress report for each test. +- Total results of the test run such as number of tests, assertions, and failures. To manage the level of verbosity, use `-v` or `--verbose` flag in the `mftf` commands. To enable verbosity using the `codecept` commands, refer to the Codeception [Console Commands][codeception]. @@ -152,18 +152,18 @@ When a test step fails, MFTF always saves a screenshot of the web page with the A screenshot of the fail goes at the `acceptance/tests/_output` directory in both PNG and HTML formats: -- `Magento.AcceptanceTest._default.Backend.AdminMenuNavigationWithSecretKeysTestCest.AdminMenuNavigationWithSecretKeysTest.fail.html` -- `Magento.AcceptanceTest._default.Backend.AdminMenuNavigationWithSecretKeysTestCest.AdminMenuNavigationWithSecretKeysTest.fail.png` +- `Magento.AcceptanceTest._default.Backend.AdminMenuNavigationWithSecretKeysTestCest.AdminMenuNavigationWithSecretKeysTest.fail.html` +- `Magento.AcceptanceTest._default.Backend.AdminMenuNavigationWithSecretKeysTestCest.AdminMenuNavigationWithSecretKeysTest.fail.png` The file name encodes: -- `Magento` namespace -- with the `AcceptanceTest` test type -- generated as a part of the `_default` suite -- defined at the `Magento_Backend` module -- implemented in the `AdminMenuNavigationWithSecretKeysTestCest` PHP class -- with the `AdminMenuNavigationWithSecretKeysTest` test name -- and execution status `fail` +- `Magento` namespace +- with the `AcceptanceTest` test type +- generated as a part of the `_default` suite +- defined at the `Magento_Backend` module +- implemented in the `AdminMenuNavigationWithSecretKeysTestCest` PHP class +- with the `AdminMenuNavigationWithSecretKeysTest` test name +- and execution status `fail` Actions after `FAIL` are run as a part of the [`after`][] hook of the test. @@ -235,14 +235,14 @@ full_page 18. $I->saveScreenshot() at tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:63 ``` -- `1) AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test` - the failed Codeception test is *AdminMenuNavigationWithSecretKeysTestCest*. It references to the PHP class that implemented the failed test. +- `1) AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test` - the failed Codeception test is *AdminMenuNavigationWithSecretKeysTestCest*. It references to the PHP class that implemented the failed test. -- `Test tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest` - the test is implemented in the *AdminMenuNavigationWithSecretKeysTest* test method of the *tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php* file under `/dev/tests/acceptance/`. +- `Test tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest` - the test is implemented in the *AdminMenuNavigationWithSecretKeysTest* test method of the *tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php* file under `/dev/tests/acceptance/`. It matches the corresponding test defined in XML that is *AdminMenuNavigationWithSecretKeysTest* defined in `...` -- `Step See "#something"` - the failing test step is the *see* action with the *#something* selector. It would correspond the `` test step in the XML defined tests. +- `Step See "#something"` - the failing test step is the *see* action with the *#something* selector. It would correspond the `` test step in the XML defined tests. -- `Fail Failed asserting that on page /admin/admin/system_config/index/key/678b7ba922c.../` - the fail occurred on the web page `/admin/admin/system_config/index/key/678b7ba922c.../`. +- `Fail Failed asserting that on page /admin/admin/system_config/index/key/678b7ba922c.../` - the fail occurred on the web page `/admin/admin/system_config/index/key/678b7ba922c.../`. ```terminal --> ... diff --git a/docs/section.md b/docs/section.md index 745d2e574..98221ec44 100644 --- a/docs/section.md +++ b/docs/section.md @@ -6,18 +6,18 @@ A `
` can define: -- An explicit element that has a selector equal to the constant string. Example: `selector="#add_root_category_button"` -- A parameterized element that contains substitutable values in the selector. Example: `selector="#element .{{var1}} .{{var2}}"`. +- An explicit element that has a selector equal to the constant string. Example: `selector="#add_root_category_button"` +- A parameterized element that contains substitutable values in the selector. Example: `selector="#element .{{var1}} .{{var2}}"`. ## Substitutable values Substitutable values in the test can be of the following formats: -- String literals (`stringLiteral`) -- References to a [data entity][] (XML data from the corresponding `.../Data/*.xml`) such as `entityName.Field`. -- Persisted data: - - `$persistedCreateDataKey.field$` for data created in the scope of a [test][] using the [``][] action with `stepKey="persistedCreateDataKey"`. - - `$$persistedCreateDataKey.field$$` for data created in [before][] and [after][] hooks. Even though ``and `` are nested inside a [test][], persisted data is stored differently when it is done in a test hook. Therefore it must be accessed with a different notation. +- String literals (`stringLiteral`) +- References to a [data entity][] (XML data from the corresponding `.../Data/*.xml`) such as `entityName.Field`. +- Persisted data: + - `$persistedCreateDataKey.field$` for data created in the scope of a [test][] using the [``][] action with `stepKey="persistedCreateDataKey"`. + - `$$persistedCreateDataKey.field$$` for data created in [before][] and [after][] hooks. Even though ``and `` are nested inside a [test][], persisted data is stored differently when it is done in a test hook. Therefore it must be accessed with a different notation. The following diagram shows the XML structure of an MFTF section: @@ -44,10 +44,10 @@ The format of a `
` is: The following conventions apply to MFTF sections: -- `
` name must be alphanumeric. -- `*Section.xml` is stored in the _Section_ directory of a module. -- The name format is `{Admin|Storefront}{SectionDescription}Section.xml`. -- Camel case is used for `
` elements. +- `
` name must be alphanumeric. +- `*Section.xml` is stored in the _Section_ directory of a module. +- The name format is `{Admin|Storefront}{SectionDescription}Section.xml`. +- Camel case is used for `
` elements. They describe the function of the element rather than attempting to describe the selector used. ## Example @@ -70,8 +70,8 @@ This example uses a `AdminCategorySidebarActionSection` section. All sections wi The `AdminCategorySidebarActionSection` section declares two buttons: -- `addRootCategoryButton` - button with a `#add_root_category_button` locator on the parent web page -- `addSubcategoryButton` - button with a `#add_subcategory_button` locator on the parent web page +- `addRootCategoryButton` - button with a `#add_root_category_button` locator on the parent web page +- `addSubcategoryButton` - button with a `#add_subcategory_button` locator on the parent web page The following is an example of a call in test: @@ -149,4 +149,4 @@ Whenever the `signIn` button is used in a test, the MFTF will add a 30 second `w [XPath]: https://www.w3schools.com/xml/xpath_nodes.asp [CSS]: https://www.w3schools.com/cssref/css_selectors.asp [Locator function]: ./section/locator-functions.html -[Parameterized selectors]: ./section/parameterized-selectors.html \ No newline at end of file +[Parameterized selectors]: ./section/parameterized-selectors.html diff --git a/docs/selectors.md b/docs/selectors.md index 870072e15..376140819 100644 --- a/docs/selectors.md +++ b/docs/selectors.md @@ -14,21 +14,21 @@ Instead you SHOULD use `contains(@attribute, "foo")` where `@attribute` is any v ### CSS and XPath selectors SHOULD be implemented in their most simple form -* GOOD: `#foo` -* BAD: `button[contains(@id, "foo")]` +* GOOD: `#foo` +* BAD: `button[contains(@id, "foo")]` ### CSS and XPath selectors SHOULD avoid making use of hardcoded indices Instead you SHOULD parameterize the selector. -* GOOD: `.foo:nth-of-type({{index}})` -* BAD: `.foo:nth-of-type(1)` +* GOOD: `.foo:nth-of-type({{index}})` +* BAD: `.foo:nth-of-type(1)` -* GOOD: `button[contains(@id, "foo")][{{index}}]` -* BAD: `button[contains(@id, "foo")][1]` +* GOOD: `button[contains(@id, "foo")][{{index}}]` +* BAD: `button[contains(@id, "foo")][1]` -* GOOD: `#actions__{{index}}__aggregator` -* BAD: `#actions__1__aggregator` +* GOOD: `#actions__{{index}}__aggregator` +* BAD: `#actions__1__aggregator` ### CSS and XPath selectors MUST NOT reference the `@data-bind` attribute diff --git a/docs/suite.md b/docs/suite.md index b24b3b9bf..e49dd02ab 100644 --- a/docs/suite.md +++ b/docs/suite.md @@ -41,25 +41,25 @@ The format of a suite: ## Principles -- A suite name: - - must not match any existing group value. +- A suite name: + - must not match any existing group value. For example, the suite `` will fail during test run if any test contains in annotations ``. - - must not be `default` or `skip`. Tests that are not in any suite are generated under the `default` suite. + - must not be `default` or `skip`. Tests that are not in any suite are generated under the `default` suite. The suite name `skip` is synonymous to including a test in the ``, which will be deprecated in MFTF 3.0.0. - - can contain letters, numbers, and underscores. - - should be upper camel case. -- A suite must contain at least one ``, or one ``, or both. -- Using `` in a suite, you must add the corresponding `` to restore the initial state of your testing instance. + - can contain letters, numbers, and underscores. + - should be upper camel case. +- A suite must contain at least one ``, or one ``, or both. +- Using `` in a suite, you must add the corresponding `` to restore the initial state of your testing instance. ## Conditions Using suites enables test writers to consolidate conditions that are shared between tests. The code lives in one place and executes once per suite. -- Set up preconditions and postconditions using [actions] in [``] and [``] correspondingly, just similar to use in a [test]. -- Clean up after suites just like after tests. +- Set up preconditions and postconditions using [actions] in [``] and [``] correspondingly, just similar to use in a [test]. +- Clean up after suites just like after tests. The MFTF enforces the presence of both `` and `` if either is present. -- Do not reference in the subsequent tests to data that was persisted in the preconditions. +- Do not reference in the subsequent tests to data that was persisted in the preconditions. Referencing to `$stepKey.field$` of these actions is not valid. ## Test writing @@ -70,25 +70,25 @@ You cannot isolate this test from preconditions of the suite; it cannot be used There are several ways to generate and execute your new test in the context of a suite: -- Edit the appropriate `suite.xml` to include your test only and run: +- Edit the appropriate `suite.xml` to include your test only and run: ```bash vendor/bin/mftf run:group ``` -- Temporarily add a group to your test like `` and run: +- Temporarily add a group to your test like `` and run: ```bash vendor/bin/mftf run:group foo ``` -- To limit generation to your suite/test combination, run in conjunction with the above: +- To limit generation to your suite/test combination, run in conjunction with the above: ```bash vendor/bin/mftf generate:suite ``` -- To generate any combination of suites and tests, use [`generate:tests`] with the `--tests` flag. +- To generate any combination of suites and tests, use [`generate:tests`] with the `--tests` flag. ## Examples @@ -189,9 +189,9 @@ The suite includes a specific test `SomeCacheRelatedTest` and every `` tha This example declares a suite with the name `PaypalConfiguration`: -- `` block persists a Paypal Configuration enabling all tests in this suite to run under the newly reconfigured Magento instance. -- `` block deletes the persisted configuration, returning Magento to its initial state. -- The suite includes all tests from the `Catalog` module, except the `PaypalIncompatibleTest` test. +- `` block persists a Paypal Configuration enabling all tests in this suite to run under the newly reconfigured Magento instance. +- `` block deletes the persisted configuration, returning Magento to its initial state. +- The suite includes all tests from the `Catalog` module, except the `PaypalIncompatibleTest` test. ## Elements reference @@ -233,9 +233,9 @@ A set of filters that you can use to specify which tests to include in the test It may contain filters by: -- test which names a specific ``. -- group which refers to a declared `group` annotation. -- module which refers to `test` files under a specific Magento Module. +- test which names a specific ``. +- group which refers to a declared `group` annotation. +- module which refers to `test` files under a specific Magento Module. The element can contain [``], [``], and [``]. @@ -253,9 +253,9 @@ There two types of behavior: It may contain filters by: -- test which names a specific ``. -- group which refers to a declared `group` annotation. -- module which refers to `test` files under a specific Magento Module. +- test which names a specific ``. +- group which refers to a declared `group` annotation. +- module which refers to `test` files under a specific Magento Module. The element may contain [``], [``], and [``]. @@ -291,4 +291,4 @@ Attributes|Type|Use|Description [``]: #test-tag [``]: #group-tag [``]: #module-tag -[``]: #include-tag \ No newline at end of file +[``]: #include-tag diff --git a/docs/test.md b/docs/test.md index 462cd138e..dcda0ba50 100644 --- a/docs/test.md +++ b/docs/test.md @@ -44,11 +44,11 @@ The format of `` is: The following conventions apply to MFTF tests: -* All names within the framework are in the CamelCase format. -* `` name must be alphanumeric. -* Each action and action group has its own identifier `` for reference purposes. -* A test may have any number of [assertions][assertion] at any point within the ``. -* If `` is included in ``, it **cannot be generated in isolation** to the rest of the contents of the suite (see [suites] for details). +* All names within the framework are in the CamelCase format. +* `` name must be alphanumeric. +* Each action and action group has its own identifier `` for reference purposes. +* A test may have any number of [assertions][assertion] at any point within the ``. +* If `` is included in ``, it **cannot be generated in isolation** to the rest of the contents of the suite (see [suites] for details). Multiple `` tags per XML file can make it hard to find and organize tags. To simplify, we generate one `test.php` file per `` tag provided, though we support both single and multiple `` tags per XML file. @@ -90,8 +90,8 @@ Allure annotations provide metadata for reporting. `` may contain these child elements: -* Any [``][actions] -* [``] +* Any [``][actions] +* [``] ### after {#after-tag} @@ -100,8 +100,8 @@ The steps are run in both successful **and** failed test runs. `` may contain: -* Any [``][actions] -* [``] +* Any [``][actions] +* [``] ### actionGroup {#actiongroup-tag} @@ -144,4 +144,4 @@ See [Action groups][action group] for more information. [Codeception]: https://codeception.com/docs/07-AdvancedUsage [extend]: extending.md [merging]: ./merging.md#insert-after -[suites]: ./suite.md \ No newline at end of file +[suites]: ./suite.md diff --git a/docs/update.md b/docs/update.md index 1a3137eb6..129742346 100644 --- a/docs/update.md +++ b/docs/update.md @@ -77,4 +77,3 @@ To update the MFTF from the previous minor version: [Changelog]: https://github.com/magento/magento2-functional-testing-framework/blob/master/CHANGELOG.md [WYSIWYG settings]: getting-started.md#wysiwyg-settings [Security settings]: getting-started.md#security-settings -[Find your version]: introduction.md#find-your-mftf-version \ No newline at end of file diff --git a/docs/versioning.md b/docs/versioning.md index d018058d5..ecb8438c5 100644 --- a/docs/versioning.md +++ b/docs/versioning.md @@ -34,11 +34,11 @@ X.Y.Z For example: -- Magento 2 ships with MFTF version 2.3.9 -- A patch is added to fix a bug: 2.3.10 (Increment Z = backward compatible change) -- New action command added: 2.4.0 (Increment Y, set Z to 0 = backward compatible change) -- New action added: 2.4.1 (Increment Z = backward compatible change) -- Major new features added to MFTF to support changes in Magento codebase: 3.0.0. (Increment X, reset Y and Z to 0 = backward incompatible change) +- Magento 2 ships with MFTF version 2.3.9 +- A patch is added to fix a bug: 2.3.10 (Increment Z = backward compatible change) +- New action command added: 2.4.0 (Increment Y, set Z to 0 = backward compatible change) +- New action added: 2.4.1 (Increment Z = backward compatible change) +- Major new features added to MFTF to support changes in Magento codebase: 3.0.0. (Increment X, reset Y and Z to 0 = backward incompatible change) ### Z release - patch