diff --git a/source/fusiondirectory/development/automatedtesting/index.rst b/source/fusiondirectory/development/automatedtesting/index.rst new file mode 100644 index 0000000000000000000000000000000000000000..f4d1faabaac74ec59bc1f0a717574d2bc9350cbb --- /dev/null +++ b/source/fusiondirectory/development/automatedtesting/index.rst @@ -0,0 +1,118 @@ +Testing +======== + +The base project for testing is called `automated-testing`. Here all source code for the tests and test framework is placed. + +Prerequisites +------------- + +In order for the existing tests to run properly, some prerequisites are needed: + +* **Selenium server**: We are using Selenium as a running server for our tests. The latest version can be downloaded from the official page: `<https://www.selenium.dev/downloads/>`_. +* **Java**: Tests are written exclusively using Java 17. The version can be installed using `<https://computingforgeeks.com/install-oracle-java-openjdk-on-debian-linux/>`_ as a reference for Debian-based systems. +* **Geckodriver**: Tests are written for the Firefox browser, for which the Geckodriver is needed. We are using version 0.35, which can be downloaded from `<https://github.com/mozilla/geckodriver/releases>`_. +* **VNU Jar**: Some tests need the VNU jar to be available on the machine. This can be downloaded using `<https://www.npmjs.com/package/vnu-jar>`_ as a reference. +* **DSA server**: Some tests need a DSA server to be available on the machine. This is done by modifying the ``orchestrator.conf`` file from your configuration path to have ``tester`` as both username and password. + +.. _setup: + +Setup +----- + +In order to run the tests properly, follow these steps: + +1. Clone the `automated-testing` project. +2. Run ``cd ./automated-testing/qa``. +3. Edit ``qa/src/resources/testsConfig.ini`` file to match your data: + + * Modify the LDAP-related values. + * Find the VNU jar path on your device and paste it under the correct variable name. + * Make sure that the folders for downloads and screenshots exist. + * Do not modify the name of the variables, as the tests will not work otherwise. + +4. Save the file modifications. +5. Run ``./gradlew clean`` (if using a Linux-based system) or ``gradle clean`` (if using Windows). +6. Run the tests: + + * To run all the tests, execute ``./gradlew test`` (this will take some time as the code base is large, but you will see the test progress). + * To run specific tests, execute ``./gradlew test --tests "..."``, where ``...`` is the path to your test starting from ``qa/src/test/java``. This can be: + * A full subfolder (e.g., ``org.fd.tests.core.*`` to run all tests in the core folder). + * A specific class of tests (e.g., ``org.fd.tests.core.LoginTest`` to run all tests from the ``LoginTest`` class). + * A specific test (e.g., ``org.fd.tests.core.LoginTest.testGoodLogin`` to only run ``testGoodLogin`` from the ``LoginTest`` class). + +7. After tests complete, you will see: + + * ``BUILD SUCCESSFUL`` (in green) if all tests passed. + * ``BUILD FAILED`` (in red) if any test failed. + * In both cases, the test results will display: + + * ``PASSED`` for successful tests. + * ``FAILED`` for failed tests, along with a short message (for troubleshooting, see :ref:`troubleshooting`). + +.. _troubleshooting: + +Troubleshooting +--------------- + +For every failed test, we have implemented features in our framework to aid troubleshooting: + +* If the test is running in a visual interface, we take a screenshot of the last visual screen of the driver (saved in the ``screenshots`` folder defined in ``qa/src/resources/testsConfig.ini``). +* Logs are saved in a file in the same ``screenshots`` folder. These logs help track the workflow of the test and identify where and why it crashed. + +Project Structure +----------------- + +All important modifiable parts of the code base are inside the ``qa`` folder: + +* ``config`` folder: Defines lint rules in XML format, checked by the pipeline whenever code is pushed to GitLab. +* ``build.gradle`` file: Defines the needed plugins for this project. +* ``src/test`` folder: Contains all the test code: + + * ``resources`` folder: Stores additional files: + + * ``testsConfig.ini``: The initial configuration for the tests, should be modified as stated in :ref:`setup`. + * ``ldifs`` folder: Contains LDIF files inserted into the LDAP server before the tests run. + * General rule: All additional non-Java files required for tests should be placed here. + + * ``test/org/fd`` folder: Contains all test classes: + + * ``Utils.java``: A class with static fields, mainly Java translations of ``testsConfig.ini`` variables. + * ``LdapConnection.java``: Defines LDAP-related actions like emptying the LDAP (``emptyLdap()``) and inserting an LDIF (``insertLdif(String filename)``). + * ``Assertions.java``: Contains methods for assertions, such as checking if a user is logged in (``assertLoggedIn(String username)``). + * ``ScreenshotTestWatcher.java``: Defines actions executed after a test completes, including distinguishing between failed, successful, and aborted tests. See :ref:`troubleshooting`. + * ``FusionDirectoryTestCase.java``: The main test class template in FusionDirectory. It integrates ``LdapConnection``, ``Assertions``, and ``ScreenshotTestWatcher`` with Selenium to interact with FusionDirectory’s web interface. + + * ``tests`` folder: Contains explicit tests: + + * ``core`` folder: Tests verifying the core functionality of FusionDirectory. + * ``plugins`` folder: Contains one subfolder for each FusionDirectory plugin that has tests. + * ``tools`` folder: Contains tests for various tools (schema manager, plugin manager, configuration manager, migration manager, and orchestrator client): + + * Unlike other folders, these tests often require Unix console access rather than web interface interaction. + * ``CommandLineTestCase.java``: Defines methods for interacting with the command line: + + * ``executeCommandWithWait(String command)``: Used when no user input is required. + * ``executeCommandWithoutWait(String command)``: Used when user input is required, handled through the returned ``Response`` object. + * ``CommandLineTestWatcher.java``: A smaller version of ``ScreenshotTestWatcher.java`` that only copies log files, as screenshots are not applicable. + + * ``installation`` folder: Contains a single test verifying the installation page functionality. + +How to Write a Test +------------------- + +Before writing a test, determine where it should be placed by asking, *What am I testing?* If the answer is a plugin ``abc``, place the test in the ``plugins/abc`` folder. Then: + +1. Create a class in the appropriate folder, ensuring the name ends in ``Test``. +2. In the class constructor, define the ``initLdifs`` array with required LDIF files, or leave it empty if none are needed. +3. If altering FusionDirectory-related concepts, revert changes before the test ends or use an ``@AfterEach`` annotated method to clean up automatically. +4. When fetching web interface elements, check ``FusionDirectoryTestCase.java`` for existing methods before writing new ones to maintain clean tests and ensure easy maintenance. +5. Annotate test methods with ``@Test`` (or similar annotations like ``@RepeatedTest(n)`` or ``@ParameterizedTest(methodName)``) for Gradle compatibility. + +Before pushing to GitLab, perform these checks: + +* Run ``./gradlew checkstyleTest`` and ``./gradlew spotbugsTest`` and fix any issues. +* Run the tests in your virtual machine and ensure they pass. +* Run the tests in a Bullseye Docker image and an Ubuntu Docker image to verify cross-platform compatibility. + +By following these steps, we minimize failed pipelines and expedite code merging, ensuring tests facilitate development rather than hinder it. Reliable tests are key to this process. + diff --git a/source/fusiondirectory/development/index.rst b/source/fusiondirectory/development/index.rst index 584616ada701dc82e1a7f2ebaf0312bd8b307fe0..3492fec56c5fd8d9df3189343cc68630dd1b6070 100644 --- a/source/fusiondirectory/development/index.rst +++ b/source/fusiondirectory/development/index.rst @@ -9,4 +9,5 @@ Development codingstyle/index.rst writeplugin/index.rst api/index.rst + automatedtesting/index.rst documentation/index.rst