Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean up the documentation for the library testing #59401

Merged
merged 15 commits into from
Nov 8, 2021
Merged
40 changes: 34 additions & 6 deletions docs/workflow/testing/libraries/testing-android.md
Original file line number Diff line number Diff line change
@@ -1,18 +1,23 @@
# Testing Libraries on Android

## Prerequisites

The following dependencies should be installed in order to be able to run tests:

- OpenJDK
- Android NDK
- Android SDK
- OpenJDK
- OpenSSL

To manage the dependencies, you can install them via terminal or using Android Studio.

### Using a terminal

OpenJDK can be installed on Linux (Ubuntu) using `apt-get`:
```bash
sudo apt-get install openjdk-8 zip unzip
```

Android SDK, NDK and OpenSSL can be automatically installed via the following script:
Android SDK and NDK can be automatically installed via the following script:
```bash
#!/usr/bin/env bash
set -e
Expand All @@ -21,7 +26,6 @@ NDK_VER=r21b
SDK_VER=6200805_latest
SDK_API_LEVEL=29
SDK_BUILD_TOOLS=29.0.3
OPENSSL_VER=1.1.1g-alpha-1

if [[ "$OSTYPE" == "darwin"* ]]; then
HOST_OS=darwin
Expand All @@ -47,8 +51,21 @@ yes | ${ANDROID_SDK_ROOT}/cmdline-tools/tools/bin/sdkmanager --sdk_root=${ANDROI
${ANDROID_SDK_ROOT}/cmdline-tools/tools/bin/sdkmanager --sdk_root=${ANDROID_SDK_ROOT} "platform-tools" "platforms;android-${SDK_API_LEVEL}" "build-tools;${SDK_BUILD_TOOLS}"
```

### Using Android Studio

Android Studio offers a convenient UI:
- to install all the dependencies;
- to manage android virtual devices;
- to make easy use of adb logs.

## Building Libs and Tests for Android

Before running a build you might want to set the Android SDK and NDK environment variables:
```
export ANDROID_SDK_ROOT=<PATH-TO-ANDROID-SDK>
export ANDROID_NDK_ROOT=<PATH-TO-ANDROID-NDK>
```

Now we're ready to build everything for Android:
```
./build.sh mono+libs -os Android -arch x64
Expand All @@ -66,16 +83,27 @@ The following shows how to run tests for a specific library
./dotnet.sh build /t:Test src/libraries/System.Numerics.Vectors/tests /p:TargetOS=Android /p:TargetArchitecture=x64
```

### Running the functional tests

There are [functional tests](https://github.com/dotnet/runtime/tree/main/src/tests/FunctionalTests/) which aim to test some specific features/configurations/modes on a target mobile platform.

A functional test can be run the same way as any library test suite, e.g.:
```
./dotnet.sh build /t:Test -c Release /p:TargetOS=Android /p:TargetArchitecture=x64 src/tests/FunctionalTests/Android/Device_Emulator/PInvoke/Android.Device_Emulator.PInvoke.Test.csproj
```

Currently functional tests are expected to return `42` as a success code so please be careful when adding a new one.

### Test App Design
Android app is basically a [Java Instrumentation](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AndroidAppBuilder/Templates/MonoRunner.java) and a simple Activity that inits the Mono Runtime via JNI. This Mono Runtime starts a simple xunit test
Android app is basically a [Java Instrumentation](https://github.com/dotnet/runtime/blob/main/src/tasks/AndroidAppBuilder/Templates/MonoRunner.java) and a simple Activity that inits the Mono Runtime via JNI. This Mono Runtime starts a simple xunit test
runner called XHarness.TestRunner (see https://github.com/dotnet/xharness) which runs tests for all `*.Tests.dll` libs in the bundle. There is also XHarness.CLI tool with ADB embedded to deploy `*.apk` to a target (device or emulator) and obtain logs once tests are completed.

### Obtaining the logs
XHarness for Android doesn't talk much and only saves test results to a file. However, you can also subscribe to live logs via the following command:
```
adb logcat -s "DOTNET"
```
Or simply open `logcat` window in Android Studio or Visual Stuido.
Or simply open `logcat` window in Android Studio or Visual Studio.

### AVD Manager
If Android Studio is installed, [AVD Manager](https://developer.android.com/studio/run/managing-avds) can be used from the IDE to create and start Android virtual devices. Otherwise, the Android SDK provides the [`avdmanager` command line tool](https://developer.android.com/studio/command-line/avdmanager).
Expand Down
71 changes: 63 additions & 8 deletions docs/workflow/testing/libraries/testing-apple.md
Original file line number Diff line number Diff line change
@@ -1,34 +1,89 @@
# Testing Libraries on iOS and tvOS
# Testing Libraries on iOS, tvOS, and MacCatalyst

In order to build libraries and tests for iOS or tvOS you need recent version of XCode installed (e.g. 11.3 or higher).
## Prerequisites

Build Libraries for iOS Simulator:
- XCode 11.3 or higher

## Building Libs and Tests

You can build and run the library tests:
- on a simulator;
- on a device.

Run the following command in a terminal:
```
./build.sh mono+libs -os <TARGET_OS> -arch x64
```
where `<TARGET_OS>` is one of the following:

| simulator | device|
|:--------------|:------|
| iOSSimulator | iOS |
| tvOSSimulator | tvOS |
| MacCatalyst | |


e.g., to build for a iOS simulator, run:
```
./build.sh mono+libs -os iOSSimulator -arch x64
```

Run tests one by one for each test suite on a simulator:
```
./build.sh libs.tests -os iOSSimulator -arch x64 -test
```

### Building for a device

In order to run the tests on a device:
- Set the os to `iOS` instead of `iOSSimulator`
- Specify `DevTeamProvisioning` (see [developer.apple.com/account/#/membership](https://developer.apple.com/account/#/membership), scroll down to `Team ID`):
- Set the `-os` parameter to a device-related value (see above)
- Specify `DevTeamProvisioning` (see [developer.apple.com/account/#/membership](https://developer.apple.com/account/#/membership), scroll down to `Team ID`).

For example:
```
./build.sh libs.tests -os iOS -arch x64 -test /p:DevTeamProvisioning=H1A2B3C4D5
```
[AppleAppBuilder](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AppleAppBuilder/AppleAppBuilder.cs) generates temp Xcode projects you can manually open and resolve provisioning issues there using native UI and deploy to your devices.
[AppleAppBuilder](https://github.com/dotnet/runtime/blob/main/src/tasks/AppleAppBuilder/AppleAppBuilder.cs) generates temp Xcode projects you can manually open and resolve provisioning issues there using native UI and deploy to your devices.

### Running individual test suites
- The following shows how to run tests for a specific library:
```
./dotnet.sh build src/libraries/System.Numerics.Vectors/tests /t:Test /p:TargetOS=iOS /p:TargetArchitecture=x64
```

### Running the functional tests

There are [functional tests](https://github.com/dotnet/runtime/tree/main/src/tests/FunctionalTests/) which aim to test some specific features/configurations/modes on a target mobile platform.

A functional test can be run the same way as any library test suite, e.g.:
```
./dotnet.sh build /t:Test -c Release /p:TargetOS=iOSSimulator /p:TargetArchitecture=x64 src/tests/FunctionalTests/iOS/Simulator/PInvoke/iOS.Simulator.PInvoke.Test.csproj
```

Currently functional tests are expected to return `42` as a success code so please be careful when adding a new one.

### Testing various configurations

It's possible to test various configurations by setting a combination of additional MSBuild properties such as `RunAOTCompilation`,`MonoEnableInterpreter`, and some more.

1. Interpreter Only

This configuration is necessary for hot reload scenarios.

To enable the interpreter, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=true` to a build command.

2. AOT only

To build for AOT only mode, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=false` to a build command.

3. AOT-LLVM

To build for AOT-LLVM mode, add `/p:RunAOTCompilation=true /p:MonoEnableInterpreter=false /p:MonoEnableLLVM=true` to a build command.

### Test App Design
iOS/tvOS `*.app` (or `*.ipa`) is basically a simple [ObjC app](https://github.com/dotnet/runtime/blob/main/src/mono/msbuild/AppleAppBuilder/Templates/main-console.m) that inits the Mono Runtime. This Mono Runtime starts a simple xunit test
iOS/tvOS `*.app` (or `*.ipa`) is basically a simple [ObjC app](https://github.com/dotnet/runtime/blob/main/src/tasks/AppleAppBuilder/Templates/main-console.m) that inits the Mono Runtime. This Mono Runtime starts a simple xunit test
runner called XHarness.TestRunner (see https://github.com/dotnet/xharness) which runs tests for all `*.Tests.dll` libs in the bundle. There is also XHarness.CLI tool to deploy `*.app` and `*.ipa` to a target (device or simulator) and listens for logs via network sockets.

### Existing Limitations
- Most of the test suites crash on devices due to #35674
- Simulator uses JIT mode only at the moment (to be extended with FullAOT and Interpreter)
- Interpreter is not enabled yet.
15 changes: 14 additions & 1 deletion docs/workflow/testing/mono/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,20 @@ For example, the following command is for running System.Runtime tests:
make run-tests-corefx-System.Runtime
```
### Mobile targets and WebAssembly
Build and run library tests against Webassembly, Android or iOS. See instructions located in [Library testing document folder](../libraries/)
Build and run library tests against WebAssembly, Android or iOS. See instructions located in [Library testing document folder](../libraries/)

## Running the functional tests

There are the [functional tests](https://github.com/dotnet/runtime/tree/main/src/tests/FunctionalTests/) which aim to test some specific features/configurations/modes on Android, iOS-like platforms (iOS/tvOS + simulators, MacCatalyst), and WebAssembly.

A functional test can be run the same way as any library test suite, e.g.:
```
./dotnet.sh build /t:Test -c Release /p:TargetOS=Android /p:TargetArchitecture=x64 src/tests/FunctionalTests/Android/Device_Emulator/PInvoke/Android.Device_Emulator.PInvoke.Test.csproj
```

Currently the functional tests are expected to return `42` as a success code so please be careful when adding a new one.

For more details, see instructions located in [Library testing document folder](../libraries/).

# Running the Mono samples
There are a few convenient samples located in `$(REPO_ROOT)/src/mono/sample`, which could help you test your program easily with different flavors of Mono or do a sanity check on the build. The samples are set up to work with a specific configuration; please refer to the relevant Makefile for specifics. If you would like to work with a different configuration, you can edit the Makefile.
Expand Down