Approach - Audio & Video Quality Testing by TestDevLab

E2E audio & video quality testing

When it comes to audio & video quality testing, we cover everything from test data capture, to data processing, data analysis, and reporting. Take it all or one specific part - it's your call!

  1. Test data capture

    1. Define scope

      • Platform selection

        • Mobile (iOS, Android)

        • Desktop (Win, Mac, Linux)

      • Device selection & placement

        • Physical - Custom set of pre-defined devices based on need

        • Virtual - 10s of thousands of virtual machines

        • Hybrid - Mix of physical and virtual users

      • Content selection

        • Video media - static/dynamic background/foreground, resolution, bitrate etc.

        • Audio media - different audio profiles, frequency profile, bitrate etc.

        • Screenshare - slides, video, scrollable content

        • Virtual background - slides, video, scrollable content

        • Complex scenarios - variable media types in one test

      • A/V app domain selection

        • Video conferencing apps

        • Video on Demand apps

        • Short Form Video apps

        • A/V SDK’s

        • Live Streaming apps

        • Audio Streaming apps

        • VoIP (WebRTC, SIP) apps

      • VoIP (WebRTC, SIP) apps

        • Current, Previous, Future app releases (n, n-1, n+1)

        • Competitor apps

      • Test data management

        • Test account sourcing

        • Test account configuration

    2. Initiate Data Capture

      • Launching network limitations

        • Video media - static/dynamic background/foreground, resolution, bitrate etc.

        • Audio media - different audio profiles, frequency profile, bitrate etc.

        • Screenshare - slides, video, scrollable content

        • Virtual background - slides, video, scrollable content

        • Complex scenarios - variable media types in one test

      • Launch test data recording

        • Network traces - Sent, Received data

        • Audio recordings - Original loopback, Receiver

        • Video recordings - Screen record/HDMI Out

        • Hardware performance - CPU, GPU, RAM

        • WebRTC dump (Chrome only)

      • Test session initiation

        • Connect participants in session using automated test framework

        • Manual session initialization, adding participants and any other UI manipulations by hand

  2. Data Processing

      • Video Quality metrics

        • Full Reference image quality assessment

          • Precisely tuned VMAF + PSNR & SSIM

        • No Reference image quality assessment

          • VQTDL - deep learning MOS prediction

          • BRISQUE - NSS based on luminance coefficients

        • Subjective MOS group of people mean opinion score 1-5

        • Frame rate avg fps captured on receiver media

        • Freezes duration, count, time between freezes, total, avg

        • Stalls duration, count, time between stalls, total, avg

        • Blurriness score of blurriness intensity

      • Audio Quality Metrics

        • Audio Quality

          • POLQA voice quality MOS prediction

          • VISQOL voice quality MOS prediction

        • Audio Delay

          • Audio Delay

          • End to End latency measured in ms

      • Video conference specific metrics

        • Virtual background precision score determine overcropping and undercropping of virtual background feature

        • Video delay end-to-end delay from sender to receiver

      • Other

        • Network Consumption

        • Network Latency

        • Network Quality

        • Audio Video Sync

        • CPU, GPU, RAM

      • Short Form Video apps specific metrics

        • Audio

          • Loudness metrics

          • Frequency metrics

          • Overall metadata about audio samples

          • Stereo sound analysis (stereo preservation)

          • Audio / Video desync during upload process

        • Video Codec

          • app content encodings possible for download (yt-dlp)

          • real time playback codec type retrieving (xcode, android studio)

        • Latency

          • First reaction

          • Page readiness

          • Element readiness

          • Upload

          • Sharing

          • Commenting

          • Messaging

          • Deleting etc.

      • WebRTC specific metrics

        • Connection count

        • Codec

        • Bitrate

        • Packets Lost

        • Jitter

        • Frame Height & Width

        • FPS

        • Jitter BUFFER

        • Round Trip Time

  3. Data Analysis & Reporting

    1. Data analysis

      • Metric correlation analysis

        • Network bandwidth vs delivered quality & frame/sample rate

        • Audio and video synchronization data inspection & reproduction

        • Hardware performance vs network bandwidth and frame/sample rate

        • Data outlier detection

        • QoE metrics correlation to QoS metrics

      • Statistical analysis

        • Time series data analysis - trends and cyclic patterns

        • Five number summary (min, max, median, Q1, Q3)

        • Statistical mean, standard deviation, variance coefficients

        • Standard error and confidence intervals

        • Percentile values

      • Graphs and visual representation of data

        • Time-series graphs per metric

        • Average comparison bar charts per metric

        • Difference (positive/negative) bar charts

        • Competitive benchmarking heatmaps

        • Image data histograms

        • Audio spectrograms

    2. Data reporting

      • Data reporting types

        • In-depth slide deck reports

          • including comprehensive findings and conclusions

        • Results Portal

          • for immediate visual overview of test results and trends

        • API

          • for raw time-series data with filtering and data structuring options

        • Excel sheets

          • with custom tables & graphs

        • Raw files

          • time-series data reports as files