/*===================================================================================================
DTT_Guide.doxygen

Copyright 2015, 2017, 2023-2024 Avid Technology, Inc. 
All rights reserved.

This file is part of the Avid AAX SDK.

The AAX SDK is subject to commercial or open-source licensing.

By using the AAX SDK, you agree to the terms of both the Avid AAX SDK License
Agreement and Avid Privacy Policy.

AAX SDK License: https://developer.avid.com/aax
Privacy Policy: https://www.avid.com/legal/privacy-policy-statement

Or: You may also use this code under the terms of the GPL v3 (see
www.gnu.org/licenses).

THE AAX SDK IS PROVIDED "AS IS" WITHOUT ANY WARRANTY, AND ALL WARRANTIES, WHETHER
EXPRESSED OR IMPLIED, INCLUDING MERCHANTABILITY AND FITNESS FOR PURPOSE, ARE
DISCLAIMED.

===================================================================================================*/

/** @defgroup DTT_Guide DTT Guide
        @ingroup CompatibleEnvironmentsDocs
        
        @brief How to automate different test scenarios for DSH
        
        @section dtt_guide_contents Contents
		\li \ref dtt_guide_00_what_is_dtt
		\li \ref dtt_guide_01_how_to_run_test_and_suites_in_dtt
		\li \ref dtt_guide_03_writing_dtt_scripts
		\li \ref dtt_guide_04_logging_in_dtt_and_debuggin_dtt_scripts
		\li \ref dtt_guide_02_working_with_dtt_test_suites

<DIV class="section">
	@section dtt_guide_00_what_is_dtt What is DTT

DishTestTool (DTT) is a Ruby-based command line tool, which provides the ability to script and thus automate DSH test scenarios. It is included in the DigiShell Tools package in the /DTT directory.

\note Ruby is installed by default on macOS. On Windows, you will need to install Ruby and add it to your PATH variable manually. For information regarding Ruby version compatibility with a specific build of DTT, see the ReadMe.txt file in /DTT/sources.

The DTT folder consists of:
<UL>
   <LI CLASS="Bullet">
      <em>Sources</em>
      <UL>
         <LI CLASS="Bullet">DTT core</LI>
         <LI CLASS="Bullet">
	    <em>scripts</em> folder - place all DTT script files here
		
		By default, this folder includes a few example scripts demonstrating basic DTT operations and plug-in testing steps:

            <UL>
               <LI CLASS="Bullet"><em>DSH_SigCancellation.rb</em> - script for the cancellation test </LI>
               <LI CLASS="Bullet"><em>DSH_TI_CycleCounts</em> - script for performing cycle count test</LI>
	       <LI CLASS="Bullet"><em>SuiteGenerator.rb</em> - generates suites for the DSH_SigCancellation and DSH_TI_CycleCounts tests</LI>
	    </UL>
	 </LI>
         <LI CLASS="Bullet"><em>suites</em> folder - place your DTT suite files here</LI>
      </UL>	
   </LI>
   <LI CLASS="Bullet"><em>run_test.command</em> (on Mac) or <em>run_test.bat</em> (on Windows) - command file for runnung tests</LI>
   <LI CLASS="Bullet"><em>run_irb.command</em> (on Mac) or <em>run_irb.bat</em> (on Windows) - command-line interpreter</LI>
</UL>
</DIV>

<DIV class="section">
        @section dtt_guide_01_how_to_run_test_and_suites_in_dtt How to run tests and suites in DTT

<TT>run_test</TT> is the main DTT execution program. <TT>run_test</TT> is able to execute Ruby scripts which have been placed in the scripts folder within the DTT directory.

<UL>
   <LI CLASS="Bullet"><TT>run_test.command -l</TT> - lists all the available scripts and suites</LI>
   <LI CLASS="Bullet"><TT>run_test.command 1</TT> - runs test by number</LI>
   <LI CLASS="Bullet"><TT>run_test.command DSH_SigCancellation</TT> - runs a test by name. Pay attensions that the name of test should be without (!) extension.</LI>
   <LI CLASS="Bullet"><TT>run_test.command --script DSH_SigCancellation -a sample_rate=48000 -a threshold=-80</TT> - runs a test with test script arguments, which are specified using the <TT>-a</TT> option</LI>
</UL>

<IMG width="70%" alt="Running DTT tests" src="DTT_run_tests.png" />
<DIV CLASS="imagecaption">Figure 1: running DTT tests.</DIV>

For more information about script arguments, see \ref describing_and_using_input_arguments_of_the_script

</DIV>

<DIV CLASS="section">
	@section dtt_guide_03_writing_dtt_scripts Writing DTT scripts

Most of the DTT scripts require <TT>DigiShell</TT>, which allows them to run dsh and execute different dsh commands. Each script should be represented in the form of class, which inherits Script class, and also each script must have at least two elements: self.inputs section, where all the input arguments of the test should be described, and run method, which is the main body of your script.

\code
require 'DigiShell'

class ScriptSample < Script
   def self.inputs
      return {}
   end

   def run
      return pass("Well, it didn't explode.  So that's something.");
   end
end
\endcode  
<DIV CLASS="fragmentcaption">Listing 1: Skeleton of the script</DIV> 

		@subsection describing_and_using_input_arguments_of_the_script Describing and using input arguments of the script 

The available parameters and their values for a script are listed in the static <TT>self.inputs</TT> routine. Input arguments must be organized in the form of a hash map which is returned from this routine.

\code
def self.inputs
    return {
      :sample_rate     => [44100,[44100,48000,88200]],
      :path_to_tfx     => ['none'],
      :threshold       => [-96],
    }
end

@dsh.init_dae(sample_rate)
\endcode
<DIV CLASS="fragmentcaption">Listing 2: Desribing input arguments for the script and using them</DIV>

Hash entries should be in the following format:
<DIV CLASS="TextInd1"> :arg_name => [default_value, [range of allowed values]] </DIV>

These arguments can be used then by just calling them by name, like in the example above with <TT>sample_rate</TT> argument.

		@subsection writing_body_of_the_script Writing body of the script

The body of the script must be enclosed in the body of the <TT>run</TT> method of the script class. As far as most DTT tests need DSH, in the example below it's shows how to create a DSH instance in the script and how to use it then. DSH instance can be creted with <TT>DigiDhell.new</TT> method, which requites DigiShell module, as has already been said. Then all the DSH commands become available as methods of the DSH instance, and input arguments can be passed to these command as input arguments for the methods, i.e. in parentheses <TT>dsh.load_dish("DAE")</TT>. Also it's recommended to handle possible exceptions that may occure during the execution of the code, and to make sure that DSH has been closed, if it was instantiated on the moment of the failure.

\code
def run
   begin
      dsh = DigiShell.new(target)
      dsh.load_dish("DAE")
      dsh.init_dae(sample_rate)
      dsh.close

      return pass("Well, it didn't explode.  So that's something.")

   rescue Exception => e
      # make sure to close down dsh before returning
      if (dsh)
         dsh.close
      end
		
      return fail(e)
   end
end
\endcode
<DIV CLASS="fragmentcaption">Listing 3: Example of the body of the script.</DIV> 

</DIV> 


<DIV CLASS="section"> 
	@section dtt_guide_04_logging_in_dtt_and_debuggin_dtt_scripts Logging in DTT and debugging DTT scripts

DTT tool has logging, and all the logs collected in the Logs folder, which is located in root directory of the DTT. DTT creates a separate folder for each test and names these folders with the corresponding names + the time when the particular test has been executed. For example: 

<TT>DSH_SigCancellation_20131225_185146_0001</TT>

Inside each folder there are several log files:
<UL>
   <LI CLASS="Bullet"><em>xxx.html</em> – contains info about input & output of the test in a fancy form (tables)</LI>
   <LI CLASS="Bullet"><em>xxx_c.txt</em> – contains the list of the DSH commands that have been executed</LI>
   <LI CLASS="Bullet"><em>xxx_d.txt</em> – DSH output</LI>
   <LI CLASS="Bullet"><em>xxx_i.txt</em> – info about your system</LI>
   <LI CLASS="Bullet"><em>xxx_l.txt</em> – standard output</LI>
   <LI CLASS="Bullet"><em>xxx_v.txt</em> – verbose output</LI>
</UL>

		@subsection dtt_guide_04_interactive_mode Interactive mode

There is an option to run DTT in interactive mode using interactive ruby shell (irb). When running in this mode, DTT creates a shell which is an extended version of the standard Ruby interpreter. Besides the standard functionality, it knows how to work with DTT classes and can give hints on their methods. In particular, the DTT interactive mode shell knows how to work with DigiShell.

To run DTT in interactive mode, go to the DTT folder and launch the <TT>run_irb</TT> program. At this point you will send ruby commands to dsh through the pipe in YAML format:

\code
t = Target.new # creates an instance of Target. In this case target is a local machine, though we have a possibility to run test on remote machine.
dsh = DigiShell.new(t) # creates an instance of DigiShell(aka launching dsh binary)
dsh.load_dish('DAE') #Loads 'DAE' dish
dsh.help('init_dae') # Requests help from dsh for 'init_dae' command
dsh.init_dae
plugins = dsh.run #returns an array of plug-ins and writes them to plugins var.
plug-ins[0] #reaching first plug-in from the list
#... whatever you want to do
\endcode
<DIV CLASS="fragmentcaption">Listing 4: Running DTT in interactive mode</DIV> 

</DIV>

<DIV class="section">
        @section dtt_guide_02_working_with_dtt_test_suites Working with DTT test suites

Suites are files which contain the list of DTT scripts that should be run, and parameteres for these tests. These files should be created in YAML format. The list of the tests should be preceeded by <TT>tests:</TT> line. Then tests to be run should be described as a map  with the following members:

<UL>
   <LI CLASS="Bullet"><TT>name: </TT> - name of the test</LI>
   <LI CLASS="Bullet"><TT>enabled: </TT> - determines whether test will be run or skipped</LI>
   <LI CLASS="Bullet"><TT>args: </TT> - input parameters of the test</LI>
</UL>

The input parameters of the test should be orginized as a hash map. That means that all keys should start with ":" and look like ":plugin_spec: ".

Also suite may contain a section with the general parameters like:

<UL>
 <LI CLASS="Bullet"><TT>verbose: false</TT> which determines whether the output of the test in the console will be verbose or not.</LI>
 <LI CLASS="Bullet"><TT>timeoutFactor: "16.0"</TT> which defines the time period, after which test will exit in case it stuck on the execution of the certain peice of code.</LI>
</UL>

Example of the suite:
\code
suitesettings:
  verbose: false 
tests: 
#
# Cycle counting test
#
- name: DSH_TI_CycleCounts 
  enabled: true 
  args: 
    :plugin_spec: 'Digi,Pich,Psmm' 
    :sample_rate: 48000

\endcode
<DIV CLASS="fragmentcaption">Listing 5: Example of the DTT test suite</DIV> 

\note All the suites files should have an extension .gss

		@subsection autogeneration_of_the_suites Autogeneration of the suites

Sometimes it is necessary to generate the suites for the particular script for all the plug-ins from the bundle and/or for different sample rates. In this case instead of the copy-paste, which may lead to some mistakes and erratums, suitegenerator can be used. This is a special script, which takes as arguments the name of the script(s), for which the suites should be generated, and the list of their input parameters. Strange as it may sound, this data should be formed as a suite. Script itself is available as SuiteGenerator.rb along with other scripts in the DTT.

\note SuiteGenerator.rb can generate suites only for the two tests: DSH_SigCancellation test and DSH_TI_CycleCounts test

Here is the example of how to use this script to generate the suites for all the plug-ins from the 'D-Verb' bundle for all the sample rates the cancellation test %AAX Native vs %AAX DSP, and for the cycle counts test:

\code
tests:
# Generate suite for Cancellation test: AAX Native vs AAX DSP
- name: SuiteGenerator
  args: 
    :plugin_name: 'D-Verb'
    :path_to_audio_files: /Volumes/G_Audio/GS_Test_Resources/audio/
    :path_to_presets: /Volumes/G_Audio/GS_Test_Resources/PL_Settings/
    :test_script: DSH_SigCancellation
  enabled: true
# Generate suite for Instance Count test
- name: SuiteGenerator
  args: 
    :plugin_name: 'D-Verb'
    :path_to_presets: /Volumes/G_Audio/GS_Test_Resources/PL_Settings/
    :test_script: DSH_TI_CycleCounts_se
  enabled: true
suitesettings: 
  verbose: false
  timeoutFactor: "16.0"
\endcode
<DIV CLASS="fragmentcaption">Listing 6: Example of how to use SuiteGenerator script for generating suites for all the plug-ins from the 'D-Verb' bundle for the all sample rates for the cancellation and for the cycle counts test.</DIV>

To generate the suites, one should run this suite for the SuiteGenerator as an ordinary suite by executing the <TT>run_test.command \<the name of the suite for the SuiteGenerator\></TT> command. All the suites will be generated into the single file, which will be located inside the suites folder, and will be called like: 

<TT>dspVSnative(optional)_\<the name of the plug-in\>_\<the name of the test\>.gss</TT>

Examples:
<UL>
   <LI CLASS="Bulet">TRIM_DSH_TI_CycleCounts.gss</LI>
   <LI CLASS="Bulet">dspVSnative_TRIM_DSH_SigCancellation.gss</LI>
</UL>

</DIV> 

*/
