GFM TAE runs

NOTE: This article is specifically about the regular, centralized TAE runs executed for the GFM regional teams on private builds from REGF branches. This article is NOT about the TAE runs executed by the SE team.

 

Having centralized, regular TAE runs will help capture cross-feature impact and regressions earlier, as well as help testers identify unstable automation before it is handed off to SE.

 

 

 

 

Which branches does this apply to?

• This will apply to builds from the DAX5HF_REGF (Ax2009 RTM) and DAX5SP1HF_REGF (Ax2009 SP1) branches.

 

Which tests are executed in TAE runs?

• Initially, only those tests added to the current Rollup will be executed.
• More specifically, tests will be picked up by the TAE queries based on the following properties:
  • Is a testcase (eg. $/Granularity/Test Case)
  • Is Enabled (ie. Disabled = False)
  • Is under $/Ax5.0/GDL/
    • Path does not contain "RFH" (to exclude SE hotfix testcases)
    • NOTE: For runs on each specific layer, the path is refined to only capture the relevant testcases
  • Has Milestone attribute set to the current rollup (eg. $/Milestone/SP1/RU4)
  • Has Automation Status set to Complete (eg. $/Automation Status/Complete
  • NOTE: The query does not search for the Frequency attribute since all WTT testcase jobs which have not yet been handed over to SE should not have $/Frequency/Weekly attribute set.
• In the future, this will expand to include more GDL tests.
• See specific queries here:
  • Ax2009 RTM for SYS
  • Ax2009 SP1 for SYS
  • Ax2009 SP1 for GLS_CONS
  • Ax2009 SP1 for GLS_EE_RU
  • Ax2009 SP1 for GLS TR

 

Which layers will be used in TAE runs?

• Separate TAE runs will be scheduled with individual layers installed:
  • For Ax2009 SP1, we will have runs for SYS, GLS_EE_RU, and GLS_CONS.
  • For Ax2009 RTM, we will have runs only for SYS

 

Which machines will be used for TAE runs?

• The CoreServices team will provide 5 machines on which the TAEs will be run.
• TAE test machines can be found in the JobsRedProd datastores under $/Lab/TAE/DAX5SP1HF_REGF
  • TODO: Søren will later provide 5 machines in the Vedbaek datastore (owner = mikebc).
• Permissions to this node are given to Test Leads & Managers.
• In the future, this will expand to cater for the number of testcases included in the run.

 

What configuration/platform is used for TAE runs?

• The basic configuration is used on the TAE test machines:
  • Windows Server 2003 x86
  • SQL Server 2005
  • Standard setup

 

When are TAE Runs scheduled?

• TAEs for all layers/branches will run once a week.
• The schedule is as follows:
  • Machines prepared Friday with the week's best builds.
  • TAEs kicked off on Monday.
  • Status mail is sent out once TAEs are done executing.
  • Analysis begins and a final analysis report is sent out on Friday.

 

Who kicks off and maintains the TAE runs?

• Lucius Lu (v-lucilu) from Beyondsoft is currently assigned to kicking off the TAE runs.
• Lori (Liangli Huang from Beyondsoft) is the person responsible for ensuring we always have someone assigned to kicking off the TAE runs.
• NOTE: The Beyondsoft vendors are located in China and report to Chris Mitchell.

 

Who is responsible for analyzing TAE run results?

• TAE failures will be analyzed by Beyondsoft using the following process:
  1. Failure logs are analyzed and, if necessary, tests re-executed on the same TAE build.
  2. If a product/test issue has caused the failure, a bug is logged in AXSE database following the standard bug guidance (TODO: Need this from SE/Kamil)
  3. The bug is linked to the WTT Result for the failing testcase (all failures must have a linked bug).
  4. The WTT Result for the failed testcase is marked as Status=Completed to indicate analysis is done.
  5. When all failures have been analyzed, the report mail is sent out containing the list of bugs found.

 

Who fixes failures resulting from TAE runs?

• Depending on whether the issue is a product or test bug, the developer and tester of the feature team causing the failure should fix the bug.

 

Where can I go to find out about TAE results and progress?

• WTT TestCase Results are added to individual result collections in JobsRedProd datastore with the following pattern:
                   GFM_TAE_RU4_[RTM/SP1]_[SYS/CONS/EERU]_[BuildNum]
• A TAE run mail is posted to the Dynamics Ax GFM Rollup TAE (DAXGFMRT) discussion group.
  • Join this group to keep informed of TAE progress.
  • If you wish to compare with tests running in SE's TAE (HF/STAB branches), then also join group DAX SE TAE Info.