ePO\MA TS -- From Mattew

Common issues (ePO)
A far-from-exhaustive set of ideas to get you started.

Agent to Server Communication
•    Always review and collect both sides of the communication (masvc.log on endpoint and server_servername.log on handler)
•    In masvc, search for “connecting” from the bottom up to find the most recent connection attempt. Note the IP/hostname of the handler used (important for logging) and the port.
o    Test connectivity to the handler with telnet (telnet IPADDRESS PORT)
•    Curl error codes are searchable on Google. 503 message means check server log – not necessarily a real “server is busy” scenario.

Agent Deployment (Push Agent)
•    Review Server Task Log result and most importantly, server_servername.log (DB\Logs)
•    Keyword “push” in server.log – don’t forget that if multiple handlers exist in an environment, the push could be in a different server.log (when deploying, you can use the handler to use!)
•    Relies heavily on access to \\machinename\admin$ of endpoint. For all requirements and testing, see KB56386.

Replication failure
•    What type of repository? Think about the protocol used (SADR = connection to 8081 via macmnsvc; UNC = windows file sharing, etc.)
•    ePOAPSvr_Servername.log logs all replication activity and is in ePO\DB\Logs\ folder (MUCH more detailed than server task log information)
•    For SADR replication failures, valid analysis often requires both server-side (ePOApSvr) AND client -side (macmnsvc.log)  - if it IS a SADR, FIRST thing is use ONLY one: replication, or LazyCaching

Console login failure
•    Error messages on console? Read them carefully.
•    Orion.log in server\logs folder is critical. If in doubt, stop Application Server (Tomcat), delete or rename old orion.log, and restart service. The new log created will be “fresh” and the first errors at the top of the log will likely be the relevant ones.
•    If orion.log isn’t helpful, check and review errors in stderr.log, also within server\logs folder.
•    If console displays normally, but login fails – try a different account. Is Windows auth in play?
•    Clear browser cache/try a different browser/try from a different machine entirely.

Console performance
•    Check database fragmentation. Run the maintenance script in KB67184.
•    Restart the Application Server service. Does the issue go away? If so, for how long? 
•    Turn off debug logging if enabled in orion.log (KB86593)
•    If you stop all handlers (Apache/Event Parser services on ePO and remote handlers) does the problem go away? If so, this helps isolate issue.
•    INTERNAL KB78219 – Data collection for slow console issues (if none of above help)

ePO/Handler install or upgrade
•    All install/upgrade logs are in %temp%\McAfeeLogs folder
•    Always start in the “ePO5XX-Install-MSI.log” – this is the primary installer log
o    Find the error at the bottom of the log (typically 1603) and translate this into a new search string (for example, 1603 turns into value 3)
o    Search from TOP DOWN for the string (value 3, most of the time) – this finds the FIRST error, which is the only one that matters
•    Be wary of strings in MSI logs that start with the word “Property” – these are essentially saved variables, not actual error messages
•    ePO upgrades often fail and don’t roll back successfully (they appear to). Sometimes it’s best to err on the side of caution and do a DR (Disaster Recovery) before trying the upgrade again (though make sure to FIX the problem that caused the upgrade failure after completing the DR first!)
•    Use and love the Pre-Install Auditor (PIA). Always use the latest version available – if the customer isn’t upgrading to the latest build of ePO, you may need to download the latest separately.

Pull/Master Repository check-in failure
•    Try a different source site (McAfeeHTTP is default, configured in Server Settings – Source Sites)
•    Is it a repeating failure? Could be a one-off (check history in server task log)
•    All MR pull content is logged in detail in ePOAPSvr_servername.log (ePO\DB\Logs)
•    For check-in fails, verify no remnants of package in branch (Master Repository = ePO\DB\Software), and/or try a different branch

Database connection failure
•    Database connection is configured in the /core/config page. To access, example:
o    https://eposervername:8443/core/config (replace name, custom port if used, etc.)
o    Can be access remotely (or on ePO server itself, of course)
o    /core/config information is also stored locally, in the db.properties file (epo\server\conf\orion). If necessary, this file can be updated manually (always take a backup!)
•    Most common cause – Windows authentication, password changed. Always test new password before saving changes and restarting the ePO services.
•    If in doubt, use the SA account. It may need to be enabled (also known as mixed-mode authentication), but this account has the highest privileges, so if rights are in question, this can help isolate the problem)
•    Is SQL Express being used? It has a 10GB file size limit. When the limit is reached, ePO will be unable to login to the database. Check in SSMS by running the following query:
o    select @@version

 
Common Issues (McAfee Agent)

Injection
Injection can occur when third-party DLLs which either have untrusted certs or no certs at all load up with McAfee processes, like McScript_InUse.exe. In that scenario, updates will end up failing with curl error 28 (meaning a timeout) and will be seen in the McScript.log or the McScript_deploy.log. 

This is due to McAfee Agent’s Self-Protection functionality – the self-protection rules are working as designed in this scenario – we WANT to prevent the process (McScript_InUse.exe, in this example) from successfully making network connections because it could be compromised by a potentially malicious file. 

Example from McScript.log:

network      URL(https://172.24.208.16:443/Software/SiteStat.xml?hash={0e773b7e-9786-11e7-3115-73e51b00cce7}) request, failed with curl error 28, Response 0, Connect code 0, 
downloader        Downloading file from https://172.24.208.16:443/Software/SiteStat.xml?hash={0e773b7e-9786-11e7-3115-73e51b00cce7} to C:\Windows\TEMP\SiteStat.xml failed. 

The log that contains the rest of the data is called mfemactl.log. It will show entries like:

C:\PROGRAM FILES\MCAFEE\AGENT\X86\MCSCRIPT_INUSE.EXE>(7208) was blocked from accessing('CREATE' (1)) <AAC_OBJECT_SECTION:C:\WINDOWS\SYSWOW64\BMNET.DLL 

Run the sysprep tool first to see if there are any DLLs that it finds and trusts.

Scheduling
The scheduler service provides scheduling services to MA components and point products via the message bus. Scheduled tasks can be seen in the DB files (AgentDataDir\DB\) when they are decrypted via the MER tool. 
1.    Look for the matask.db file for the task assigned and its run time. 

The mascheduler.db will contain the task object ID, and internal non-configurable tasks (such as ma.cert.update.task.id and ma.property.collect.timeout.task.id). Example of an ODS task from the matask.db file:

- <Task timestamp="2018-12-24T18:43:31.000" priority="0" type="EAM_PolicyBased_Scans" name="On-Demand Scan - Full Scan" id="45" obj_id="193">
- <Section name="General">
  <Setting name="TaskType">PolicyBasedScan</Setting> 
  <Setting name="szGuid">ODS_TASK_ID_FULL_SCAN</Setting> 
- <Section name="Schedule">
  <Setting name="DayOfWeek">4</Setting> 
  <Setting name="GMTTime">0</Setting> 
  <Setting name="MaskMonthsOfYear">4095</Setting> 
  <Setting name="MonthOption">1</Setting> 
  <Setting name="RandomizationEnabled">1</Setting> 
  <Setting name="RandomizationWndMins">181</Setting> 
  <Setting name="RunIfMissed">1</Setting> 
  <Setting name="RunIfMissedDelayMins">50</Setting> 
  <Setting name="StartDateTime">20180516110000</Setting> 
  <Setting name="StopDateValid">0</Setting> 
  <Setting name="TaskRepeatable">0</Setting> 
  <Setting name="Type">2</Setting> 
  <Setting name="WeekNumOfMonth">3</Setting> 
- <Section name="Settings">
  <Setting name="Enabled">1</Setting> 
  <Setting name="StopAfterMinutes">0</Setting> 

For information about DayOfWeek, see TN300740.

Updates
One of the most common call drivers, “my DATs/AMCore isn’t updating” can have many causes but is generally troubleshot in the same three-step manner:

1.    Reproduce the issue
•    It’s possible to simply review historical data, of course (if the customer uploaded a MER, for example) but if you’re on a remote with the customer it is sometimes best to create and assign a new task (use an easily searchable name, like TestTask123). Remember that after assigning a new client task, the machine will have to communicate to receive the task (so send an Agent WakeUp or hit Collect & Send Props!)
2.    Confirm the task invoked and note status
•    To see where and when a task started, review the masvc_machinename.log. Search the task name from the bottom up – the first thing you find should be the result of the task (if it has completed). For example:
2019-01-09 17:00:14.426 masvc(444.4768) Updater.Info: Updater engine exited with exit status as 0 and  term signal 0.
2019-01-09 17:00:14.497 masvc(444.4768) compatservice.Info: is_compat_running: 1, is_compat_required: 1
2019-01-09 17:00:15.428 masvc(444.4768) scheduler.Info: The task Daily Update Task is successful
3.    Review logging
Which log(s) you need depends on the result. For example: 
•    Did the task start, but fail? Check McScript.log. In McScript, find the timestamp at the beginning of the update and search for the FIRST error – that is often the most important. For example, note the change between “I” (Informational) and “E” (Error):
2019-01-08 10:12:31    I    #9268    pm_service    [1] checking install order compatibility for <MAR_____1000>
2019-01-08 10:12:31    I    #9268    pm_service    getting spec entry for <MAR_____1000>
2019-01-08 10:12:31    E    #9268    pm_service    Entry not found in spec.
2019-01-08 10:12:31    I    #9268    pm_service    
2019-01-08 10:12:31    I    #9268    pm_service    check_conflict <start> for only installing software.
2019-01-08 10:12:31    I    #9268    pm_service    check_conflict <ends>
2019-01-08 10:12:31    I    #9268    pm_service    
2019-01-08 10:12:31    I    #9268    pm_client    checking final status

•    Was the task never received, or when you sent a WakeUp did the machine fail to communicate to ePO? Check masvc_machinename.log. Look for the most recent connection to ePO – search upward from the bottom for “connecting” and note the results. If the machine successfully communicated but never received the task, double-check your task assignments in the ePO console and verify extension are up-to-date (MA’s especially!)

Installation/Upgrade
Install logs will be in one of two locations – the temporary file paths:
•    C:\Windows\Temp\McAfeeLogs – For Push/Deploy Agent from ePO
•    %temp%\McAfeeLogs – For user-executed/local installs. 
See the Log file reference guide for log details. General recommendations include:
•    Start with the FrmInst.log. This is primarily useful for checking Agent installation status (at the very bottom of the log). The most common MSI errors are 1603 and 1602. 
•    After you’ve viewed the result (and timestamps, critically as always) switch to the MFEagent.msi.log. This contains the “good stuff” – if you had a 1603 error at the bottom of the log, search down from the top of the log for “value 3” (in English, this will change in other languages…so if you get no hits, double-check if the log is written in non-English). The first error observed from the top down is almost always going to be the only concern. Everything that happens after this error is merely a side effect of the initial failure.
•    Search the errors you find on the Agent portal. If you get no hits, switch to Teams or ask a colleague.
•    If you notice error 5 (Access Denied), check self-protection functionality: VSE, ENS, even the Agent’s own self-protection could somehow be blocking the Agent itself. Disable and test again if possible.

 
Non-Windows Agent Guide

Good news! The McAfee Agent itself is functionally identical between Windows and non-Windows platforms. The most difficulty technicians have is with basic navigation and data collection when it comes to the McAfee Agent on non-Windows platforms (including Linux and MacOS/OSX).
Keep in mind:
•    The McAfee Agent has separate packages for the different platforms. For example – a Windows package, a Linux package, etc. These packages must be checked in to the ePO Master Repository separately. It’s not a one-size-fits-all situation!
•    The McAfee Agent can still be deployed (Push Agents) to non-Windows platforms, however it works entirely differently. Since a Windows deployment utilizes Windows file sharing, obviously that’s impossible for non-Windows clients. Instead, SSH protocol is utilized (port 22 by default). Red Hat/centOS have specific requirements to enable deployment an is a common source of push failures. See the McAfee Agent Installation Guide for details.
•    The Agent still has three services on non-Windows platforms: masvc, macmnsvc and macompatsvc.
•    Non-Windows platforms are case-sensitive when working in the terminal/command line. Be wary to make sure your cases match, otherwise it will appear that the locations you’re attempting to access do not exist.

Basic command-line syntax
pwd – lists current directory (where am I?)
cd – change directory. Example: cd /var/McAfee/agent/bin/
ls – list contents of directory
find / -name – search for a file or directory from within root. Example: find / -name McAfee
uname -a – list out kernel information/OS details.
df -h – shows partition/disk space uutilization.
ps aux – list all running processes
top – similar to task manager in Windows, but keep in mind that load/usage is calculated differently.
man – view the manual for a command, listing out switches and instructions. Example: man vi

Opening and reading logs
Use vi, a default text editor present in most non-Windows platforms. Vi is incredibly powerful and contains nearly any functionality you’d desire (far beyond something default to Windows, like notepad), though advanced functionality will require reading beyond what is reasonable to include in this guide. 
Opening a file in vi is simple. For example, just run vi and point it to the path of the log. For example, to open masvc log (you can use tab when typing out commands in terminal to “auto complete” the path based on what exists in that location. Watch your case!)
vi /var/McAfee/agent/logs/masvc_machine.log

Commonly-used commands in vi:
•    To go to the bottom of the log file - SHIFT-G
•    To search for a string - / (example /connecting to find the word “connecting”)
•    To find the next occurrence of a string you searched for  - n 
•    To find the LAST occurrence of a string you searched for (searching up) – N
•    To close the file without saving - :w!
•    To save a file without closing: :w
•    To close and save: :x
For more, search Google!
Another commonly-used tool on non-Windows platforms is tail. Tail opens a text file and allows you to follow its movement with the -f switch (follow) – this is especially helpful for watching issues as they happen – for example, if you can reproduce a failure when running an update task and want to ‘watch it happen’ instead of just reviewing the results afterward with vi. 
For example, to monitor the McScript.log during an update:
fail -f  /var/McAfee/agent/logs/McScript.log
at times, following the log can roll VERY quick and you miss the last few lines. To quickly review what was last logged you can type:
tail -fn100 /var/McAfee/agent/logs/McScript.log
It will show you the last 100 lines (you can change that number to whatever you like)
To exit out of tail, use CTL-Z.

Log locations and data collection and service information

MacOS
MAC MER tool: KB86785
/Library/McAfee/agent/ (install files)
/var/log/install.log (to view install logs)
/var/McAfee/agent/ (data directory: includes logs, db files, etc.  Equivalent on Windows is ProgramData)
/etc/ma.d/ (product plugins)

To view the status of a service:
Sudo /Library/McAfee/agent/scripts/ma status

Stopping and starting services:
Sudo /Library/McAfee/agent/scripts/ma start
Sudo /Library/McAfee/agent/scripts/ma stop
Sudo /Library/McAfee/agent/scripts/ma restart

All other non-Windows platforms (Linux, UNIX, etc.)
Linux MER tool: KB83005
/opt/McAfee/agent (install files)
/var/McAfee/agent (data directory)
/etc/ma.d/ (product plugins)

On Linux builds, you can view the status of a service anywhere within the terminal session by typing:

Service ma stop
Service ma start
Service ma restart

SQL Reference Guide

SQL Server Management Studio (SSMS) – The freely available tool used by most administrators to access, configure and manage their SQL databases (including the ePO database(s)). Can be installed anywhere – not just on the SQL server itself. Not installed/included by default when SQL is installed.
SQL Instance – The unique SQL Server service (typically named, or with the default named of MSSQLSERVER) that is capable of hosting multiple SQL databases. In larger environments (25,000 nodes roughly), ePO should have its own instance with no other databases present for performance reasons.
SQL Configuration Manager – A built-in configuration tool installed with SQL Server. Primarily used in support for setting/checking the port that an instance is bound to.
Data file (.MDF) – The physical file which contains the SQL database itself. Must be paired with a matching log file (.LDF).
Log file (.LDF) – The physical file, also known as the transaction log, which contains temporary transactional information associated with the SQL database.

Backing up/restoring the database
•    In SSMS, log in and locate the ePO database by expanding the “Databases” section. 
•    Right click the database and select Tasks – Back up.
o    Default options here are usually fine. Make sure it’s set to “Full” backup type and that the destination is correct. Remember that multiple files configured in the destination box means that the database will be spanned across these files – in other words, you’ll need both pieces to restore the database. For this reason, stick with one file. (To make things easier on yourself, check the “Backup Options” tab and enable backup compression. This will reduce the file size, making it easier to handle).
•    To restore, a similar process – log in to SSMS and right click the Databases header. Choose “Restore Database” and select the “Device” option (this means restore from a device/disk instead of from another database itself). Then, browse out to the known-good database .bak file using the (…) button. Again, default options should be fine in most cases. 

Maintaining the database
•    Take regular backups. Backups can be done manually, or automated using the SQL Server Agent. The SQL Server Agent is non-default functionality that can ONLY be added to full instances of SQL (SQL Express has no such feature). We do not configure this for the customer, but we can point them in the right direction.
•    Run a regular maintenance task. This task, and the script itself, are attached to KB67184. Like backups, automated and regular execution of the maintenance script must be accomplished using SQL Server Agent, otherwise it can simply be executed manually (though this is obviously difficult to do every night!)
•    Run regular data purges from within the ePO console. These are Server Tasks and can be configured in any environment. By default, ePO contains a task called “Purge Threat and Client Events Older than 90 Days,” though this should be enabled and configured to run daily – that way it only purges one day of data at a time (resulting in a quicker execution). A customer’s data retention policy will vary based on their requirements.

 
Useful KB Reference Guide

Disaster Recovery:
ePolicy Orchestrator server backup and disaster recovery procedure
ePolicy Orchestrator cluster backup and disaster recovery procedure
Overview of the ePolicy Orchestrator 5.x Disaster Recovery Snapshot feature

SQL:
How to collect data from the SQL Server for troubleshooting ePolicy Orchestrator and SQL-related issues
SQL permissions required to install and use ePolicy Orchestrator
Recommended maintenance plan for ePolicy Orchestrator databases using SQL Server Management Studio
How to access the Database Configuration page and set SQL authentication account information in ePO
ePolicy Orchestrator supported SQL collation types

Install and Upgrade:
Supported platforms, environments, and operating systems for ePolicy Orchestrator
ePO data collection for ePO version upgrade or patch installation failure
ePolicy Orchestrator installation and update checklist for known issues
Installation or upgrade to ePolicy Orchestrator 5.9.x or 5.3.3 fails when using SSL connection for SQL Server
Upgrade to ePolicy Orchestrator 5.9 fails (a third-party application generated a self-signed certificate)

Server tasks:
ePolicy Orchestrator 5.1 or 5.3 server tasks do not expire after reaching abort time
An Agent Wakeup call or Agent Push server task does not expire if the client computer cannot be reached
An unexpected error occurred (when accessing dashboards; server tasks do not run)
LDAPSync server task never completes or takes an excessively long time to complete
Server task configured to execute the 'Run Client Task Now' task, might run indefinitely, or get stuck 'In Progress'
Some server tasks are stuck indefinitely showing 'in progress' and the EPODataChannelDataMT database table is excessively large

Apache and server-side communication:
How to capture Apache -X output for troubleshooting Apache service issues with ePolicy Orchestrator
Versions of Apache, Tomcat, and Java used by ePolicy Orchestrator
Support for custom changes to the Apache configuration in the httpd.conf file
INTERNAL - ePO data collection for an application crash on an ePO process (Apache, EventParser, or Tomcat)
How to update the cipher suite used by Apache and Tomcat in ePolicy Orchestrator to remove outdated ciphers
Port conflicts with ePolicy Orchestrator services when netstat only displays listening process as belonging to System

Tomcat and console problems:
INTERNAL - ePO Data collection for slow or unresponsive ePO console issue
Several dependency errors display when attempting to log on to the ePO console
Extensions are disabled (unable to log on to the ePolicy Orchestrator console after upgrade to ePO 5.x and system restart)
The license for ePolicy Orchestrator is invalid (when accessing the ePO console log on page if the connection with the SQL database fails)
Unable to log on to the ePO console after migrating ePO to a new server
 
Tools/Links

MER Analyzer: Useful for reviewing MER results. Allows for built-in decryption of MA database files. Includes file browser options to make finding logs easier.
\\ca-server\Products\McAfeeB2B\Supportability\MER Analyzer\

B2B file share: Contains most software releases and versions, including old and EOL software. Useful for internal reproduction purposes or for providing ancient file releases to customers.
\\ca-server\Products\McAfeeB2B\

NFS case share: Save this link to quick access on your COE machine for easy access to the network location where all case files are stored. Simply add the the SR number (including 4-) to the end of the network path.
\\dnvcorpvf2\nfs_dnvspr\

Agent portal: Internal-only portal that allows for all-inclusive searching, including KnowledgeBase, Bugzilla, Sharepoint (previously Planet), and Service Requests. 
https://agent.mcafee.com/

Product Documentation: Customer-facing portal for accessing wiki-style product documentation. Includes release notes, product and installation guides, etc. Does not include KnowledgeBase articles.
https://docs.mcafee.com/

Consult Advanced Support Process: Internal PR (process) document that describes the accepted support process for officially consulting the Advanced Support team.
https://kb.mcafee.com/agent/index?page=content&id=PR500746

Products by Vertical – Includes transfer numbers, Insight IDs, etc.: Useful when dealing with a product you don’t normally support, this list includes nearly every McAfee product along with the internal number to transfer calls to support, and the Insight ID for saving the appropriate point-product code.
https://kb.mcafee.com/agent/index?page=content&id=PR500305

Who’s my SAM? – Enterprise SAM list: More useful once you start taking Enterprise calls, but may come in handy if you need to point an Enterprise customer to their SAM (Support Account Manager).
https://mcafee.sharepoint.com/sites/Platinum/Lists/Account%20List/accountsbymanager.aspx?viewid=79694453-28a7-467e-908e-49da9792d6ea