SEP client install fails and rolls back

Posted on Updated on

When you try to install the Sympantec Endpoint Protection (SEP) client, it’s possible it rolls back after being confronted with an error. First everything seems to go smooth till you see the “Rolling back action” appear in the installer’s GUI. Aaargh… Of course the installer doesn’t tell us what exactly went wrong, so we have to find out ourselves. That’s exactly what happened to me. Here’s my story 🙂

First thing to do is checking for events in the Windows event logs. We’re lucky as the following event is logged in the Application event log of Windows:
Type: Information
Source: MsiInstaller
Category: None
Event ID: 11708
Description: Product: Symantec Endpoint Protection — Installation operation failed.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Mmm… It just logs the fact the installation has failed. Of course it’s a good thing this is logged, but when trying to find out why this failure has happened, this kind of logging is worthless! It doesn’t even give us an indication! Anyway, it’s time to take a look at the full blown install log, which is SEP_INST.LOG in %temp%. In my case this file’s size was 7 to 8 megabyte! Exploring this file takes a while, but in my case I was lucky to find the very first failure:

MSI (s) (5C:04) [09:52:26:133]: Invoking remote custom action. DLL: C:\WINDOWS\Installer\MSI60.tmp, Entrypoint: RegWithLiveUpdate
LUCA: RegWithLiveUpdate
LUCA(1157): CustomActionData=Add SESC Virus Definitions Win32 v11 MicroDefsB.CurDefs SymAllLanguages Antivirus and antispyware definitions {C60DC234-65F9-4674-94AE-62158EFCA433} {855BA5F4-6588-4F09-AE61-847E59D08CB0} 3 {DA47E166-7F7A-4039-9768-7AFFB5E99CE8} Add SESC Virus Definitions Win32 v11 Hub SymAllLanguages Antivirus and antispyware definitions {B36CDA3C-B15B-421c-A2A4-7EC70E3B852B} 0 {DA47E166-7F7A-4039-9768-7AFFB5E99CE8} Add SESC Submission Control Data 11.0 SymAllLanguages Submission Control signatures {4F889C4A-784D-40de-8539-6A29BAA43139} 0 Add SESC AntiVirus Client Win32 11.0 English Symantec Endpoint Protection client {678BF7F9-F8E9-468b-B890-F55E159CAA3C} 0
LUCA: HandleLiveUpdateAction
LUCA(874): HandleLiveUpdateAction: Action=”Add” Product=”SESC Virus Definitions Win32 v11″ Version=”MicroDefsB.CurDefs” Lang=”SymAllLanguages” Description=”Antivirus and antispyware definitions” GUID=”{C60DC234-65F9-4674-94AE-62158EFCA433}” CallbackCLSID=”{855BA5F4-6588-4F09-AE61-847E59D08CB0}” CallbackFlags=”3″ Group=”{DA47E166-7F7A-4039-9768-7AFFB5E99CE8}”
LUCA(1020): Exception calling IluProductReg RegisterProduct
LUCA: HandleLiveUpdateAction: COM Exception:
LUCA: Class not registered
LUCA: Call to HandleLiveUpdateAction FAILED.
Action ended 9:52:26: InstallFinalize. Return value 3.

It seems the function RegWithLiveUpdate has failed, while trying to add virus definitions. This has caused an exception and the function returns error code 3. It’s not clear why the virus definitions couldn’t be added and thus why this function has failed. Yup, we know more now, but we still don’t know the exact cause, let alone how to solve the problem.

Wait a minute… once an “old” version of the SEP client was installed on this machine. Perhaps I should run CleanWipe first? CleanWipe is a tool of Symantec that cleans your system as to Symantec products; it probably doesn’t surprise you that many uninstalls don’t uninstall everything (sometimes with a reason, wometimes not) and this statement is also valid for the SEP client. Fulfilled with new hope, I downloaded CleanWipe, cleaned as much as I could and restarted the server (which was recommended by CleanWipe). Then I tried again to install the SEP client, but… still the same issue…
Perhaps CleanWipe hadn’t cleaned literally everything? I looked for the last traces and found several references left in registry and file system (mmm, this is not what I would expect from a tool named “CleanWipe”). But even after deleting those the issue remained…

Perhaps there was something wrong with the requirements? Was my system compliant with the requirements of the SEP client? To check this I downloaded the Symantec tool “Symantec Endpoint Protection Support Tool” and checked to see if there were issues with the server. Nope, no problem, everything should be fine for a perfect SEP client install… Sight… BTW, starting from SEP client version 11.0.5002.333 this tool is included in the client itself, but I couldn’t use it this way as, well, the client wasn’t installed, remember? 🙂

The weird thing was I had this issue on only 2 machines and both of them were almost identical, each running Windows Server 2003 SP2 R2 and ISA Server 2006. But in the past an install of the SEP client was never a problem on those machines and I couldn’t even find a single note in an article or forum thread SEP wasn’t supported for ISA Server: at the contrary!

(You could wonder why those machines had a SEP client once, but were removed later. Well, in the beginning the SEP client was installed on both machines without a single issue. After a while both SEP clients didn’t run correctly anymore (see below) and I decided to uninstall both clients. Of course I needed to reinstall the SEP client on those machines again and rebuilding the servers wasn’t really an option on a short term. But as I’ve just explained this reinstall didn’t work out it should have been…)


It was time to try another SEP client version. So I decided to try a newer version I had already downloaded a while ago. Nope, didn’t work. Then I even downloaded the very last version (11.0.6 MP3), but even this one failed to install.

I guess I’ve read every article and forum thread about this failure, but all of them were unresolved or their solutions didn’t work for me. I’ve checked DCOM settings, NTFS permissions, registry permissions, deleted every piece of Symantec reference I could find, used all the tools Symantec has offered,… but to no avail. It was time to dig deeper 🙂

I decided to try again and monitor with Process Monitor in the meantime. I had to filter out A LOT, but it was worth it. I had the timestamp of the just mentioned error in the SEP client’s log file and saw Process Monitor was dealing with certificate stores. Wait a minute… perhaps the SEP client installer deals with a package (the virus definitions?) that’s signed and tries to validate the signature? And perhaps the SEP client installer couldn’t find a trust for the root CA of the certificate used for this signature? Of course!! A while ago I had customized the Trusted Root Certification Authorities\Certificates folder of the system’s certificate store (I’m telling you why in a minute). This way a lot of root CA certificates were removed from this folder. One of them could be needed for the installation and start of the SEP client (remember my very first problem was the SEP clients started to behave badly)! Yup, indeed, after some trial and error I found which certificate was missing from the customized folder: Class 3 Public Primay Certification Authority! Add this certificate to this certificate folder of the system’ certificate store again and the SEP client installs as smooth as it can be!

(Oh yes, you’re probably wondering why the hell I customized the trusted root CA list of the machines? Well, if this list is too big, there is some sort of a corruption. I’m going too fast, rewind, play: our servers with ISA Server 2006 are used as reverse proxy servers. One thing they do is publishing web applications (by bridging by the way). For some of those applications client certificate authentication is required. The end user contacting such an
application over the Internet is asked to select a client certificate (well, it can be selected automatically by the browser too, it depends, but I feel we’re deviating if I explain this). It’s possible a user has many certificates installed, some of them with a certificate path with a root CA not trusted by the reverse proxy server. To avoid an access denial and to make the list smaller for the user, the client should be able to select a certificate among all its certificates supported by the reverse proxy server (well, at least supported in the sense their root CA is trusted by the reverse proxy server). This means the client’s browser should know which root CAs are trusted by the reverse proxy server and this is indeed the case: when the client contacts the server, the server provides the client with a list of its trusted root CAs. If this list is too big, actually an incorrect list is provided to the client without failures or error messages, so it’s possible you’re not even aware of this problem! That’s why this list should be stripped down till what’s necessary on some machines. We only left those certificates in the list that Microsoft states as necessary to keep Windows working, some other certificate not stated by Microsoft but which
later appeared to be needed if you want to use Microsoft Update (MU) and our own explicitly needed root CA certificates. We were not aware of the need of “Class 3 Public Primay Certification Authority” though and that’s why and when problems began to rise…)

Note that there are many other reasons why an installation could fail and roll back. I guess my situation was quite rare, but still it’s not a bad idea to check if your system’s certificate store has the aforementioned certificate as a trusted root CA. Be aware to check the system’s certificate store, not your personal another cert store (it’s possible the certificate is residing there, as the final trusted root CA list is the superset of the system’s list plus your own additions, and it’s possible this can hide some problems (haven’t tested this out though), but at the end will never be a complete solution as the SEP client doe sthings at system level). Also, check the Trusted Root Certification Authorities\Certificates folder and not the Third-Party Root Certification Authorities\Certificates folder or another one. If you’re missing the certificate you can copy it from Third-Party Root Certification Authorities\Certificates to Trusted Root Certification Authorities\Certificates (drag and drop is moving by the way, I recommend you copy/paste it). I’m not going to explain the differences between those folders or give extra information about certificates, validation, root CAs, etc. This is out of scope for this article. Just remember a certificate has a root CA and its certificate must be trusted by adding it to the right folder in a certificate store of Windows.

One last thing some of you would like to know is how to access those certificate stores. Most of you use Internet Explorer (IE) for this, but this only offers a subset of the whole thing. For the whole thing start the Microsoft Management Console (MMC) by starting mmc.exe and then load the Certificates snap-in. That’s it! 🙂

Downloads:
* CleanWipe: contact Symantec Technical Support and open a support case (by calling them or opening a web case). You should be able to achieve CleanWipe then. AFAIK the latest version is 6.2.
* Symantec Endpoint Protection Support Tool: http://www.symantec.com/business/support/index?page=content&id=TECH105414
* Process Monitor: Microsoft tool from the hand of Mark Russinovich and Bryce Cogswell. Latest version is 2.95 and can be retrieved from http://technet.microsoft.com/en-us/sysinternals/bb896645.

Anyway, problem solved for me, but it was a tough one to tackle!

Advertisements

4 thoughts on “SEP client install fails and rolls back

    Anonymous said:
    15/11/2011 at 08:36

    Awesome Technical Analysis and problem resolution!Loved the way you described the troubleshooting process!! Respect 🙂

    Debar said:
    27/06/2012 at 10:39

    GREAT article. I had also an issue on server with an older version of Symantec. I have searched for a few days, prototyped the upgrade to SEP and ended in the same issue as you . I had also understood a certificate was missing (for the same reasons as you). Your article have saved me many hours trying to find which certificate was missing. Also, I was able to return the current installation of the antivirus to a stable state by importing the certificate.

    Anon said:
    26/09/2014 at 21:48

    Helped out on a Win XP machine. Used Procmon and found it hitting the certificate stores. Now it seems Symantec has an article: http://www.symantec.com/business/support/index?page=content&id=TECH217284

    Maurizio said:
    22/08/2015 at 10:17

    In my case (windows server 2008 32 bit) SEP 12.1.2015.2015 failed to start after an inplace upgrade from windows server 2003. after a lot of work and research through the internet, i finally made it as following:
    1. Downloaded CleanWipe tool from https://fileshare.symantec.com/
    LoginID: cleanwipeutility
    Password: CL3@nw!p3
    you receive a zip password-protected – the password is: symantec
    2. run CleanWipe and rebooted
    3. After reboot, opened a privileged regedit and scanned for ALL occurences of “Symantec” and deleted them at the key level (e.g. in the Components subtree found the word “Symantec” in the right pane – tabbed to the left pane and climbed the subkeys tree until reached the key level, then deleted the whole key)
    4. Deleted the folder c:\program files\symantec – c:\program files\shared files\symantec – c:\program data\symantec
    5. installed SEP
    HTH someone else
    Mauri

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s