Syed Khairuddin

Symantec Endpoint and Network Related Issues

Last night I received a complain from my customer saying that his Excahnge 2007 CCR cluster has failed on both the nodes. Upon troubleshooting I found that File Witness share was not access from both the nodes which made Exchange CCR go down. I checked all the permissons on the fileshare withness and they were Ok. Then I confirmed with the customer if he had lately installed  patches or any applications on the Hub transport server where the File witness share was hosted and to my question he replied he has changed the Antivirus to Symantec EndPoint. I stopped the AV services restarted the Hub transport server and every thing Worked fine.  After things went fine I started searcching Microsfot KB articles if any Symantec End Point  related issues were published and to my eyes I found 4 KB articles which describes the problem.

·        Error message: “The network path was not found” or “The specified network name is no longer available” when attempting to open shares, map a drive, run DCDIAG to the to the affected server, use netdom to reset secure channel

·        Error message: “RPC Server is unavailable” when trying to connect via Active Directory Users and Computers

·        Error message: “RPC Server is too busy to process the request” when attempting to join the Windows server domain

·        Error message: “No network provider accepted the given network path” or “File or network path no longer exists” when copying a file over the network to affected servers

·        Printing issues (cannot update printer IP address via DNS)

·        AD replication failures

·        Cluster service fails to start, or inability to access existing File Share resources even if they are online according to the Cluster Administrator snap-in

·        Event log Event ID 4226 and or 2022 may occur frequently (up to every 20 to 30 seconds)

Related KB Articles

KB 961293 Unable to access Shares "The specified network name is no longer available" when Symantec Endpoint Protection prior to 11.0.4202 (MR4-MP2) or Symantec Antivirus 10.2 are installed on a Windows 2003, 2008 or 2008 R2 Server

KB 961654 A file sharing connection to a Windows Server 2008-based server drops unexpectedly if the server has Symantec Endpoint Protection prior to 11.0.4202 (MR4-MP2) or Symantec Antivirus 10.2 installed

KB 948732 Network shares become unresponsive after some time on a Windows Server 2003 or 2008 or 2008 R2-based-based computer running Symantec Endpoint Protection prior to 11.0.4202 (MR4-MP2) or Symantec Antivirus 10.2, and you receive an error message

KB 923360 You may experience various problems when you work with files over the network on a Windows Server 2003-based or Windows 2000 Server-based computer

Windows Server 2008 Domain Controllers fails NcSecDesc with DCDIAG

One of my firends just introduced Windows 2008 DC in Existing  Windows 2003 Domain, by following all the prerequisites mentioned on technet website  http://technet.microsoft.com/en-us/library/cc754670(WS.10).aspx and succeeded. Then to make it sure he ran dcdiag on Windows Server 2008 domain controller and ended up with error the Naming Context Security Descriptors  (NcSecDesc) test fails. The test passes for Windows Server 2003 domain controllers in the same domain.

The Error comes when you dont run adprep /rodcprep while preparing the Forest. In his case he did not ran the command because he never wanted to implement RODC in the future.

According to the KB article http://support.microsoft.com/kb/967482

If you do not plan to add an RODC to the forest, you can disregard this error. If you plan to add an RODC to the forest, you must run adprep /rodcprep.

Conclusion: Please ignore the error in case you havent the run adprep /rodcprep.See below the log

Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn’t have Replicating Directory Changes In Filtered Set access rights for the naming context: DC=DomainDnsZones,DC=Goodies,DC=COM

Error NT AUTHORITY\ENTERPRISE DOMAIN CONTROLLERS doesn’t have Replicating Directory Changes In Filtered Set access rights for the naming context: DC=ForestDnsZones,DC=Goodies,DC=COM

……………………. Server2k8 failed test NCSecDesc

How to Backup Share Permissons

Backup and Restore of Share Permissions

To backup share permissions, export the Shares registry key.

1. Open Regedit to the following location:
   
   HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Shares
   
2. Right-click the Shares registry key and select Export.

When You want to restore it you can simple click the file you have exported.

Note: See the attached file

 http://www.virmansec.com/cfs-filesystemfile.ashx/__key/CommunityServer.Components.PostAttachments/00.00.00.06.38/Shares.JPG

RDP Black Screen

I must say it was really a weird issue which made me scratch my head for hours together. I had a Windows 2003 Server which was behaving was having a typical problem, whenever I was trying to do remote desktop i was getting Black Patches on some on some parts of the screen. In the beginning I thought it might be a problem with VGA drivers or some third party products so I uninstalled all of them but nothing worked. At last I thought it would be nice if I can export the registry entries of the other running server from HKEY_USERS\.DEFAULT\Control Panel\Colors and import it. Here are the steps i am listing down

1)On a working windows 2003 system
2)Open regedit
3)Connect to remote registry
4)Navigate to HKEY_USERS\.DEFAULT\Control Panel\Colors (on the effected system)
5)Create a export of the current key
6)Go to the working system
7)Create a export of HKEY_USERS\.DEFAULT\Control Panel\Colors
8)Import the working registry settings to the remote registry system that is effected by the black screen
9)Reboot the effected system the color should be restored now.

 Note: Please see the link for screen shot 

 http://www.virmansec.com/cfs-filesystemfile.ashx/__key/CommunityServer.Components.PostAttachments/00.00.00.06.37/Black.JPG

Server Health Checks

As a  Technical Consultant I usually visit  my customers for checking the health of the servers, troubleshooting and discussing the new implementations scenarios. Listed below are some of the best practices which I follow when troubleshooting or checking the Server healths. Lets divide the server health checks in 2 parts Hardware realted checks and Software Related Checks.

 Check CPU Hardware:

1) Open Device Manager  underneath the Processors make it sure that no CPU’s have red cross marks (X) or yellow Exclaimation Marks (!). If you find this then please take the vendor support.

2) Check CPU Usage from the Taskmanager ensure that there are no processes consuming excessive CPU. I usually use Process Explorer from Sysintrnals to trouble shoot high CPU spikes.

 I had an Issue with one of my customer  the applications and the Operating System on the behavious was sluggish and the problem turned out was one of the processors clock was mismatched. There might be lot of problem which makes the OS sluggish.

Memory:

1)Open Task Manager select the Performance tab look at the Physical memory box, and multiply the total memory by 2,If the total available memory is less than this number then the Server is currently utilizing more than 80 percent of the memory.

Hard Drives:

 Lot of times I have seen my customers ignoring the disk space and ending up with serious problems. Do check the Disk space and
Validate that each disk has more than 10 percent of free space.

I had a customer who was facing problem with Excahnge Services hanging that was due to disk space.

Network Controllers

Verify the connectivity between the NIC and the Switch is fine.On the back of the server verify you have a green blinking link light on the NIC port. Check the Drivers updated.

Note: Microsoft Product Services Support  does not Supports NIC teaming on the Domain Controllers. 

http://support.microsoft.com/kb/272294

Event Viewer

Event logs are one of the most important logs which are used for troubleshooting scenarios. Events have 3 categories in the event viewer.

Informational: Noted with a white icon and letter ‘i’. Successful operations are logged as informational.

Warning: Noted with a yellow icon and exclamation point. These usually are looked up as they serve as predictive future failure indicators, such as disk space running low, dhcp ip address lease renewal failures, etc.

Error: Noted with a red circle icon and ‘x’. These are indications that something has failed outright and are a good starting point for troubleshooting.

Note:  EventCombmt is a multithreaded tool that you can use to search the event logs of several different computers for specific events, all from one central location.

http://www.microsoft.com/downloads/details.aspx?FamilyId=7AF2E69C-91F3-4E63-8629-B999ADDE0B9E&displaylang=en

Services:

  Each server will have specific set of services depending upon the Application Installed (Like Exchange has different set of services to run the application on the Operating System). These services are very crucial to run the application failing of one service of a particular application can make your application unresponsive.

Note: Have you ever noticed an error  "At Least One Service Or Driver Failed"  when you restart the server. in this case you can go to services and startup type and check the Automatic Services. If the Automatic Services are not started correctly while the server is booting this error pops. You can always troublshoot by arranging them with startup types.

Name Server Resolution (DNS).

One of the important thing you always have to check is your DNS server which is responsible for name resolution on the entire Network. If you dont have this confgured correctly then you have problems with Active Directory and Exchange. The Command  DCDIAG /TEST:DNS is used to validate DNS health.

Microsoft Product Support Reports

This tool is used to collect the information of all the Services running on the Server and its heavily used by Microsoft Product Support Services group.

Read here about the utility http://support.microsoft.com/kb/818742 

Download and try from here http://www.microsoft.com/downloads/details.aspx?familyid=cebf3c7c-7ca5-408f-88b7-f9c79b7306c0&displaylang=en

http://blogs.technet.com/askperf/archive/2009/05/01/two-minute-drill-the-new-mps-reports.aspx

Note:

Each application type of server needs its own set off health checks.  For example web servers, terminal servers, Excahnge Servers and database servers. This is just the baseline for each server. You have to diagnose accordingly.

Windows 2003 DNS issue A Records are continuously dissapearing and DHCP SERVICE Access Denied

Last couple of days back I received a call from one of my friend describing a typical problem he was facing with his Windows 2003 DNS server. So I took the remote session and started troubleshooting the problem step by step and came to know that  DHCP client service was not started on the Server which is the most important service on the Windows 2003 DNS SERVER. Dynamic DNS registration relies on the Dynamic Host Configuration Protocol (DHCP) client service to perform dynamic updates. When you disable or set the DHCP client service to start manually, it prevents dynamic DNS updates from occurring. Even if the client or server uses a static Internet Protocol (IP) address, the DHCP client service must be running for dynamic DNS updates to occur.

http://support.microsoft.com/kb/264539

http://support.microsoft.com/kb/268674

When I tried to start the DHCP Client Service from the services  I ended up with a pop  "Could not start the DCHP Client service on Local Computer.
Error 5: Access is denied. " (see the screenshot attached below) even though I was logged on with the Administrators account then I followed this article  http://support.microsoft.com/kb/895149 which mentions

This problem occurs because the Network Service account does not have sufficient permissions to access the following registry subkeys when you upgrade to Windows Server 2003:

Desktop WallPaper Group Policy setting is not applied on Windows 7 and Windows 2008 R2 Machines