KB2509007 Strikes ConfigMgr 2012!

An issue that plagued many admins over the last couple of years is that download (and thus installation) when a task sequence installs many Software Updates would hang and/or be stuck on the first update (KB2509007).  This was a problem relegated to CM07.  However, I am experiencing the same ‘ol thing in my home lab environment.  One resolution discussed for CM12 has been to 1) ensure the DP is set to allow anonymous connections, and 2) set the client install parameter SMSMP=FQDN.

However, these did not work for me in my lab and since no hotfix is available at this time, I’ve taken the quick workaround approach for my image build by using a VBScript to automate the installation of updates directly using the Windows Update Agent. While I cannot take credit for the below script, I did modify it to eliminate the prompt messages and automatically “accept” installation of updates.  I just run this script as part of my task sequence via Run Command Line and it does the trick.

http://msdn.microsoft.com/en-us/library/windows/desktop/aa387102(v=vs.85).aspx

Download revised script: http://sdrv.ms/QT0F0S

ConfigMgr 2012: “Multiple connections to a server” error 0x800704c3

I recently had this error (while troubleshooting account credentials) in the CM12 console when running “Test Connection”.  I was attempting to validate the account credentials using C$ of the site server, but it generated error 0x800704c3 for “Multiple connections to a server or shared resource by the same user, using more than one user name, are not allowed”.

The resolution?  Don’t use a hidden share!  Instead, I tested the connection with the default SMS_sitecode share and it worked.

Create Task Sequence Media: Directory name is invalid

I’ve recently rebuilt my home lab environment to run Windows Server 2012 with Hyper-V 3.0.  After installing the ConfigMgr 2012 console on it, I was attempting to create new task sequence media when I received an error that the directory name was invalid.  I tried different file locations to save the .iso, such as c:\, d:\ (an external USB device), my desktop, and a UNC path.  All failed.

Figuring that this was more a problem with “user account control”, I launched the console with elevated permissions and the error did not occur.  But to see if there was some other problem, I then launched the console normally and when creating the boot disc again, Windows Server 2012 then correctly prompted to allow for to allow execution.

It’s an odd problem, but if it occurs for you, first try just restarting the console!

Batch File to Schedule WinXP Restart

In a troubleshooting situation with Windows XP hanging during a restart of Windows after a CM07 package deployment, I built a workaround script that creates a scheduled task to perform a forced Windows restart (for 2 minutes after install completion).  The reason for doing 2 minutes is because there are times when the scheduled task can be created in the 59th second of a minute, but there isn’t enough time for the installation to exit.

@echo off 
Set /a MIN=%TIME:~3,2% + 2 
Set /a HOUR=%TIME:~0,2%
:: Remove the echo command to actually schedule the restart 
echo at %HOUR%:%TIME% /interactive C:\Windows\system32\shutdown.exe -r -t 60 -f

BitLocker Recovery Mode Guidance

Created this BitLocker recovery guidance information for a client recently.  Not overly technical, but I thought it would be useful to share.

What to Do When BitLocker Recovery Mode Occurs

If a computer should be unexpectedly prompt the user for a recovery key, use following high-level steps to correct the system and help prevent future unnecessary BitLocker security trips.

1. Obtain the recovery key from Active Directory and enter it into the prompt
   
2. Login to Windows
3. Suspend BitLocker protection. By suspending BitLocker, restarting Windows, and resuming BitLocker protection again, the TPM module will be resealed with the current PCR settings.  This will help prevent subsequent OS starts from prompting for BitLocker recovery mode.  Note that it is not necessary to decrypt and re-encrypt the drive
   
4. Shutdown computer
5. Boot into BIOS configuration and ensure that the hard disk is listed first in the boot order.  If not, make the appropriate change and the save the modifications
6. Restart and login to Windows
7. Resume BitLocker protection
8. Restart Windows to ensure the recovery mode was not tripped

Possible Causes of BitLocker Recovery

This is additional information as provided by Microsoft to help isolate causes of BitLocker recovery.  It should be noted that integrity check failures are not a result the Windows 7 image.  Recovery mode being initiated is a sign of BitLocker successfully detecting and interpreting a possible threat. The following types of system changes can cause an integrity check failure and prevent the TPM from releasing the BitLocker key to decrypt the protected operating system drive.

• Moving the BitLocker-protected drive into a new computer.
• Installing a new motherboard with a new TPM.
• Turning off, disabling, or clearing the TPM.
• Changing any boot configuration settings.
• Changing the BIOS, master boot record, boot sector, boot manager, option ROM, or other early boot components or boot configuration data.

This functionality is by design; BitLocker treats unauthorized modification of any of the early boot components as a potential attack and will place the system into recovery mode. Authorized administrators can update boot components without entering recovery mode by disabling BitLocker beforehand.

The following list provides examples of specific events that will cause BitLocker to enter recovery mode when attempting to start the operating system drive.  Microsoft’s full list has been reduced and categorized for ease of reading and potential scenarios that may occur.

Software changes

• When you are installing additional language packs onto the system, and selecting the option to apply the language settings to all users and system accounts. This causes a locale change in the BCD (Boot Configuration Database), which BitLocker with TPM interprets as a boot attack.
• Malicious software

BIOS changes

• Changing the BIOS boot order to boot another drive in advance of the hard drive.  (e.g. the hard drive needs to be first in the boot order)
• Having the CD or DVD drive before the hard drive in the BIOS boot order and then inserting or removing a CD or DVD.
• Upgrading critical early startup components, such as a BIOS upgrade, causing the BIOS measurements to change without first suspending BitLocker protection.

Hard drive changes

• Changes to the master boot record on the disk.
• Changes to the boot manager on the disk.
• Changes to the NTFS partition table on the disk including creating, deleting, or resizing a primary partition.
• Changing any boot configuration data (BCD) boot entry data type settings

Other hardware changes

• Docking or undocking a portable computer. In some instances (depending on the computer manufacturer and the BIOS), the docking condition of the portable computer is part of the system measurement and must be consistent to validate the system status and unlock BitLocker.
  • This means that if a portable computer is connected to its docking station when BitLocker is turned on, then it might also need to be connected to the docking station when it is unlocked.
  • Conversely, if a portable computer is not connected to its docking station when BitLocker is turned on, then it might need to be disconnected from the docking station when it is unlocked.
• Adding or removing hardware. For example, inserting a new card in the computer, including some PCMCIA wireless cards.
• Disabling the code integrity check or enabling test signing on Windows Boot Manager (Bootmgr).
• Adding or removing add-in cards (such as video or network cards), or upgrading firmware on add-in cards.

Additional Guidance for BitLocker Recovery Troubleshooting

PCR configurations

• A platform validation profile consists of a set of Platform Configuration Register (PCR) indices. Each PCR index is associated with components that run when Windows starts.
• Current enabled PCRs via GPO (all defaults): 0, 2, 4, 5, 8, 9, 10, 11
• Per a support forum thread: Dell Support recommended to shut off PCR 0 and 2 and test further. Please understand that this will further reduce security from the default configuration.
• PCR 0 (recommend to disable and test): Core Root of Trust of Measurement (CRTM), BIOS, and Platform Extensions
• PCR 2 (recommend to disable and test): Option ROM Code

Software changes

• Document the software installations that are being completed manually.
• Install software during encryption of the disk then restart Windows to determine if that caused a change.
• When you are installing additional language packs onto the system, and selecting the option to apply the language settings to all users and system accounts. This causes a locale change in the BCD (Boot Configuration Database), which BitLocker with TPM interprets as a boot attack.
• Note: if a PCR is tripped, it will be logged into system events, but not the specific PCR that changed.  Other events surrounding that registered change can be evaluated as a possible source.
• Note: Windows Updates have built-in logic to not trip BitLocker into Recovery Mode.

BIOS / TPM

• Ensure that the hard drive is first in the BIOS boot order. The reason for this is the boot device makes up part of the system measurement used by BitLocker and this must remain consistent to validate the system status and unlock BitLocker
• Update or modify BIOS to a newer version only after suspending BitLocker protection.  Remember to resume protection after the changes.

Malicious software

• Ensure AV is fully functional and up-to-date
• Perform a full AV scan of the hard drive
• Use additional 3rd party tools to perform a cross-scan for malware

BitLocker & BIOS Boot Order

One of the “gotchas” of BitLocker security is that by not having the hard drive first in the boot order within BIOS, can cause BitLocker security to become enacted and thus needing manual entry of the 48-character key upon the next system restart.  This can be a frustration for users who have this happen to them, especially while travelling and unable to reach the help desk.  So, during an OS deployment, make efforts to change the boot order in BIOS.

To do this with HP

• Obtain the BIOSConfigUtility in the Systems Software Manager
• Create a text file named “BootOrder.REPSET”.  The text file contains the below content.  Note that I found it is necessary to define two devices to modify the boot order.

English
Boot Order
     Hard Drive(C:)
     Notebook Upgrade Bay

• Run command

BiosConfigUtility.EXE /SetConfig:BootOrder.REPSET

To do this with Dell

• Obtain the Client Configuration Toolkit
• Run command

cctk.exe bootorder --sequence=hdd

If you find yourself in a position that you did not do this during the initial deployment of the OS, never fear, SCCM is here!  Using task sequences, you can automate the process as to set the hard drive to be first in the boot order and re-seal the TPM by performing the following steps:

1. Suspends BitLocker protection
   
2. Reconfigure the boot order (for HP or Dell)
    
3. Restarts Windows
   
4. Resumes BitLocker protection
   

Microsoft Live Meeting 2007 Conflict with Outlook 2010 Transport

Found an issue with LM 2007 on Outlook 2010 on a recent deployment.  The LM client (version from May 2011) is configured to use the Live Meeting Service from URL (such as https://www.livemeeting.com/cc/CompanyName).  If the user’s Outlook profile is NOT configured to use cached exchange mode, then a scheduled live meeting gets an undeliverable message.  If cached mode is off, a workaround is to change the “From:” field to use “Live Meeting Transport” instead of “Microsoft Exchange Server”.

Microsoft has confirmed this is a known issue, though I do not know if a KB has been publicly published.

Console connection error 0x800706BA

Ran into a situation with a remote CM12 console being unable to connect to a site server.  According to the local SMSAdminUI.log file:

Transport error; failed to connect, message: 
'The RPC server is unavailable. (Exception from HRESULT: 0x800706BA) 
'\r\n Microsoft.ConfigurationManagement.ManagementProvider.SmsConnectionException 
\r\n The RPC server is unavailable. 
(Exception from HRESULT: 0x800706BA)\r\n  at 
Microsoft.ConfigurationManagement.ManagementProvider.WqlQueryEngine.WqlQueryProcessor.ExecuteQuery
(String query, Int32 blockSize, Dictionary`2 contextInformation)

Initial troubleshooting  indicated that this was a problem with the Windows Firewall blocking remote administration.  However, the firewall was off one the site server.  Yet some posts indicated that the ports needed to be allowed, regardless of the firewall being on/off.  So I opened up those, but it still couldn’t connect.

I took a step back and went to the basics – could I ping the FQDN?  No, I couldn’t!  Yahtzee!  I could ping the short name just fine, so I just needed to have Windows map the IP to the FQDN.  So as a quick fix, I just added an entry into the C:\Windows\System32\drivers\etc\Hosts file – and voila, the remote CM12 console could now connect!

Recovering SCCM Site from a Failed Bad Backup

Helped a client recently migrate their existing SCCM environment to new hardware.  We ran into some challenges and thought it would be good to share how we were able to work through the problems to get SCCM successfully migrated.

Going from:

• Win2k3 Standard x86
• SQL 2005 x86SP2
• SCCM 2007 SP2 R2

Going to:

• Win2k8 R2 Datacenter (x64 of course)
• SQL 2005 x64SP4
• SCCM 2007 SP2 R2

At a high level, the operations to migrate a site are:

• Perform a backup, then shut everything down.
• Replace the hardware and ensure the configuration is identical – drives, names, paths, etc.
• Install the same software and pre-requisites
• Install Configuration Manager using the same settings and paths.
• Run the repair wizard from the shortcut on the menu and perform a restore. (not from the console)

However, the site repair wizard was failing on the first step of verifying the backup path. The GUI said that the SQL backup files and ConfigMgr inbox files are out of sync and the file stamps are different. Additionally, the RepairWizard.log file has several instances of “Initializer {GUID} will no be run, unsupported application type”.  Additionally, SMSbkup.log states “Backup task completed successfully with zero errors but there could be some warnings, AFTERBACKUP.BAT will be started if available in its predefined location”.  However, looking at the logs more closely above, I see line after line of errors.  Such as:

• Error: Failed to backup \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\SMS\BackupTemp\SMSbkSiteRegNAL.dat up to D:\SMSBackup\Backup\SiteServer:\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\SMS\BackupTemp\SMSbkSiteRegNAL.dat is not readable.
• Failed to copy file(s) Backup\SiteServer.
• Error: Failed to backup \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\SMS\BackupTemp\SMSbkSiteRegSMS.dat up to D:\SMSBackup\Backup\SiteServer:\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy2\SMS\BackupTemp\SMSbkSiteRegSMS.dat is not readable.
• Error: Backup Failed  for Component – Backup\SiteServer\SMSbkSiteRegSMS.dat.

So this showed that the site backup was truly not successful and was incomplete.  Upon comparing the contents of the directory against a known good backup in another environment, the backup was missing the following items

• SMS/NAL registry key backups (as a .dat)
• logs
• 1/3 of the inboxes, including the site control
• data folder
• srvacct folder

Essentially to workaround the problem and move forward, I

• Copied those folders straight from the old server’s installation directory (IMPORTANT: Do not recover the srvacct folder!! More info below)
• Detached the “new” site databases in SQL and attach the “old” databases
• Ran the site repair wizard EXCEPT selecting to not restore the database

Except for the restore of the SMS/NAL registry keys, the site restore seems to have worked at that point and the site is functioning (activity, inventory, SWD, reporting, etc.).  However, it was still critical to get the registry keys imported.  On the old site, I had exported those registry locations (HKLM\Software\Microsoft) and tried to just import them directly on the server (NOTE that since I was going from a 32-bit OS to a 64-bit OS, I had to a bulk search/replace to add Wow6432Node into the path).  The import action was blocked/prevented.

So, the recommendation was to boot the server into safe mode and then import the registry items.  So we did that but continued to get access denied problems with the SMS key.  So I started a process of elimination by cutting the registry file in half each time until finally we identified the problematic key that was causing the whole value to not import.  The guilty key?  The Certificates location – HKLM\SOFTWARE\Wow6432Node\Microsoft\SMS\MP\Certificates – which is logical that this would cause the entire import to fail.  Which is OK because installing a new MP will generate new cert keys.

Now the site is finally up and running.  Right?  Wrong!  When attempting to use run a task sequence, I received the issue as described in KB2509330 because of restoring the srvacct folder.  The “resolution” is to rebuild the entire server over again from scratch, which is not a good idea because of the effort to get this far.  Fortunately, I had a file system backup of the VM such that the original srvacct folder could be restored.  That backup saved me from having to start from the very beginning!  So, not overwriting the srvacct folder is critical.

Other items needing resolution after the migration

• Recreate any boot media afterwards with the new site certificates
• Reinstall an SMP to fix a cert mismatch
• Fixing client certs by running “ccmsetup.exe RESETKEYINFORMATION=TRUE”

SCCM certificates are like sand, they get into everything :-)  Anyhow, this was quite a process to go through.  MORAL OF THE STORY?  Make sure you have good and complete backup of your site before migrating to new hardware!

Troubleshooting ACT Log Processing

Ran into a problem where ACT was deployed to many computers, their data was successfully being created on the file share, but the computers were not showing up in the App Compatibility Manager.  What I discovered (as it’s been awhile since I last used ACT) was that there is a log processing service which imports the logs from the share into the database.

In troubleshooting the issue, I came across and excellent guide by Microsoft for troubleshooting the log processing service.  I went through every step and in nearing the end of the guide, the computers were still not imported.  Ironically, the very last step was my fix and it was the easiest step – check that the Windows service “Act Log Processing Service” is running.  Really?  That couldn’t have been listed as one of the first thing to check?