HMC4.5, Deployment Halted

I have started deploying HMC4.5 all over again for performance and scalability testing. I hit a new error message fairly early on in the deployment this time from the Provisioning Deployment Tool. The error appearer at the point I had completed all the settings for the Deploy Core MPS Components section and clicked Start Deployment. After a few seconds the following error message appeared -

Deployment Halted

Deployment interrupted for Mpf Audit And Recovery/MPS01 because SQL Server MPSSQL is either offline or non-existent

I attempted all the normal troubleshooting of ensuring IP connectivity, browsing, DNS records were all okay. Rebooted both servers, checked Event Logs and searched for *.log files, all to not result. After changing the following, it seemed to resolve the situation. I do not know enough about SQL to know if this is the fix, or if it was just some sort of coincidental timing...
On the SQL Database server, open the SQL Server Configuration Manager. Expand SQL Server 2005 Network Configuration > Protocols for MSSQLSERVER. Right click TCP\IP and select Enable. Restart the relevant SQL services. After completing this the Deployment finally continued without the above error.

(NB. I have changed the server names in the errors and post above)

Are you capacity bound or performance bound?

Calculating Hosted Exchange database disks

I have read a fair few articles about calculating your disk storage requirements for the Exchange Mailbox role. The first time I ever read the question, “Are you performance bound or capacity bound?”, I translated the question rather badly in my head and answered, well I want both capacity and performance. Having worked through the calculations of how many disks are required for capacity and how many disks are required for storage, I realised what the question meant. Satisfying which causes you to use more disks?

Resources

http://technet.microsoft.com/en-us/library/bb738147.aspx
http://technet.microsoft.com/en-us/library/cc671168.aspx
http://en.wikipedia.org/wiki/Gigabyte

Summary
I recently had to calculate how many disks are required to meet both performance and capacity criteria for the Exchange databases on a Hosted Exchange solution. These calculations show which RAID type should be used to achieve both criterion with the least amount of disks. These calculations are irrelevant of the Storage Architecture and are applicable for both SAN and DAS technology and only shows Exchange database calculations (not Transaction Logs). All calculations are of course baseline predictions, with many assumptions and therefore cannot be 100% guaranteed. To protect my companies internal design I have changed all numbers with regards to number of mailboxes, quotas, Send\Receive profile, etc to produce different numbers. The theory is still the same though.

Assumptions

In this environment it has been calculated that there is 15,000 mailboxes per MBX server. All users are classified as “Light Users” and send\receive 25 emails a day. Mailbox servers have been calculated with the maximum 8 cores and 32GB memory.

Performance Calculations

Database Cache
Database cache = (MBX Server memory - 2GB) / Total users per MBX server
Database cache= (32GB – 2GB) / 15,000
Database cache=2.048MB per user

Database Reads per user
Multiply the 25 messages per day by 0.0048, which results in 0.12. Next, take the amount of database cache per mailbox (2.048 MB) to the -0.65th power (2.048 ^ -0.65), which results in 0.6275. Finally, multiply the two figures, which results in database reads per user (0.12 × 0.6275 = 0.0753).

Database Writes per user
Multiply the number of messages per user (25) by 0.00152, which results in 0.038 database writes per user.

Database I/O (Front End)
Total database IOPS per user = ((0.0048 × M) × (D ^ -0.65)) + (0.00152 × M)
Total database IOPS per user= 0.0753 + 0.038 = 0.1133
Total read IOPS per MBX server = 0.0753 x 15,000 = 1129.5
Total write IOPS per MBX server = 0.1133 x 15,000 = 1699.5

Database I/O (Back End)
RAID 10 = Write x 2 + Read
RAID 10 = (1699.5 x 2) + 1129 = 4528 sustained IOPS
RAID5 = Write x 4 + Read
RAID5 = (1699.5 x 4) + 1129 = 7927 sustained IOPS

Disks required
Assuming an average 15,000rpm disk can sustain an average of 180 IOPS and a 10,000rpm disk can sustain an average of 140 IOPS, the following calculation shows the amount of disks required to cope with the Exchange database performance.
RAID 10, 15K Disks = 26 disks
RAID 10, 10K Disks = 33 disks
RAID5, 15K Disks = 45 disks
RAID5, 10K Disks = 57 disks

Capacity Calculations
Database capacity = Mailbox Capacity + Database whitespace + Dumpster
Mailbox Capacity = Total Users x Mailbox quota x OverSubscription ratio (see previous post about Oversubscription)
Quota = 1GB
OverSubscription ratio = 20%
Total Users = 15,000
Mailbox Capacity = (15000 x 1 x 20%) = 3000GB

Database Whitespace = Total amount of users x Average amount of mail sent per day x Average message size
Database Whitespace = 15,000 x25 x50KB = 17.9GB

Database Dumpster = Email retention period (days) x Average amount of mail sent per day x Average message size
Dumpster = 14 x 15,000 x 25 x50KB = 251GB

Database capacity = 3000GB + 17.0GB + 251GB = 3268GB (/200 = minimum 17 Storage Groups required)

Database Capacity disks required
RAID10 Capacity = (Amount of disks x Capacity of disk) \2
RAID5 Capacity = Capacity of disk x (Amount of disks -1)
400GB disk is actually 372GB, 300GB disk is actually 278GB
RAID10, 300GB Disks = 24 Disks
RAID10, 400GB Disks = 18 Disks
RAID5, 300GB Disks = 13 Disks
RAID5, 400GB Disks = 10 Disks

Conclusion
These calculations show that this Hosted Exchange solution is more performance bound than it is capacity bound. It shows that the RAID type should be RAID10 with 15,000rpm disks. To meet performance it is advisable to design the solution with 26, 300GB, 15K disks. Interestingly, the actual calculations I used in my performance showed that the RAID5 would have been the preferable solution.

Want to really remotely manage Windows 2008, Server Core?

Following my last post - I wanted to track down when the Administrator password for my domain was changed. Realising that Windows 2008, Server Core would not have a GUI for the Event Log, I assumed the easiest way to access the Security event logs of my Domain Controller would be to use another Windows 2008 server (Standard), open Event Viewer and remotely connect to my Windows 2008, Server Core Domain Controller. However, I was presented with the error message;

"Event Viewer cannot connect to computer DC01. The error reported is: The RPC server is unavailable"

My first thoughts were whether this was related to another Server Core feature or whether my POC network had issues due to being neglected for so long after being moved to a different ESX host. It turns out that the Windows Firewall in Windows 2008 Server Core is not configured to allow remote management. The following commands needed to be run on all my Windows 2008, Server Core Domain Controllers;

Netsh advfirewall firewall set rule group=”Windows Firewall Remote Management” new enable =yes

Netsh advfirewall firewall set rule group=”remote administration” new enable=yes

Some Links
Using the Netsh Advfirewall Command-Line Tool

Netsh Commands for Windows Firewall with Advanced Security

There are other Firewall Rule Groups available (taken from Technet)

Event Viewer = "Remote Event Log Management"
Services = "Remote Services Management"
Shared Folders = "File and Printer Sharing"
Task Scheduler = "Remote Scheduled Tasks Management"
Reliability and Performance = "Performance Logs and Alerts" and "File and Printer Sharing"
Disk Management = "Remote Volume Management"
Windows Firewall with Advanced Security = "Windows Firewall Remote Management"

To allow only specific MMC snap-ins to connect, at a command prompt, type:

Netsh advfirewall firewall set rule group=“%Rule Group%” new enable=yes

Q: How secure is your Windows 2008 Active Directory?

A: Only as secure as the physical access to any one of your Domain Controllers. If I have physical access to any of your Domain Controllers I will have administrative access to your domain within 10 minutes.

After several weeks of being away from my Hosted Exchange project (it got extended again!) I was back to work on it this week. Whilst I was away I was asked if the Development team could move my Proof of Concept environment to a different ESX host. Of course I had no issue with this as it was not being used whilst I was away. When I attempted to log on today I found that the administrative password I had set was no longer working. I always set the same password for all my development passwords (there is no real requirement for them to be secure) so I know I had not just forgotten it. The team that moved my VMs had no knowledge of the change of password. With it being a POC environment, I had no reason to create other administrative users in the Domain either.

I now have a scenario where I have a dozen servers configured with HMC4.5 and Exchange 2007 that I cannot access. It would take a week or so to reproduce this environment again, so I set about trying to hack the Domain Administrator password. ERD Commander and LockSmith allows you to reset the password on most new Windows Operating Systems. I have in the past tried this on a Windows 2003 Domain Controller to see what would happen and it did allow me to change the Domain's Administrator Active Directory password! As my first port of call I tried this boot disk and tool to see if it would work with a Windows 2008, Server Core Domain Controller. And guess what? It does! This has saved me a lot of work.

I guess this is not really new news, but it is relevant that it also works with Windows 2008 Domain Controllers. So - better make sure that Server Room door is always locked - eh?

Orphaned Exchange Store recovery

Guest blog entry by Rick Eveleigh (rick_eveleigh at hotmail dot com)

Got an intact Exchange Store but the Active Directory has gone? Save yourself thousands by following this procedure. No need for EDB recovery tools, Quest, OnTrack PowerControls, etc

Note: this is not a mail migration method: there are better textbook ways of doing that.
Note: this process is not warranted but comments are welcome.

This is Exchange 2003 but may well work on 2000 or 2007. The instructions cover all steps but are not fully detailed as they assume you know the basics of Exchange administration.

Install Exchange 2003 into any domain but with:
1. same server name as the orphaned store
2. same Org name as the orphaned store
3. same Admin Group name as the orphaned store
(You might only find out what the last two should be if installed wrong: the correct ones are in the Application Event Log when the stores fail to mount. In this case you will have to uninstall Exchange and reinstall with the correct Org & Admin Group names)

Make empty Stores in the Exchange System Manager (ESM)
Move to desired location
Mark as 'can be overwritten by a restore'
Dismount

In Windows Explorer, rename the new store names as .bak
Copy the recovered stores into the newly created file location
Rename the recovered stores to match newly created stores
Cross fingers and mount in ESM.

If it doesn’t mount you might have the wrong names as above.

Once mounted:
Expand the store
Right click Mailboxes
Choose Run Cleanup Agent. All will be marked as disconnected.

In Tools, choose Mailbox Recovery Center
Right click Mailbox Recovery Center and click Add Store
Type the first word of the relevant Store (e.g. Staff) and click Check Names
Store should be resolved, click OK
If there are a lot of mailboxes the 'resolve with AD' will take a minute or two
All the mailboxes will be listed
Click Mailbox to sort by Mailbox name (very important!)
Hit CTRL+A to select all Mailboxes
Right click (might be a delay if there are a lot of mailboxes) and choose Export
In the wizard, click Next, Next
Click Browse and find an OU for the users to be made in e.g. 'Recovered Staff Mailbox accounts' (you might need to make this first in AD Users and Computers)
Click Next
Type a name for the exported file (e.g. d:\temp\staff.ldf), choose Replace existing content and click Next
When the export has been completed click Finish

Run a command prompt
Type ldifde -i -f and hit enter

It is very likely there will be an error: all you get is a line number e.g. 'Add error on line 1481: Already exists', if this happens:
Open Active Directory Users and Computers
Browse to the new OU
Open your file (e.g. staff.ldf) in Notepad: for convenience arrange the notepad window alongside the ADUC window
See which account was the last to be created (this is why the list was alpha sorted before export).
Very likely the next account in the ldf file is the problem (might already exist e.g. Administrator, might be a duplicate to the last imported account, might just exist elsewhere in AD, or might be an 'invalid' account e.g. SMTP).
Delete all records from the ldf file up to and including the 'problem' account
Save (don’t close!) the ldf file.
Leave AD Users and Computers and Notepad open and run the ldifde command again.
Repeat until the result of the ldifde command is "The command has completed successfully"

In ESM, Mailbox Recovery Center, ensure all the mailboxes are still selected, right click and choose Find Match.
In the wizard, click Next. The Task will progress and the User Name column in the Mailbox list will populate.
Click Finish when the wizard completes.
Click the User Name heading to bring unmatched mailboxes to the top of the list
Press CTRL+A to select all mailboxes, then holding CTRL, click the mailboxes to deselect them.
Right click on the selected mailboxes and choose Reconnect
In the Reconnect Wizard click Next, Next.
The mailboxes will be reconnected to the newly created AD accounts
When the wizard completes, click Finish

Right click Mailbox Recovery Center and choose Remove Store
Type the first word of the relevant Store (e.g. Staff) and click Check Names
Store should be resolved, click OK.

If you are doing this on a temporary or test server you can now run exmerge to extract the mail. Or your Exchange Stores are now recovered.

Repeat for any other stores.

You can contact Rick via rick_eveleigh at hotmail dot com

OT: Recent lack of blogging

I thought I would drop in this quick Off Topic post after receiving a few emails recently asking about my lack of blogging recently. My Hosted Exchange project was put on temporary hold after I was asked to help out in another area of business for a few weeks. This few weeks away was immediately followed by an incredible few weeks holiday in northern Vietnam.

Hopefully I will be getting back into the detailed design phase of designing a hosted Exchange platform over the next few weeks and the blogs should start rolling in again.

Many thanks to those that take time to comment on the blog or to drop me an email. For me, that is what makes blogging worthwhile. (As well as reminding myself what I learned a few weeks ago :o))

Oversubscription...or is it contention or Thin Provisioning?

I have not blogged in a wee while as I have been working on financials and cost planning for implementing the hosted Exchange solution I am working on, rather than technical architecture.... a bit of "Excel hell".

Normally I blog with unusual or interesting findings, normally with solutions to problems. This post is a bit different as it is a bit more conceptual.

As part of my design for hosted Exchange I obviously need to design the mailbox storage, and as part of that design the capacity planning. It seems the whole world and it's dog are giving away huge mailboxes by default. Exchange Labs has 10GB, GMail has some sort of increasing figure coming up 10GB, Hotmail has 5GB and Yahoo has unlimited storage!

So the problem for a SaaS provider is, how do you cost for this? You can guarantee that none of the big vendors actually have 10GB of disk space for every one of their millions of users sitting in their data centre, just in case. The fact is that if every user has a 10GB quota on their mailbox a very minute percentage will every get anywhere close to this.

What you need to do is calculate a ratio of how much space is actually required vs the total quota limit. There seem to be a few different names for this. A few of my ISP colleague continually refer to this as the contention ratio. However after many hours Google'ing the science (or lack of) contention ratios I found that this is a bandwidth term, not a storage capacity term. It seems the correct term is an Over Subscription ratio. The other term that kept cropping up was Thin Provisioning, which is the practice of assigning less capacity that the total quota limit, but has some software fooling the hosted application into thinking it has the full available quota. Thin Provisioning @ Wikipedia

The next issue comes from the reason you want to allocate less storage capacity. In an internal deployment it is simple. The cost of the initial deployment is cheaper as you simply add storage as it is required. As a hosting provider, it is a little more complicated. You want to reduce the total storage required in order to reduce the cost of the solution altogether. Therefore you need to take a "bet" on how much storage is going to be needed based on your Over Subscription ratio, cost the cost of the solution per mailbox and therefore allocate a price. The main risk is if the Over Subscription ratio is overestimated, it is difficult to recoup the cost of extra storage costs once the price has been set.

One of the methods of determining an Over Subscription ratio is obviously to obtain statistics from our current dedicated Exchange deployments. The specific information I wanted to extract was the Total Mailbox Size and Last Logon Date (to determine mailboxes never or rarely used). The Exchange 2007 Powershell command I have used is as follows;

Get-MailboxStatistics -Database "Staff Database" | Select-Object Displayname,LastLogonTime, @{expression={$_.TotalItemsize.value.ToMB()};name="Mailbox Size"}| Export-Csv D:\StaffStats.csv

The most frustrating thing with the statistics I have obtained so far is that there is a very wide range. The first Exchange deployment I looked at has an average mailbox size of 30MB, the next had an average mailbox size of 200MB and the next about 1GB. Obviously the statistics should eventually show a Bell Curve.

Powershell v2, Vista and Proxy

I have recently downloaded Powershell v2 (CTP) and WinRM 2.0 (CTP) to attempt some Windows PowerShell (Exchange server) remote management. Having installed both components on my Windows Vista machine I attempted to create a runspace and connect to the remote host in powershell using the following synatax;

$rs = New-Runspace -Shell Microsoft.Exchange -ConnectionUri https://remoteserver.com/powershell/ -Credential $MyCred -Authentication Basic

When attempting the connection it would fail with this error message;

New-Runspace : [remoteserver.com] The client cannot connect to the remote host specified in the request. Verify that the service on the remote host is running and is accepting requests. You may use the following command to analyze the state of the WinRM service and to configure the service, if necessary: "winrm quickconfig".

I spent some (ok, quite a lot of) time trying to troubleshoot this. After confirming I could resolve remoteserver.com via DNS, I attempted to PING remoteserve.com. This failed,but of course this could be expected if ICMP is being blocked (I do not own the remote server or firewalls). I then downloaded and used Portqry.exe to see if I could connect to port 443 on myserver.com which failed also.

Now this got me thinking. My Windows Vista workstation uses a proxy server here in my company to access the internet. I wonder if this Powershell runspace is attempting to connect to the internet directly. Remembering the heartache caused by Exchange 2007 services attempting to do this, I attempted to find Proxycfg on my Vista machine. Of course I could not find Proxycfg as it has been dropped in Vista. It has been replaced by extended Netsh functionality. After some Google'ing I have found the correct syntax for Netsh, as follows from a command line;

>netsh
>winhttp
>show proxy

I can now see that there are no winHTTP proxy settings

>set proxy myproxyserver.internal:8080
>show proxy

I have now set the winHTTP proxy server, achieving the same as running Proxycfg -u on previous versions of windows.

My Powershell remote runspace now works. (Interestingly Portyqry still shows the remote server on 443 as FILTERED, but hey - it set me on the right track!)

HMC 4.5, DomainCacheTask Scheduled Task

I have found several more small configuration issues with the deployment instructions for HMC 4.5. Again, after finding no relevant resources on the Internet about this small issue I thought I would blog it here to hopefully help someone else.

When attempting to create the SMTP Domain Cache scheduled task the instructions give you the following syntax to run against the Schtasks command in Windows Server 2008.

schtasks /create /S localhost /U %USER% /P %PASSWORD% /SC MINUTE /MO %MINUTES% /TN SmtpDomainCacheTask /TR "\"C:\Program Files\Microsoft Hosting\Provisioning\SmtpDomainCacheTask\SmtpDomainCacheTask.exe\""

When I run this command (replacing %USER%, %PASSWORD% and %MINUTES%, I received the following error message; ERROR: User credentials are not allowed on the local machine.. Of course Google is my first stop for all unknown error messages, and again I found no documentation referring to this error.

To be entirely honest, as Windows Server 2008 is still new to me I had to go and research the Schtasks syntax and switch options. From what I understand the /U and /P switches are more appropriate for when you are scheduling a task on a remote machine and are therefore passing credentials to allow you to create the scheduled task. If I understand the Note: The user account must have write permission to the directory of CategorizerOverrideAgent.dll, and have read permission to MPS PlanManager database. I believe what is actually required is the user context in which you want to run the task. Therefore the switches required are /RU and /RP.

schtasks /create /S localhost /RU %USER% /RP %PASSWORD% /SC MINUTE /MO %MINUTES% /TN SmtpDomainCacheTask /TR "\"C:\Program Files\Microsoft Hosting\Provisioning\SmtpDomainCacheTask\SmtpDomainCacheTask.exe\""

This command now schedules correctly for me.

HMC 4.5, Install and Configure the OOF Agent

Whilst installing HMC 4.5 I have now reached the Install and Configure the OOF Agent section. There are some minor details that upset my installation. After some searching of the internet I have realised there is literally no resource or documentation on the web about these components, so I have decided to blog todays findings. (Try entering CategorizerOverrideAgent into Google - I only got one website returned; Technet)

In the section when you install the CategorizerOverrideAgent.msi from the Service Provisioning\MPS\Install folder the instructions state that this should be installed in the C:\Program Files\Microsoft\Exchange Server\TransportRoles\Agents\CategorizerOverrideAgent directory. By default my installation always reverted back to Program Files (x86) not the Program Files folder. I tried to change this several times with no success. It turns out this is not an issue though.

During the steps of installing the categorizer agent via the Exchange Management Shell the steps are not quite in the correct order and some syntax is missing. Rather than just running the switch -AssemblyPath CategorizerOverrideAgent.dll you need to pass it the full path to the dll as the Exchange Management Shell does not know the location of CategorizerOverrideAgent.dll.

In my environment I was not able to run the Enable-TransportAgent CategorizerOverrideAgent cmdlet whilst the MSExchangeTransport service was stopped (as the instructions suggested). I had to start the MSExchangeTransport service, enable the CategorizerOverrideAgent and then restart the service.

Running Get-TransportAgent at the end showed that despite all these annoyances the Categorizer Agent is now installed and enabled on my Hub Transport server.

HMC 4.5, Exchange Resource Manager

I am working may way through deploying Hosted Messaging and Collaboration 4.5 (HMC 4.5). I have reached the point (with a lot of frustration and time consuming tweaks) of configuring Exchange 2007 SP1 Resource Management on the Microsoft Provisioning Engine server. Now if you follow the link here http://technet.microsoft.com/en-us/library/cc501402.aspx you would think this was a fairly straight forward task? Wrong...
So I changed the required values and entered my mailbox, my public folder mailbox and my Domain Controllers FQDN. When I ran the provtest command I received the following error;

errorContext description="Mail server not found"

After trying to troubleshoot this for quite some time it turns out that I made that mistake of using a FQDN. It seems that unless these values are NETBIOS names, the script will fail. Seems ridiculous to me. Hopefully this post might save someone else the time and frustation I spent on this daft issue.

VMWare Infrastructure Client, mounting ISOs

I am working quite a bit with VMWare Infrastructure Client 2.5 to connect through to my ESX enviornment. I thought I would make a quick post about the frustation of mounting (some) Iso images. I have the following error message when attempting to some mount ISOs across the network for their virtual guests; "Please specify a valid image"

Now that is a very descriptive error message, with not a lot of help, eh? I know the image is fine as I am able to access it via WinIso and I am able to burn DVDs from it. The answer is one or both of the following problems;

1. The file extension was .ISO rather than .iso. Apparently VMWare Infrastructure Client does not like the ISO file extension to be in upper case, so it needed to be renamed to .iso

2. The file name of the ISO was too long. By default the ISO images you download from Microsoft Technet are in a very long format (6001.18000.080118-1840_amd64fre_Server_en-us-KRMSXFRE_EN_DVD.iso) and the VMWare Infrastructure Client does not like this length, rename it something smaller.

After performing one or both of the adjustments above, the ISO image mounts with no issues.

Proxycfg.exe

I originally came across Proxycfg.exe after installing one of the Exchange 2007 Update Rollup packs. Some of the Exchange 2007 services would not start up, in particular the Microsoft Exchange Service Host service.

I have now came across the exact same issue with Microsoft SQL Server 2005 and the SQL Server Integration Services service also.

This problem occurs because the server cannot reach the following Microsoft Web site: http://crl.microsoft.com/pki/crl/products/CSPCA.crl .For some reason these services do not know how to access the Internet if the server is configured to use a Proxy server.
There are various solutions for each individual service that cannot start that involve installing (many) updates to stop these services attempting to reach this site. Alternatively the easiest way to solve this is configure the server services to use the logged on users proxy configuration. To do this open a CMD prompt and change the directory to C:\Windows\System32 and run the command proxycfg.exe -u. In my case the services affected by this problem then all started immediately.

Windows 2008 Active Directory, missing tool

I have just discovered that my favourite Active Directory troubleshooting tool Replmon.exe has not made it to Windows 2008! (replmon technet link)
Here is a quote from a Technet blog;

"Unfortunately, replmon did not survive the transition to Win2008. It was actually developed by MS support, not the product group (along with many other support tools/resource kit tools), and without an actual owner to service the tool years later, it was a casualty."

The Windows 2003 version of Replmon appears to work okay though. You will need to install the Windows 2003 Support Tools (Suptools.msi)ignoring the Program Compatibility Assistant warning This program has known compatibility issues.

Let's just hope those compatibility issues do not cause instability issues on my server then!....

Windows 2008 Active Directory, deleting OUs

Today I found some new default permissions on Windows 2008 Active Directory Organizational Units (OU). I had created an OU in my nice new Windows 2008 Active Directory to provision servers into. Now that I have created my OU structure I tried to delete my redundant OU and received the error message –

You do not have sufficient privileges to delete MyOUName, or this object is protected from accidental deletion.

So I immediately switched on Advanced Features in Active Directory Users and Computers so that I can access the Security tab of the OU. When I clicked Advanced there was one explicit Deny permission set for Everyone with Special permissions. These Special permissions were Deny Delete and Deny Delete Subtree. Of course by un-checking these options I could delete the OU.

I think this is an awesome subtle improvement. I have actually worked for a company were a user with Administrative permissions accidently deleted an enormous OU with thousands of users, computers, printers and customisations. By denying delete permissions to Everyone by default is means that you can no longer accidently delete an OU. You need to be fully aware of what you are doing to have to go and remove this permission each time. Kudos MS.

Microsoft Provisioning Services, Locales

I am currently setting up the Microsoft Provisioning Services(MPS)components as part of the HMC4.5 installation for my Hosted Exchange POC. When attempting to assign some of the core MPS components to a server I received the following error message - The server xxxxx either does not exist, is offline or fails one or more prerequisite checks. Do you want to assign the server anyway? Now is that a cover-all error message or what!? When I click on details I found a much more explanatory reason, but a very frustrating one too

Exception: Microsoft.Provisioning.DeploymentTool.Engine.ServerPrerequisiteException
Message: Default server locale is 2057, should be one of: 1033

It turns out that Microsoft have only tested the MPS on the English (United States) locale only and therefore that is the only locale that I can install MPS onto! So after ensuring I had installed every locale configuration as English (United Kingdom) for my MPS servers, I now have to change them all back. Joy.

Side note - when you have changed all the locales on your servers to English (United States) you will need to close the Provisioning Deployment Tool and open it again. For some reason it still seemed to think the servers had the wrong default until the application was restarted.

Active Directory “List Object Mode”

Active Directory normally has three visible READ permissions; List Contents, Read All Properties and Read Permissions. These permissions cover the majority of Active Directories READing related permissions. There is however a fourth READ permission not enabled by default; List Object.

The List Contents permission would normally list all immediate child objects. With the List Object permission enabled Active Directory has the ability to hide objects returned by the List Contents function.

Why is the useful?

In the shared Active Directory configuration of a multi-tenancy hosting solution, different organizations share the same domain. In this shared hosting environment, it is important to ensure that only authorized users can access the information and configuration settings for a given organization.

To set Active Directory to List Object mode open ADSIEdit.msc. Expand the Configuration container, CN=Services , CN=Windows NT. Right-click Directory Service, and click Properties. Change the dsHeuristics attribute to 001.

Windows 2008 Hibernation

Why on earth would anyone want the ability to hibernate their server? It seems that Windows 2008 server (Standard, Enterprise and Core) all have hibernation enabled by default. I found this strange setting out when trying to figure out where all my system volume disk space was being utilised and came across Hiberfil.sys in the root of the system volume.

Of course this file is a hidden system file, so the easiest way to see it is to open a CMD prompt and enter Dir C:\ /A:SH

Now when I say Hibernation is enabled – it is not enabled in any of my Power Options in the control panel. If I select the currently selected power plan and navigate to the options to change the advanced settings the Hibernate After setting is disabled (as is Sleep after). It seems the only way to safely get rid of this large file (assuming you do not want to hibernate your servers?!) is to run the following command

Powercfg /Hibernate off

Windows 2008 Activation (KMS)

This morning I went to work with my new POC environment which contains 8 Windows Server 2008 machines installed using my company Volume License Key. They all had a warning message:

Windows could not be activated.
Key management services (KMS) could not be located in Domain name system (DNS), please have your system administrator verify that a KMS is published correctly in DNS.

I think these kinds of error messages from MS are amusing. Who is this higher entity known as “System Administrator”? He seems to be the oracle of all knowledge according to all my Operating System errors! Oh yeah, that is supposed to be me? Well, I have not configured anything to do with KMS in my DNS just yet. I am aware that the licensing and activation of Windows Server 2008 is different. Guess I better RTFM...

Okay, after a bit of RTFM it seems that I use Volume Licensing Keys in my company, I need to become familiar with MS Volume Activation 2.0.
http://technet.microsoft.com/en-us/library/bb892849(TechNet.10).aspx .
Of the two methods, Key Management Service (KMS) or Multiple Activation Key (MAK), it seems KMS is more relevant to my requirements in a POC environment. KMS allows me to activate servers within my own network, whilst MAK activates servers MS’s hosted activation services.

To install KMS on one of my Windows Server 2008 machines

From the command prompt change the directory to C:\Windows\System32
Type slmgr.vbs /ipk %Insert your VLK% (i.e. slmgr.vbs /ipk ABCDE-12345- ABCDE-12345- ABCDE)
After a few minutes I get a message stating “Installed product key %My VLK% successfully"
Type slmgr.vbs /ato
After a few minutes I get a message stating “Product activated successfully”
Restart the Software Licensing service
Create a DNS SRV record in DNS
Open DNS Manager
Right-click the domain, and then click Other New Records.
Click Service Location (SRV), and then click
Type the following information:
Service: _VLMCS
Protocol: _TCP
Port number: 1688
Host offering the service:
Click and then click

Client side

In theory if all your other Windows 2008 servers requiring activation are domain members and can resolve the SRV record for KMS, they should self activate in the next 180 minutes. If you are like me, and you are impatient, you could use the switch slmgr.vbs /ato on each of the client machines. For the first five you will receive an error message. This is because KMS is for VOLUME licensing and is therefore not applicable until at least 5 machines attempt to activate. Once you have reached your fifth machine, all machines that have attempted to activate will then be activated.

Notes:

Slmgr.vbs /dli Displays all standard licensing information
Slmgr.vbs /dlv Displays verbose licensing information
Slmgr.vbs /xpr Displays the expiration date of the current key

KMS clients must renew their activation by connecting to the KMS host at least once every 180 days to stay activated. By default, KMS client computers attempt to renew their activation every 7 days

KMS has a minimum amount of physical (not virtual) computers that require activation before it works; this is called the activation threshold. For Windows 2008 it is 5. Below that number, activation will not occur.


There is obviously a wealth more information on KMS and MAK, but as this is all I require to carry on with my now licensed POC, that’s all from me, for now.

Windows 2008 (Standard) strange disk recommendation

Windows 2008 Server (Standard) has a system volume disk space recommendation of 12745Mb.

This value is not the same as any online documentation I have read. I disovered this obscure value as I was setting up some servers for my POC lab and setting the system volume as 12Gb (12288Mb) by default. Windows give me some friendly advice that it recommends it should be at least 12754Mb, it did let me carry on and install it.

Interestingly Microsoft's online documentation actually recommends 40Gb or more disk space, with a minimum requirement of 10Gb. What are they going to do with 40Gb of disk space? (Apart from make the Server Core footprint look comparatively small?).
Microsoft Windows 2008 requirements link

Windows 2008, Server Core - DCPromo

To make a Windows 2008, Server Core a Domain Controller you need to run DCPromo and link it to an unattend.txt file. I am sure it is probably possible to accomplish this in one command line, but that would be a lot of switches to write!
One of the items in this unattend.txt file is the Safe Mode admin password. I thought this was a little dodgy as a large amount of folk will accidently leave that unattend.txt file on the computer and therefore accidently leave the password accessible in plain text. But it seems MS were thinking the same thing as I discovered when I copied contents to use on another server and realised the password was gone. DCPromo actually removes the password value in the text file as soon as it is run.
Here is a copy of my unattend.txt for my new Forest –

[DCINSTALL]
InstallDNS=Yes
NewDomain=Forest
NewDomainDNSName=BGibson.Internal
DomainNetBiosName=BGibson
ReplicaOrNewDomain=Domain
ForestLevel=3
DomainLevel=3
DatabasePath="D:\NTDS"
LogPath="E:\Logs"
SYSVOLPath="F:\Sysvol"
RebootOnCompletion=Yes
SafeModeAdminPassword=P@ssword1


Funny errors

To create my first unattend.txt file I plagiarised someone else’s file from the internet. It originally has RebootOnSuccess as one of the switches. Windows 2008 very helpfully give me some advice “Warning: RebootOnSuccess is deprecated, although it is still supported. Consider using RebootOnCompletion instead.” I thought this was quite amusing suggesting the syntax I was using was deprecated.

Not so funny error

Now that I have one Domain Controller up and running, I went to my next Windows 2008, Server Core machine and tried to promote it to be a replica Domain Controller. Unfortunately I immediately received this error message “The wizard cannot access the list of domains in the forest. The error is:
Access is denied.”

After doing some research I became convinced it had to be DNS related. Back to the first Domain Controller and type NSLookup. Immediately got an error as it was trying to connect to ::1 first. Using similar syntax to my last post for adding DNS Servers, I removed this IPV6 reference: netsh int ipv6 delete dnsserver name=2 address=::1. Now when I run NSLookup it connects to its own DNS server as expected. DCPromo of the replica Domain Controller now completes successfully. Here is the unattend.txt I used;

[DCInstall]
InstallDNS=Yes
RebootOnCompletion=Yes
ReplicaDomainDNSName=HostedExchange.Internal
ReplicaOrNewDomain=replica
ReplicationSourceDC=HE-DC01.HostedExchange.Internal
SafeModeAdminPassword=**********
UserDomain=HostedExchange.Internal
UserName=administrator
Password=************
CreateDNSDelegation=No

Windows 2008, Server Core - First Installation

Windows 2008 Server Core

Today I performed my first installation of Windows 2008 Server Core....into a VMWare environment. As this is completely new to me I will probably blog about a lot simple RTFM tasks to start with as I get used to going back to the command line.

Speed

First thing I noticed was how incredibly quick it was. The Operating System installation took a matter of minutes, not the normal hour or so I have come to expect with Windows Server 2003. I guess this is down to the fact the Core platform installs only the necessary components only and not all the normal bloatware that comes with Operating Systems to date. In fact I was so impressed with the time I deleted my VMWare image and started again so I could time how long it took. (Sad I know). It took only 6 minutes to complete the installation.
So after being impressed with t he speed of installation, when I log on it takes 3 minutes “Preparing your desktop”. WHAT DESKTOP!? Surely the whole point of Server Core is there isn’t one, just the lonesome CMD.exe?

VMWare Tools

Okay, first thing you want to do once you have installed an OS into VMWare is install VMWare Tools. Challenge number 1 – Server Core does not have a GUI, so therefore the nice GUI installation of , , does not exist. So, I click VM > Install VMWare Tools, which of course appears to do nothing – except it has loaded the VMWare Tools ISO...Back to the command line and change my directory to D:\ and type MSIExec /i “VMWare Tools.msi” /passive. Hey presto, I have the VMWare Tools installation wizard.
Of course the install is not going to be that easy, I immediately receive two errors; Error loading tpvmmon.dll & Error loading printui.dll, The specified module could not be found. I click to both as this is my only options. After a reboot it seems that VMWare Tools are more or less installed, despite both the errors. However, I also installed VMWare Tools on my ESX environment, this produced no errors and installed seamlessly.

Display settings and Power Management

By default the screen resolution is not great (does it need to be for a CMD line only interface), but more annoying is the fact it has the automatic screensaver/power management turned on so it forces you to log on again after a period of inactivity. After a bit of RTFM, I found you have to change this in all in Regedit.

ScreenSaver settings:
HKCU\Control Panel\Desktop\

Resolution:
HKLM\System\CurrentControlSet\Control\Video\GUID\0000\DefaultSettings.XResolution
HKLM\System\CurrentControlSet\Control\Video\GUID\0000\DefaultSettings.YResolution

To determine which GUID it is, look in each of them at the Device Description. For me it was the GUID associated with VMWare SVGA II. Make sure you remember to change the Base to decimal too!

Simple Tasks

Next issue was how do achieve the things you take for granted with a GUI?

Reboot the server: Shutdown –r (Use shutdown /? to get all the other related tasks)
Set the IP Address:
· Get the interface IDX Number:
netsh int ipv4 show int
· Set IP Address: netsh int ipv4 set address name=%IDX% source=static address=%IP% mask=%SM% gateway=%DG%
· Set DNS Server: netsh int ipv4 add dnsserver name=%IDX% address=%DNS%