Observations from a Full-Scale Migration to Windows Azure, Part 1 (Highlights)


Over the past several years, we have been designing and developing our systems in preparation of getting them up “into the cloud”. Whether this means Microsoft, Amazon, or whomever was unimportant as the architecture needed to allow for high-availability and load-balanced deployments of our systems – the cloud-specific issues could be figured out later. About 1 1/2 years ago, we deployed some minor systems to Azure and consumed some of their services (most importantly queueing and blob storage). Over the past month and a half, we’ve been making changes specific to Azure. And last weekend, a co-worker of mine (who I can’t express enough gratitude towards) and I spent a grueling 72 hours beginning Friday morning migrating all of our databases and systems to Azure. We learned a lot through our various successes and failures during this migration, and in the time leading up to it.

For our system, we have a single set of internal WCF services hitting the database and half a dozen internal applications hitting those internal services. One of those internal applications is a set of externally-accessible WCF services, and on our customers’ behalf, we have some custom applications consuming those “public” services. Technologies/systems that we employ include the following:

  • SQL Server (50GB database where 23GB exists in a single table)
  • SQL Server Reporting Services
  • SQL Server Analysis Services
  • SQL Server Integration Services
  • WCF
  • .NET 4.0/MVC 2.0/Visual Studio 2010
  • Claims-Based Authentication (via ADFS)
  • Active Directory
  • Probably some more that I’m forgetting. If they’re important, I’ll add them back here.

By the end of the weekend, we had successfully migrated all critical systems to Azure (that we planned to) and only a couple non-critical apps still needed migration. We (temporarily) pulled the plug on one of our non-critical applications, in part due to migration difficulties and in part due to a pre-existing bug that needs fixed in it ASAP, so we decided to just tackle both at once the following week after getting some sleep. I can’t say the migration went without a hitch. While we had some unexpected major victories on some high-risk areas, we also had some unexpected major problems in some low-risk areas.

I’ll go over some specific experiences in some follow-up posts, but here were some major key points we took away from this experience that might help others. Some of these we knew about in advance and were prepared to deal with them. Others caught us by surprise and caused problems for our migration

  1. If you have a large database (anything more than 5GB), do a LOT of testing before you start migration! Backups, dropping/recreating indexes on large tables, etc! For instance, we have one table that we can’t drop and recreate an index on and the default ways to create backups take 8-10 hours for our database!
  2. When migrating your database to Azure, don’t do it with your “base system” being locally. Upload a .bak backup file to Azure blob storage using a tool like Cerabata’s Cloud Storage Studio (which allows you to upload in small chunks to easily recover from errors and improve bandwidth speeds) and create a medium-sized Azure Virtual Machine with a SQL Server Evaluation image, and base all of your data migration work from there. You’ll save so much time doing it this way unless you get everything working perfectly your very first try (unlikely). Otherwise, for just a couple bucks (it literally cost us ~$2 for the entire weekend’s worth of VMs we used), it’s totally worth it!
  3. AUTOMATION!! Automation, automation, AUTOMATION! You do the same thing over and over and over so many times, really, have a solid build server with automated build scripts for doing this! Do NOT use Visual Studio or any manual process! The ROI on investing in a build server will pay off before your 5th deployment, most likely, regardless of how complex or simple your system is!
  4. No heaps! You must have a primary key/clustered index on every single table. No exceptions! Period! Exclamation mark!
  5. Getting Data Sync up and running is a major pain in the ass! Azure’s Data Sync has stricter limitations than SQL Server’s Data Sync (for instance, computed columns don’t play nicely at all in Azure but SQL Server has no problem with them). There are just enough nuances and so much time that it takes to find them that this you can spend quite a bit of time just figuring this out. And then figuring how to automate these nuances is yet another topic of discussion since the tools are so poor right now.
  6. Use the SQL Database Migration Wizard to migrate your data from your “base system” to an Azure database. But be gentle with it, it likes to crash and that’s painful when it happens 3 hours into the process! Also, realize that it turns nullable booleans with a NULL value into FALSE and doesn’t play nicely with some special characters, so be prepared to deal with these nuances!
  7. Red Gate SQL Compare and SQL Data Compare are GREAT tools to help you make sure your database is properly migrated! SQL Data Compare fixes up the problems from the SQL Database Migration Wizard very nicely and SQL Compare gives you reassurance that indexes, foreign keys, etc. are all migrated nicely.
  8. As I said before, test test test with your database! For us, 8-10 hour database backups were unacceptable. Our current solution for this problem is to use Red Gate’s Cloud Services Azure Backup service. With the non-transactionally-consistent backup option, we can get it to run in ~2 hours. Since we can have nightly maintenance windows, this works for us.
  9. Plan on migrating to MVC 3.0 if you want to run in Server 2012 instances.
  10. If you’re changing opened endpoints in the Azure configuration (i.e. opening/closing holes in firewalls), you have to delete the entire deployment (not service) and deploy again. Deploying over an existing deployment won’t work but also won’t give you any errors. Several hours were wasted here!
  11. MiniProfiler is pretty awesome! But the awesomeness stops and becomes very confusing if you have more than 1 instance of anything! Perhaps there’s a fix for this but we haven’t yet found one.
  12. If you have more than just one production environment, it’s very handy to have different subscriptions to help you keep things organized! Use one subscription for Dev/QA/etc, one for Production, one for Demo, one for that really big customer who wants their own dedicated servers, etc. Your business folks will also appreciate this as it breaks billing up into those same groups. Money people like that.
  13. Extra Small instances are dirt cheap and can be quite handy! But don’t force things in there that won’t fit. We found that, with our SOA, Extra Small instances were sufficient for everything except for two of our roles. Except for those two roles, we actually get much better performance with 7 (or fewer) Extra Small instances than 2 Small instances for a cheaper price (1 Small costs the same as 6 Extra Small).

In the next post, we’ll go over the things that we did leading up to this migration to prepare for everything. From system architecture to avoiding SessionState like the plague and retry logic in our DAL, we’ll cover the things that we did to help (or we thought would help) make this an easier migration. And I will also highlight the things we didn’t do that I wish we had done!



Azure’s Service Bus and EnergyNet (PDC Day 0)


On Day 0 of Microsoft PDC, I attended the Software in the Energy Economy workshop. Much to my surprise (and disappointment), we didn’t talk about energy for the entire first half of the workshop. Instead, it was about Azure’s Service Bus. BAD Microsoft!! It was explained to us by Juval Lowy that he wanted to do it entirely on energy but Microsoft forced him to have half of the talk on the service bus. Now I can understand this from Microsoft, but this should have been clear to the people there. Honestly, I would have much preferred to go to another workshop than to learn about the Azure Service Bus. Cool stuff but my current analysis of it is that it’s way too unreliable (I don’t mean bugs, I mean lack of transactional support) and it is simply missing some of the things that I would want/need in such a system, like queues! There are hacks to implement them, but I don’t want to have such an important foundational part of my architectures built on hacks! Anyways, that can be for another day (gotta get to the keynote).

(more…)



Going to PDC 2009


Yesterday I got the thumbs up that I’m being sent to PDC 2009 by my company! I planned on going whether I was sent or not but it’s nice to not have to foot the bill out of my own pocket this year. In 2005 I was lucky enough that my company covered airfare and hotel while Telerik was generous enough to cover my registration for the conference and the pre-conference. Last year I wasn’t so lucky and had to cover almost all of it out of pocket but that’s what you expect as a contractor.

Here’s what I have to do:
(more…)



Pragmatic Spaces ’09


Pragmatic Spaces 2009 is coming soon.

Known details for now:
1. Limited to 150 people
2. You want to be there
3. There will be GREAT discussions you WON’T want to miss!
4. #pragma09 is something to watch for!

While not directly related to Indy ALT.NET, many of the folks from that community are involved in this event. Alan Stevens will also be an important person for the event.



Truncate Logs for SQL Server 2008


I had previously posted how to Truncate Logs for SQL Server 2005. Unfortunately, this method does not work in SQL Server 2008. The reason is because the “WITH TRUNCATE_ONLY” command is no longer in SQL 2008. Assuming you run in full recovery mode, the new script to do this is:

1
2
3
4
5
6
7
8
USE [{DatabaseName}]
GO
ALTER DATABASE [{DatabaseName}] SET RECOVERY SIMPLE
GO
DBCC SHRINKFILE({TransactionLogLogicalName})
GO
ALTER DATABASE [{DatabaseName}] SET RECOVERY FULL
GO

Simply setting the database mode into simple recovery mode performs the actual truncation but the file is not shrunk by that. DBCC SHRINKFILE will take care of that second step. And don’t forget to put it back into full recovery mode at the end!!

-Shane



How to identify the user your ASP.NET app uses to authenticate as


I was recently asked how to identify the user your ASP.NET application uses to authenticate as. This can be a simple question or a bit more involved. Let’s start with the simple answers first.

Default Accounts:
Windows XP, Windows 2000, and earlier (you should NOT be caring about earlier!):
ASPNET – this is created automatically by the .NET Framework

Windows Vista, Windows 2003, and newer:
NETWORK_SERVICE

Overridden Examples:
Just because that is the default, it doesn’t mean that it is that way for your application. The first place to check is in the Advanced Settings for your Application Pool. This should tell you the account your application will default to if you don’t override it within your application.

Next, you can check your web.config to see if you’re application is impersonating a user. This is the next level of defaults to check. (and perhaps machine.config too, but you should probably NOT be overriding it there!) This will override the above defaults.

Now that you have checked there, the next thing to do is identify what type of IIS Authentication is being used. If anonymous, then you’re done – it will default to the above defaults in that overriding order. If you’re using ASP.NET impersonation, then that should default to the above as well. If you’re using Basic, Digest, Forms, or Windows authentication, then the authentication will be based on the user that the end-user logs in as.

Things can get even trickier if you do things in code, but generally this will figure it out for you.

-Shane



New Indy ALT.NET Site is up


We now have a newly revamped Indy ALT.NET site up and online! While we’re not completely done with it, we are far enough along to go ahead and turn it on for the time being. Very soon we will be filling out some of the interactive portions but for now it is fast and has all of the necessary information there!

So why did we do this? Several reasons…
1. The original site was a wiki system manipulated into being a community-type site. It worked, but was a little strange.
2. The original host was donating their services to us and we had began to overstay our welcome.
3. Our original replacement .NET-based CMS was not as efficient for us to manage as we had hoped.
4. Our original replacement host had very serious availability issues (also donated hosting).

So as a result, we ended up choosing a PHP-based CMS and are paying a little bit of money for some proper hosting. And what you see is the result! A (soon-to-be) proper community site that is very responsive for you, very flexible for everybody, and very easy for us to maintain and update. All-in-all, a good site for the group! :-)

Major kudos go out to Sasha Kotlyar for all of the hard work with getting this up and running! Without him, we’d still have a pretty crappy site! :-)



IndyTechFest registration is now open!


IndyTechFest registration is now open! This year there is a limit of 500 registrations (I believe last year’s was like 400 and it was booked up within just a couple weeks). So I strongly encourage you to register sooner rather than later!

There is a great lineup of speakers and sessions at this year’s IndyTechFest! Some of the speakers I have seen speak before include Paul Hacker, Larry Clarkin, Michael Eaton, Arie Jones, Tom Pizzato, Dan Rigsby, and Bill Steele. There are many other great speakers that I know or have heard of. This should be an excellent event and one that is worth a good long drive to get to!

Some of the sessions that I’m really looking forwards to include Test Driven Development (TDD) w/ VS 2008, Tips and Tricks for the New C#, Tips and Tricks for the New VB .NET, Duplexing WCF in the Enterprise, and Virtualization of SQL Server. There are many other sessions that I’m going to hope to get to but alas, with it being a one-day event, I doubt I’ll get to most of the ones I really want to see. :P

Props to the people who worked hard to make this event possible, including Brad Jones, Dave Leininger, John Magnabosco, Mark McClellan, and Bob Walker, as well as all of the support of the local user groups to help drive the event!

Just as I was wrapping this post up, I received a phone call. Apparently as of 1pm (1 hour after registration opened), nearly HALF of all available registration slots were filled! If you read this post and have not registered, go register NOW and don’t wait or you’ll be left out!



inkubook


A friend and colleague of mine, Eric Willeke, has helped lead a team to create and deploy an interesting new site at www.inkubook.com where an ordinary person can take the pictures they have taken with their digital camera and make a simple and professional-looking bound photo book based on them. Erik has posted a bit more about it here. It’s a great example of a nice, clean, Silverlight 2.0 app that focuses on the end user experience and not on trying to show off the bells and whistles of the platform.

Ignoring the technology, this is a great site and idea! For as little as $14 (softback) to $25 (hardback), you can have the pictures printed and bound into a nice little book. His team has done a wonderful job and this site offers a great little product! If you’ve spent hundreds to thousands of dollars on a trip or vacation, why would you NOT want to dish out another $25 to be able to cherish it and share it with others? Also, if you forgot to buy somebody a souvenir while you were out, this would be a great thing to consider as well!

-Shane

PS. Mom, if you see this and get something like this from my trip to Vegas last weekend, it’s NOT because I forgot to get you something!! ;)



Indy Invades CinArc!!


On Tuesday (July 8, 2008) evening, Sasha Kotlyar, Dean Weber, and I made a spontaneous trip to Cincinnati to check out the CinArc group (not to be confused with this CinARC). This group is Cincinnati’s Architecture User Group and seems to be mostly .NET-based. They are a very new group as this was only their second meeting. They meet monthly on the second Tuesday of the month. Their current meeting-format is that of a fishbowl meeting. You can read more on Wikipedia about this here.

I must say, the three of us Hoosiers really enjoyed ourselves at CinArc! Despite the downpours, rush-hour construction, and construction barrels we had to dodge in the middle of the road, it was great! Oh, and I won a door prize as well! I walked out with a VS2008 Pro license (only had MSDN-based licenses before, now I have a permanent license!). The group is led by Mike Wood, who also happens to lead the Cincinnati .NET User Group. Lots of other people were also in attendance (I’m not even going to attempt to name them because I’m horrible with names and I’ll surely forget some of them but turns out I follow lots of them on Twiter). There was a total of 19 people there with 5 chairs in the middle of the fishbowl (1 moderator, 3 speakers, 1 open). It was great that they veered away from the norm where it was a very interactive discussion and almost everybody participated in it.

The agenda for the meeting was different than what I’ve been used to for user group meetings, and I really liked it! I’m used to food before hand, kicking things off with announcements, then going into the discussion for the rest of the night, and door prizes at the end. What they did, instead, was kick things off with the discussion, about an hour into it take a break for food, kick the second half off with announcements, go back into the discussion after the nice little break, and then door prizes at the end. The trick to accomplishing this is timing on the food and if it can be pulled off, I may actually try to assimilate this style into the ALT.NET meetings! However, one important part of the ALT.NET meetings, I feel, is the social time spent before the meeting. Perhaps we can have snacks and drinks available then and real food available at “Halftime”.

One other thing that was really neat was that the meeting attendees were able to choose the topics to discuss. Ideas were put up on a whiteboard as recommended by the people there, and then everybody voted on the ones that they were most interested in participating in discussion with. There were 3 topics that seemed to be the most popular, and it turned out that we had time to discuss 3 topics. So it worked out perfectly!

As I said before, I had a great time at CinArc and I highly recommend it to anybody who is in the area and is interested in best architecture practices and bouncing ideas off of one another! There were some extremely intelligent guys at this group and it’s great that they are trying to expand knowledge in the community and help one another! I can already see this is going to be a very popular user group in the future! I hope some of those guys come visit some of our Indy events, and I just may try and pick up some Reds tickets some second Tuesday of the month afternoon so I have a good excuse to be in town again for another CinArc meeting! :)

-Shane

Next Page »

Jaxidian Update is proudly powered by WordPress and themed by Mukkamu