Azure’s Service Bus and EnergyNet (PDC Day 0)
On Day 0 of Microsoft PDC, I attended the Software in the Energy Economy workshop. Much to my surprise (and disappointment), we didn’t talk about energy for the entire first half of the workshop. Instead, it was about Azure’s Service Bus. BAD Microsoft!! It was explained to us by Juval Lowy that he wanted to do it entirely on energy but Microsoft forced him to have half of the talk on the service bus. Now I can understand this from Microsoft, but this should have been clear to the people there. Honestly, I would have much preferred to go to another workshop than to learn about the Azure Service Bus. Cool stuff but my current analysis of it is that it’s way too unreliable (I don’t mean bugs, I mean lack of transactional support) and it is simply missing some of the things that I would want/need in such a system, like queues! There are hacks to implement them, but I don’t want to have such an important foundational part of my architectures built on hacks! Anyways, that can be for another day (gotta get to the keynote).
So this is by no means a complete or even a good write-up. This is simply a dump of my notes that I was taking until my battery died, early on in the Energy conversation, and I have yet to even go back and re-read them, much less make them make sense. Just trying to get this out there in case it’s useful to anybody. Luckily, there are a few links in here (and I have some more in my phone that I will add to here later) that provides the slides for the Energy part (is a must have if you are interested in it)! Finally, forgive me as I have yet to master the Word to Wordpress formatting…
Software in the Energy Economy
· Azure Service Bus
o Very heavily based on WCF
o Very high performing over TCP but not HTTP
§ Demos to servers in Texas from LA perform ~70% speed of light (demo was over a cell phone even)
o Very good parallel scaling
§ 1 subscriber performance = 1,000,000 subscribers performance
o Service types
§ One-way
§ Event-based
§ ???? (another one)
o Good for intercepting/proxying public calls
§ Abstract your service calls through the Service Bus
o Network routing
§ You can force clients to always go through the relay
§ You can optionally go to “Hybrid” mode and it will negotiate the network path and use the “best” path, including intranet paths. If it cannot find a better path, it continues going through the relay
o Decent for pub/sub models
§ Standard way requires lots of code duplication, so you’ll have to dynamically build endpoints/operations/events/etc. which involves lots of reflection
o Bad for buffering
§ Buffers for only up to 10 minutes
§ At most, 50 buffered items
§ No transactions so catastrophic failures = complete failure of the buffer with little recourse for recovery or gracefully handling it
o Very bad for queuing
§ Queues were removed and are now nonexistent because they didn’t know how to bill for it.
o No end-to-end transactions
§ All communications do not have guaranteed delivery.
§ Treat as UDPesque communications.
o Requires the services to register before any consumers try to hit it
§ Does not work with IIS or WAS
§ Requires self-hosting or ASE
o Queued Services
§ Juval wrote some pseudo implementation of Queued Services – do not attempt to do this at home!
o Response Service
§ No way to get results or errors (because services are queued)
§ User a Service Buffer and a Response Buffer (i.e. have the client poll the Response Buffer until it finally gets the response)
o Service Bus Security
§ Shared Secret
· No good “easy” security frameworks for this
· Luckily, the WCF Security applies – nothing new
· Tokens (47 random characters) used for authentication
· Servers/clients may use different tokens
· NOT intended for user authentication
· No really good options for doing this on user machines
§ Transport Security
· TCP w/SSL
· WS w/HTTPS
§ Message Security
· Credentials are built into the message
· Recommendation: Message Security with no service authentication (i.e. relay service hits my service anonymously)
§ TCP Message Security
· Service-side cert to protect message
· Client validates cert
· Private/Pub Certs
· Problem is that the relay’s domain (Azure) does not match our cert
· You can get cert for Azure domain and then use partial validation based on names to decide whether or not to trust the cert.
§ TCP Mixed Security
· Does not work with Hybrid
§ One-Way Transfer Security
· Defaults to Transport
· Cannot use Mixed, only Transport or Message Security
· With Message Security, client cannot negotiate service certificate – no direct interaction with service, must specify cert explicitly
§ Transfer Security
· Requires cert for message security
· Reads from config file, else looks locally for cert that matches service namespace
§ Recommendation
· Use Message Security with Credentials
· Creds can be un/pw, cert, or token
|
Binding |
None |
Transport |
Message |
Mixed |
|
Message |
|
|
|
|
|
Message – Hybrid |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
· Demos: www.idesign.net/idesign/download/ServiceBusPDC.zip
o Features to come in future releases
§ Buffers
§ Queues
§ Routers
· Withdrawn from November 2009 CTP
· Re-introduced in second release (post 2010)
· Multiple services can subscribe to a router
· Admin administers routers independent of services (??)
· Load Balancing (quasi-round robin) with ~95% efficiency
· Router-to-router subscriptions (N to M subscriptions)
· Hierarchical (cutting off parent cuts off child) – great for admins!
· GREAT for logging implementation – the application can be done without logging (except for exceptions)!
· Can be used as a crude pub/sub BUT…. –>
· Router is limited to how many subscribers it can have (limit was 50)
· Can be used to implement queued pub/sub (i.e. also implements true queues as long as the time limit isn’t a problem)
· EnergyNet
o Slides available at www.idesign.net/idesign/download/EnergynetPDC.zip [EDIT] That previous link seems to be dead but this one seems to work: www.idesign.net/idesign/download/EnergynetPDC2.zip
§ NOT Microsoft’s opinions, just IDesign’s and Juval’s
o Opinion
§ Massive shift in energy use driven by:
· Technology
· Economics
· Peak oil
· World demographics
· Regulation and stimulation
§ Will affect every walk of life
§ Shift is primarily a software problem
§ Requires software system like the internet
· Though bigger in every respect
· Cloud will be enabling technology
§ The next big thing – “will take until 2025 to finish it”
· Will trigger massive boom in software like dot-com (“but three times larger”)
· Tectonic plates have already shifted
§ Not just the US
· Immense export market
o Roadmap
§ The energy economy boom
· Where the investors and the startups are heading
· The new California Gold Rush
§ Current vs Future energy economy
§ Why the energy economy is actually a software problem
§ Emerging Standards
§ Key technologies and skills
o The Energy Economy Boom
o Shoeshine Boy vs The Daycare
§ Joseph Kennedy made his fortune by identifying the 1929 bubble
· Legend has it he was alarmed when getting stock tips from a shoeshine boy
§ But the opposite is probably true at boom formation as well
· The Daycare Syndrome in the Silicon Valley
· Just driving around will do
o The Daycare Syndrome
§ In the school my girls visit are also the kids of
· SVP of SAP who quit to join electric cars startup
· Former CTO of BEA who quit to be ALT-E VC
§ Around in the Silicon Valley
· Founder of PayPal launched the electric sport car company
· Former president of SAP founded electric cars startup
· Kleiner-Perkins
o VC of Amazon, Google, Sun, Netscape
o Now supporting three electric cars startups and many other including smart grid startups
· What my wife said about this seminar
o “Everybody knows that”
o The Energy Economy Boom
§ Investors are all jumping in
· Shifting from info-tech to clean-tech
§ VC poured $2.6 billion in 2007
· Up 400% from 2005
§ VC poured in $5.8 billion in 2008
§ 150 new solar startups
§ ConnectivityWeek
· Energy Information Technology for Sustainability (Santa Clara, CA, June 8-11, 2009)
· Smart Grid, Smart Devices, Smart Services
§ “Google wants US to use 100% Alternative Energy by 2030”
§ IBM, Cisco, Google, Oracle
§ Books: “Green investing”, “The Clean Tech Revolution”, “Profit from the Peak”, “Profiting from Clean Energy”, “Clean Money”, “Apollo’s Fire”
§ “New Bible”: Thomas L Friedman – Hot, Flat, and Crowded (for the mainstream public and media) – “Why we need a green revolution and how it can renew America”
· Gets things both very right and very wrong
§ Summary: All successful tech companies and execs are all jumping into this industry
o The New California Gold Rush
§ It is 1849 and 1942 all over again
§ All eyes are on California
· World’s 8th largest economy
§ “California Green Rush” means severe talent shortage for Energy Companies
§ In 2008 $3.3 billion clean tech investments in California
§ 80% of nation-wide money in the country goes to California
§ California alone has 11 electric cars start-ups
· They make bumper-to-bumper cars
§ Scores of other electric transportation companies
· Bicycles
· Forklifts
· Motorcycles
· Busses
· Components and technologies
· Infrastructure
· Conversions
§ Silicon Valley leading the charge
· Should it be called Clean Valley?
§ Charge stations all over San Jose in 2010
· Topping off at parking lots, malls, etc.
· RF IDs to utility bills
· Better Place will cover California with battery charging and replacement
· Better Place will cover Silicon Valley in total first
o $1 Billion from cities
o Infrastructure first cars later
§ California is prime solar energy industry
§ 127 solar startups up CA
· And counting
· 80 in Silicon Valley
§ Massive installation in the desert
· Several gigawats in the network
§ Asura in California uses troughs and rotating mirrors
· $0.1/KW
§ eSolar uses modular mirrors and multiple small towers
· 245MW in the works
§ World largest PV plant in the works
§ US largest solar installation on 65m sq ft of commercial building roofs
§ CA is prime for geothermal
· Gets 4% of its power from geothermal
· Operates world largest geothermal field
§ Aspire to lead nation with massive wind projects
§ CA has score of micro wind startups
§ Some ideas are out of this world
§ Cool Earth has a solar earth balloon
§ LS9 uses bacteria to produce fuel (oil)
· Used genetic engineering to refine it
· Can tweak things to eliminate the need to refine the oil
§ Across town Solazyme grows algae for artificial oil and jet fuel
§ Nano tech based batteries for quick charge
· Charge 35kWh in 10 minutes
§ Aggressive policies
· 20% renewable power by 2010
· 33% in 11 years
· 30% car emissions reduction in 8 years
· AB 920
§ Solar tax perks essentially make it free to install solar panels on your roof
· $6k for individuals, more for businesses
o Perception vs Reality
§ When people hear “Clean Energy” they think “green”
§ When they think “green” they imagine tree-huggers
§ Tree huggers are about peace and love
§ Peace and love are good for hippies only
§ The real picture is more about money growing on trees
· About Energy
o 40% is oil
o 70% of oil is for transportation
o 96% of transportation is oil
o 70% of electric is wasted on transportation
· Software and Alternative Energy
o What does Alternative mean
§ Being smart about energy
§ Not necessarily “alternative”
§ Will take decades to replace coal with clean energy
· Required for base load
o Electric cars are nonviable as-is
§ Limited range
§ Long charge time
§ Charging infrastructure
§ Billing
§ Grid cannot integrate large deployments
· There is simply not enough juice in the grid to support electric cars
· Need to triple the energy capacity of all current systems to support it
o Dumb grid cannot rely on
§ Solar and wind for base load
· Incompatible due to their unpredictability
§ Millions of small generators
· Actually put grid at jeopardy
o Only way to solve is with software
· The Power Grid
o 50% is coal
o Multiple independent grids with 3200 utility companies
o Power Grid Today
§ Grid was never well designed or pre-conceived
· Grew organically over time
§ Masive patch work
· As if designed by Rube Goldberg and Wile E Coyote
§ It’s a wonder the lights never flicker
· Compensates with excess capacity