Three years ago, Comcast was delivering
video-on-demand essentially the same way McDonald’s
serves burgers or Starbucks whips up lattes: locally, out of dozens
of locations across the U.S.
The operator had to shift to a centrally
managed, hierarchical distribution
model after deciding it wanted to
give subscribers access to tens of thousands
of VOD selections — with a vision
of hitting 100,000 and beyond.
Back in January 2008, Comcast CEO
Brian Roberts outlined “Project Infinity”
in a keynote at the Consumer Electronics
Show. Among his promises: 1,000 high-definition titles on-demand by the end of
the year, and eventually a virtually limitless
selection of video entertainment.
Meeting that initial target required
some heavy lifting, given the isolated nature
of Comcast’s VOD infrastructure at
the time. “That was about adding more
storage in literally 130 locations to get to
1,000 HD choices,” John Schanz, Comcast
executive vice president of national engineering
and technical operations, said.
Today, the operator’s distributed VOD architecture looks
similar to an Internet content delivery network, or CDN. Like
its Internet counterparts, the Comcast CDN for video, dubbed
CCDN internally, stores all content on large-scale “library servers”
and stores cached copies closer to subscribers based on
the popularity of the content.
‘WORK AS A TEAM’
The project involved linking those 130 “islands” of VOD servers,
now down to about 90, into a nationwide network fed
by the library servers in four regional locations. In the middle
are about 100 mezzanine gateways that cache VOD assets
and manage the content that is distributed to the edge servers.
“We have unified those islands in a single operating platform,
monitored in our network-operations center,” Comcast
senior vice president of product engineering Mark Muehl said.
“We have taught them to work as a team.”
Most of the CDDN upgrade was executed without downtime.
About 10% of the work required taking VOD servers offline, which was done in early-morning service windows to
minimize the disruption for customers.
The CCDN project has been like “literally rebuilding an aircraft
carrier as it’s landing planes,” Muehl said.
Schanz added: “Sometimes, in deploying a new technology,
you are forgiven with low usage in a greenfield environment.
But we were stepping into the highest-scale VOD environment
in the world.”
Comcast now offers 30,000 VOD choices in
markets served by the CCDN, compared with
740 when it first launched video-on-demand
service in 2003. The CCDN serves about 80%
of the operator’s footprint.
The architecture has helped push up the
amount of content subscribers watch —
Comcast announced in May that it had surpassed
20 billion VOD sessions to date.
Moreover, with additional content and better
search features, Comcast subscribers are
watching a greater variety of content.
“We’re seeing a longer tail,” Muehl said.
And the CCDN architecture is “horizontally
scalable,” Muehl said, meaning it really is capable
of delivering an ever-expanding number
of videos as long as there’s enough bandwidth
and storage to deliver them.
“As we add more content to the library, there
are different parts of the service we need to
scale up,” he
said. “In some sense, this is a
living organism.”
NEXT-DAY DELIVERY
The centralized nature of the
CCDN has allowed the operator
to update VOD content
more quickly. For example,
Comcast now offers TV episodes
of popular shows from
the four major broadcasters the
day after they air, after reaching
deals with Fox and ABC
earlier this spring.
The CCDN also enabled
Comcast’s launch of dynamic
VOD ad insertion this summer
in 17 markets, representing
about 7.4 million subscribers, by
providing a centralized means
to distribute the ad assets.
Comcast declined to specify the vendors of the hardware
and software it used to build the CCDN. Schanz noted, however,
that the VOD back-end is based on multivendor, open
standards.
Cisco Systems is now Comcast’s primary VOD server vendor,
according to several industry sources. The operator has
removed SeaChange International’s VOD servers from the
production environment because they did not fit well into the
hierarchical CCDN architecture, but Comcast still uses the
vendor’s back-end management software, sources said.
SeaChange declined to comment except to say the company
“continues to have a very strong relationship with Comcast,
particularly for software and the associated hardware.”
Another component of Comcast’s revamped VOD service
is the UDB — the Universal Database. That software
provides a comprehensive catalog of all on-demand assets,
with the associated metadata.
Comcast publishes information about individual VOD assets
once, and that metadata is distributed to about 100 UDB
servers around the country.
When a Comcast customer navigates through the VOD
menu on TV, or does a search for shows or movies on XfinityTV.com or the iPad and iPhone apps, the query is handled by
one of those UDB servers. The Universal Database system processes
200 million menu navigations daily, Muehl said.
The UDB software was developed by Comcast, whereas
the operator worked with
various vendors for the hierarchical
caching and content
management functions
of the CCDN.
The CCDN’s first application
is video-on-demand, but
Comcast execs said it could be
extended to others, such as a
network-based digital video
recorder service. The MSO expects
to conduct a small test of
the network DVR concept this
year but has not announced
any commercial plans for
such a service.
“It’s a platform,” Schanz
said of the CCDN. “It’s optimized
for video, and ultimately
I think of it as our
platform for anything time-shifted.”
BACK END OF INFINITY
Details on the Comcast CDN:
Architecture: All VOD content stored in library servers
at four regional data centers, with the most popular
titles cached on edge servers in 90 locations.
Library server locations: Philadelphia, Atlanta,
Chicago and San Francisco
Storage: 150 Terabytes of usable disk space in each
library server (enough to hold roughly 64,000 hours
of SD or 16,000 hours of HD MPEG-2 video).
Video-on-demand lineup: 30,000 titles in CCDN-enabled
markets.
Deployment: Just over 80% of Comcast’s footprint.
SOURCE: Comcast
