Video Streaming Broadcasting
The Perform Group has access rights to one of the largest digital sports portfolios in the business. They are in partnership with all the big names in sports broadcasting including Sky, Al Jazeera, Star, the BBC, and all the major American networks. In the first nine months of 2011 they delivered over 2.5 billion individual video streams spread over more than 100 mobile and internet-based services, and by March 2012 they were reaching 105 million sports fans across the globe.The Group offers an enormous range of multimedia sports content, distributing it directly to sports fans and business partners around the world. They have offices across North America, Europe, the Middle East, India, the Far East, and Australasia.
Perform Group has a distribution and source code license for a digital streaming application called Unreal Media Server which is developed and distributed by Unreal Streaming Technologies. Under the terms of its license, Perform Group is entitled to develop their own solutions to meet their clients’ requirements, but they are obliged to seek outside assistance in dealing with two problems that they have been unable to solve internally. The first is a weakness in the system which, when one of the servers involved in broadcasting a particular feed crashed, caused the whole chain of servers to hang, and thus the feed to crash. The second issue concerns the latency of broadcast (the delay between the event happening and it appearing on the customer’s screen).
Perform Group initially selected Softage to assist them in resolving these business-critical problems and are so pleased with our performance that they invite us to remain in partnership with them and move on to the resolution of two further issues. These are the testing and maintenance of their media player and the extension of the functionality of their media servers.
The Perform Group depends entirely on the smooth functioning of the systems that they use to transmit video of the events to their customers. Any interruption of service can not only cause them immediate loss of income but also, and perhaps more significantly, considerable damage to their reputation in a business where reliability and reputation are everything. Persistent weakness or repeated failure could be catastrophic for the company in a field where the competition for customers is fierce and the field crowded. They were thus dismayed to find a significant number of coding errors Unreal Media Server. Some are minor issues that can be addressed over time, but there is one critical error that was causing repeated system failure.
The problem relates to the manner in which the broadcast was put together and delivered to the customer. In essence, a broadcast might involve a number of servers linked together, each of which could also be connected to its own source of streaming such as a web camera. In theory, the failure of one of those servers should have no effect on the overall system, but Perform discovers the failure of any one causes the whole chain to fail in a domino effect.
Perform Group’s own programmers and developers work hard on the problem, but, despite their best efforts and considerable allocation of time and resources, they are unable to replicate the defect. Without being able to replicate it, they are unable to fix it. They turn to Softage for help. Armed only with a description of the fault and a few hints about the nature of the bug that must be causing it, our team begins to crawl through the source code to pin down the problem. Over the course of the next two months they work closely with the Perform Group technicians and are able to identify the offending code, replicate the fault, and fix it successfully.
Needless to say, Perform Group is delighted. They are so pleased with Softage’s performance that they ask us to look at a second issue. A key constraint in any service which involves streaming live events is latency, or the delay between the event happening and the viewer seeing it. The customer understandably wants instantaneous reception of the event which is clearly impossible and so service providers strive to drive the latency level as far down as possible. For the majority of customers the desire for low latency is simply a matter of quality of service. For other customers latency is a rather more significant issue: Perform Group provides streaming to many bookmakers around the world and they must have very low latency values to prevent unscrupulous people from being able to take advantage of any delay between the event happening in one country and it being broadcast in another to place sure bets.
When they first turn to Softage, Perform Group is using the H264 codec to stream their video. The delay in transmission inherent in using that codec is up to two seconds, and while that may seem insignificant, it’s too long to meet their customers’ requirements. Perform asks Softage to see what we can do to reduce the latency to a much lower figure. Our team is able to reconfigure the codec to reduce latency by customizing its configuration to meet Perform’s needs. This involves a marginal and acceptable degradation of picture quality, but also a reduction in latency by roughly 75%. This service improvement meets all of Perform’s customers’ needs and is delivered during the summer of 2012.
Building on these ongoing successes, Perform asks us to help them in yet another area. They want Softage to develop a tool for them to facilitate their system monitoring and testing, and also with future system development. Each viewer of streaming video produces a load on the system and the larger the number of viewers, the higher the load. The difficulty is that testing and maintenance can’t be done on a live feed which is being watched by customers as the process itself will cause picture degradation, high latency, or even loss of the stream, something which the customers won’t accept. To get around this problem, Perform asks us to design and produce something emulating the load on the system caused by large numbers of people watching a particular stream, but not impacting at all on live feeds being watched by customers. This will allow them to conduct system tests, maintenance, and development without causing any service interruptions.
Our technicians and developers are able to design a service that emulates scalable numbers of viewers watching a stream and thus deliver realistic system loading. This allows for realistic testing and monitoring without interfering with the quality of service received by real customers watching live feed.
Having successfully delivered system emulation, Perform Group asks us to improve the functionality of the media servers by introducing an MPEG Transport Stream, or MPEG-TS. This is an industry standard format for the storage and transmission of electronic information, including video. We are able to do this quickly and efficiently, without any disruption to service delivery.
Softage is still working very closely with Perform to develop a new and exciting service which Perform believes will be very popular. At the moment, there is an option to record a feed and watch it later, but customers can’t watch any part of the broadcast until the entire recording is complete. Softage programmers and developers are producing a new module for the media servers called Archive Server which will allow a customer to begin watching the recording of a sporting event before the recording is complete. In addition, Archive Server will improve the fidelity and reliability of the archiving process and make more efficient use of the media servers.
A collaboration of this size and the sheer breadth of the tasks undertaken has necessitated the use of almost every tool in Softage’s toolbox. Our peoples’ experience and expertise has been put to the test, and we are delighted to say that we have risen to every challenge presented, delivering effective solutions quickly. Our developers are among the most capable in the business, and throughout this collaboration with Perform they have been able to draw on their vast knowledge and understanding, not only to solve the puzzle of the domino-like collapse of the servers, but also the unacceptable latency issue.
Tools and technologies
In their work on Perform’s media server, our developers worked with C/C++ on the servers themselves, and with С# on the Software Development Kits’ web configuration to deliver solutions that would work on Windows XP, Vista, and Win7. They needed to employ their expertise with Microsoft Visual Studio 2010, ATL/WTL, COM and with the SDKs for Windows 7.1, DirectX and Windows Media Format. They also need to work with Cairo, a vector graphics library with display and print output, as well as POCO.
When manipulating and optimising the H264 codec to overcome the latency issue, they used their skills in libx264, a decoding library for H264, and with the MainConcept H264 decoder.