Back to search

IKTPLUSS-IKT og digital innovasjon

Tidal News: a Middleware Platform for Adaptive and Dependable Data Dissemination in the Internet of Systems

Awarded: NOK 10.6 mill.

A tutorial on Paxos produced in the context of the project was published in the popular Hacker News web site for recent advances in IT and even made it to the prestigious list of top 30 news items: https://news.ycombinator.com. Additionally, the Tidal News project has achieved results that have made an impact on Spotify, and Internet and TV provider Altibox. Spotify is known for its music streaming services delivered to dozen of millions of customers in Europe and North America. Apart from providing instant access to over 20 million music tracks, Spotify also enhances its users' music experience by providing various features for social interaction. The Tidal News project has conducted an in-depth study of the Spotify architecture behind these features, which is designed to deliver real-time as well as offline notifications to Spotify users. The study additionally analyzes real data from the production system and derives a number of key characterizations of the performance and trends, e.g., with respect to bandwidth consumption and load balancing. The study has given Spotify a number of insights and contributed towards future improvements of the deployment. A postdoc formerly employed by the Tidal News project has become a system architect at Spotify and contributed to the design of the Spotify architecture. We have also addressed the question of resource allocation for the service of providing social notifications. Optimized resource allocation can benefit enterprises such as Spotify that deploy the service in an in-house data center as well as enterprises that utilize clouds such as Amazon EC2 for the purpose of deployment. The Tidal News project has contributed to research that makes it possible for Altibox to compute the effectiveness of individual TV commercials in real-time. This can be used as a value-added service for Altibox. The system is based on a combination of analysis of data from set-top-boxes in customer homes and advertisement detection in video streams. The data processing involves large volumes of data from multiple sources, and the results are updated incrementally, which can then be published as a live stream. Additionally, the technology for enhancing the reliability of privacy-preserving dissemination has been integrated within the popular Tor system. Tidal News has also developed a generic platform called Goxos, which can be used to develop services with demanding requirements on availability and robustness towards failures. In this platform we have developed a new and effective mechanism, called live replacement, to replace faulty components while the service continues to provide service to clients. The project has also developed a theory necessary to devise a platform for Byzantine fault tolerance in large-scale publish/subscribe systems. We are currently working towards implementing this using the Goxos platform. The proposed solutions and models of the research activities above are the invaluable and fundamental measures for the scientific quality of our research outcomes.

It is envisioned that the Internet will evolve in a system conglomeration of enormous scale. Smart energy-savvy inhabitant-friendly homes will become inter-connected in a single network. In particular, this network will provide feedback to the energy grid and allow effective distributed management of electrical appliances. Medical centers will monitor patients in need at their homes and send information to hospitals, personal doctors, and other medical centers upon demand. Thousands of business and produc tion processes at different factories will be remotely monitored and coordinated so as to form optimized production chains. Even nowadays, electronic travel agents (such as Expedia.com) build itineraries that combine flight reservations with hotel booking , car rentals, and sea cruises. Such a travel service is essentially composed of dozens of individual services working underneath, transparently for the customer. It is anticipated that electronic services of tomorrow will be composed of hundreds and even thousands of inter-connected distributed components. All of the above subsystems of the Future Internet will need to exchange information with each other. How will we manage efficient data communication on such a gigantic scale? How will we prevent a sl ower or less reliable individual component from stalling or failing the entire system? If the amount of available resources (such as bandwidth) falls below optimum, how will we ensure that the performance degrades gracefully and the composite service does not cease functioning? What is the best way to capture and express those features of data communication that enhance user-perceived experience? In the context of the Tidal News project, we tackle these and many other problems that arise due to inter-con nection of systems, in pursuit of supporting the vision of the Future Internet.

Publications from Cristin

No publications found

No publications found

Funding scheme:

IKTPLUSS-IKT og digital innovasjon