posts

building a streaming site

It’s generally well accepted that at times I am an idiot. While not unique to me, my idiocy manifests itself in large seemingly pointless projects, where I spend months to create something I don’t even need.

So, I decided to build a streaming site. My own personal Youtube, Netflix, Stream.

Why?

This is a good question. A while back I wanted to share a family video with my relatives around the world and put it on Youtube but as a private video as I do not feel a need to share this with the world. Interestingly enough, while pretty much all my relatives have a gmail account and can thus access my youtube video in a password protected way, only my mother actually managed to accomplish viewing the video.

So, that got me thinking that perhaps I could just host it myself on a password protected page and they can all access it without having to wrestle with Youtube’s horrific user interface.

But that’s not really the reason to do this.

I am an IT consultant and I make my living figuring things out, recommending solutions and generally acquiring knowledge that others benefit from. A project such as this is useful because to takes me down a lot of different rabbit holes. Each rabbit hole is a different piece of technology I end up spending a couple of weeks with while learning stuff. It’s a great way of becoming more knowledgeable.

I figure I may well share some of that knowledge.

So where I am at?

I actually started this in Spring 2019 (that’s September here in Australia) and while do I have a working site, I am not finished yet.

Here are some of the things I ended up working on:

  • Angular
  • Firestore
  • CosmosDb
  • AWS media conversion
  • Azure Media Services
  • CosmosDb graph database
  • C#
  • Asp.net
  • CloudFront
  • FrontDoor
  • HTTP streaming
  • RBAC
  • OpenID Connect
  • Azure B2C
  • Azure Functions
  • Azure Service Bus

In fact, the first iteration of my site ran on Firestore, using google storage. I’ve also used AWS media services, Azure Media servers, different CDNs etc. In fact I built two complete frontends, one in Angular, one in Asp.net.

Probably the hardest thing I struggled with was creating a role based access structure where I could give different permissions to different people.

At the moment I have a working asp.net based site, where I can upload a video, it will be converted, thumbnails created and then available for streaming through the site.

So where to now…

I am nowhere near finished. I managed to build a working permissions system that can be granular, now I want to add groups to this. I release I should document these things a bit more and write blogs because there’s a lot to share. So, I guess that’s what I’ll start doing.

From what should be obvious, I am currently mainly using Azure technologies. There’s a reason for that. One was employment related (I was employed by Microsoft for several years), the other is that of all the programming languages I know (some javascript, Powershell, C#) I enjoy working with C# probably the most, Visual Studio is rather helpful, and there’s tons of documentation available.

My first iteration was based on FireStore. FireStore is interesting and really quite nice because it’s a noSQL database that you can use without having to bother building an API. I was using Angular, and with the right libraries you can use FireStore which can handle almost everything including storing files, authentication etc. However, I had 2 significant problems within Google land:

  • Google has no service that can convert a standard video in a format that can streamed for HTTP requiring my to other using AWS to do that, or do that offline.
  • FireStore can serve storage, but works on a principle that for each file a unique secure URL is created to download that file. That is a great way to secure files, but doesn’t work for the files used when streaming videos. Streaming videos involve a lot of files and I just didn’t want so start writing a service that could create the URLS for each fragment of a video and update the index files.

So, while I manage to build a site showing a video, I could not provide any security or protection to it. Some elements with the configuration (such as access permissions) are also not really all that well documented. This is unforunate, but this whole site was working and cost me essentially nothing to run. Building an API using .net core and Cosmos DB isn’t all that time consuming either if you know what you are doing. (I didn’t, but I am much better now).

I also looked at AWS. AWS actually has a good service that can convert your video files into the streaming files (this is really a topic for another blog), and while there’s some great samples provided by AWS, they work with Node.js which uses JavaScript a language I struggle to like.

So, now I am using Azure Media Services which isn’t at all cheap, but I did manage to get everything working the way I want. The files are being converted, they do stream, and I can fully secure them (and even add DRM should I want to). Whether I’ll keep that running is another story. At one point, I even built a graph database that connected what needed to be connected. It was neat, but the trickery to ready data ultimately made this one a bit too much work.

Current plan going forward:

  • build out the RBAC structure to include groups and inheritance
  • build out the API
  • build a prettier Angular GUI

More stuff to keep me busy for at least another 6 months or so :)

While media services was an expensive way to host things. It did work very well. Microsoft has deprecated media services this platform is going away. Properly encoded video can be hosted from a storage account. I am now trying to figure out how to send a token along when requesting the video so that I can keep this an authenticated solution.