BBC Digital Media Initiative (DMI)

Summary

Project Length: 1.5 Years

Products: Custom Codec and File Container, Custom Sequence Renderer Supporting 25 effects from Avid MC / FCP / Premiere and rendering them all identically, Custom Disk and Tape MAM, Partial File Restore from Disk and Tape Application, Custom Avid MC / FCP / Premiere EDL conversion application

Roles: Tester, Promoted to Test Manager, Promoted to Test Lead, Promoted to Test and Development Lead, Solution Architect, Promoted to Technical Project Manager

HOW I MADE A DIFFERENCE

While I understand how the overall project failed, I’m proud of my contribution towards my small part of it and how hard I worked on it. With all the responsibilities I took on there was a period where I was working 9am-2am, eating all meals in the office. I was one of few to stick my head above the parapet to raise and amend issues to ensure that our products worked and were delivered in a timely manner. In addition, while only frame accuracy (25fps) was specified, it  became clear that 1/48,000 of a second (audio sampling rate) of accuracy was required and I rose to the increased challenge.

I rose to every challenge and got promoted on several occasions to take on more responsibility, beginning as a tester but being finally promoted through to being responsible for and running our entire arm of the project.

Project Deliverables

We were required to develop and deliver:

  1. A custom codec and file container that was platform agnostic
  2. A custom disk and tape MAM that could manage the same assets across multiple physical sites (for redundancy) and access the assets from the fastest / most available based on the location the user was accessing content from
  3. A transcoder and transwrapper that supports a wide range of codecs and wrappers from a very wide range of devices and software
  4. A custom sequence renderer supporting 25 effects from EDLs from Avid Media Composer 3, 4, 5 and 5.5, Adobe Premiere and Final Cut Pro
  5. An application that can restore user selected parts from a rough cut sequence of media assets from tape / disk to disk rather than the entire asset (for network transfer efficiencies) and rebuild the sequence in a local Avid MC, FCP or Adobe Premiere NLE
    • This needed to include configurable ‘handles’ so that editors can trim / extend the length of a clip on a timeline to tweak/improve an edit while only showing the part of the media that was originally specified in the sequence against the full-length source asset
  6. Custom Avid MC / FCP / Premiere EDL conversion application

ROLES

  • Test and Development Lead

    • Wrote project test strategies and plans
    • Managed 11,000+ automated and manual tests per day
    • Managed local and remote test and development teams
    • Took a ‘risk-based’ approach to testing
    • Led daily stand-up meetings
  • Solution / Software Architect

    • Reviewed and designed our response to requirement specifications
    • Redesigned low-level software workflows and UUID management
    • Defined software tolerances that would calculate resulting system behaviour
  • Technical Project Manager

    • Managed the project for Mediasmiths
    • Reported to the CEO and CTO
    • Reported to and liaised with the client
    • Responsible for ensuring project milestones were met
    • Managed a RAID log

Tester

I was originally hired as a tester, however as the pressure, project complexity and responsibilities proved to be too much for the test manager. I stepped up to fulfil the role.

Promoted to Test Manager

The SoW specified that a large range of codecs, in a range of containers/wrappers from a number of cameras (etc.) was in scope. While all of the codecs and containers are based on standards, from experience, each camera, recording device and NLE encodes slightly differently.

As test manager, I began writing the test plans for the products we were delivering. To comprehensively test the many variances of media and application configurations  (all the varieties of supported codecs, file containers, metadata, rendered effects, EDLs, etc.) I calculated that we needed to perform over 11,000 tests with results accurate down to 1/48,000 of a second. New versions were planned to be released to the test team every day. This clearly wouldn’t be possible with the few testers available and the limited amount of time between functional releases so I put together a proposal for a different approach to the testing. This consisted of the following:

  • We identify, collect and organise a full complement of test media
    • Every video and audio codec in every media container (muxed/interleaved and demuxed/atomised) from every supported camera and NLE
      • Short promo length (30 seconds) versions for faster regression testing
      • Long-Form episodes (~40-50 minute) versions for more realistic load testing
  • We develop an automated test suite
    • New tests would be created alongside the planned delivery of new functionality until all of the 11,000+ automated tests were developed
  • We hire another 2 testers
  • We take a ‘risk-based’ test approach. This included:
    • Identifying the scenarios with the highest chance of failure and the highest impact on users/business should it fail
    • Human testers would then manually perform or review the automated test results in more detail for these tests

Promoted to Test Lead

After significantly improving the testing effort I was then promoted to test lead on the supplier side. This involved writing an updating the test strategy and plans. I became responsible for managing the full test team of 6 people of varying levels of experience. Some members of the team were new to testing and all but 2 had no previous experience of testing with media assets. In addition, I became responsible for liaising with the leading our UAT sessions, where we had to identify and agree a subset of tests that would prove our solutions to be working as required.

As different parts of the project were nearing completion, it became possible to widen the integration testing with other supplier applications and hardware to test our workflow applications as part of the wider workflow. I wrote the test strategies and plans and was responsible for achieving their sign off with the BBC.

This included managing both local and remote development teams.

Promoted to Test & Development Lead

Due to my success in managing the test team and reporting defects/working with the development team and my understanding of the solution, I was also promoted to manage and lead the development team. I employed an agile-like development methodology (I say ‘agile-like’ as there was a fairly clear specification in place in advance) with daily stand-ups and a lot of interaction between development and test teams. This meant that we could immediately test and feedback the results of test into development to maintain a high pace and quality of the applications.

Solution Architect

Once leading the development team, I came to discover that the design of  a key part of the workflow  was incorrect and would not work. For example, it required that media would maintain genealogy by inheriting the ID of the parent media. Avid Media Composer required this ID to be unique, otherwise it would fail to load the assets at all. I escalated the issue to the suppliers CTO and ran tests to provide conclusive truth that the current designs wouldn’t work and presented it to the client. I then worked with the client’s Solution and Enterprise Architects to redesign a suitable alternative that could achieve the requirements in a way that would work with the required 3rd party tools. By identifying this early, it greatly improved the chance for success as the project timescales were already very short.

Promoted to Technical Project Manager

After successfully managing the test and development teams and also leading our technical design response to the BBC’s requirements specifications, I then led the project, reporting into the CEO and CTO and directly to the client. I became responsible for ensuring that all milestones were met, liaising both to internal and the client’s senior management teams and managing a Risks, Assumptions, Issues and Dependencies log.