Importer Module

The first module in the Liquid News backend pipeline is the importer. This module retrieves the metadata from the channels defined in the MongoDB database. The module makes a REST API call to the YouTube Data API for each channel in the channel collection. It requests the metadata for the max_results most recent videos, where max_results is a user-defined argument. 

Transcription Module

The downstream modules in the pipeline rely on accurate transcription and captioning of every video. Initially, the transcription module relied on captioning provided by the YouTube Data API to generate transcriptions and sentence-level captioning. However, this failed due to unreliable and error-prone captioning. Instead, the transcription module utilizes the SoTA Conformer speech-to-text model that builds on the well-known Conformer architecture provided by AssemblyAI. 

Topic Extraction & Assignment

The topic module is responsible for assigning each video in the metadata collection a topic. The list of topics that can be assigned to each video is the smallest clustering that expresses the latent clustering of all the video titles. We leverage the GPT-4 large language model to identify this set of topics. Rather than building or fine-tuning an existing language model, the topic module uses prompting as a form of few-shot learning to solve the clustering tasks. Recent work signals that few-shot learning is more robust than fine-tuning.   For each video in the collection, we formulate the following problem using natural language.  

Given a collection of videos 𝑉 = [𝑣1, . . . , 𝑣𝑛], assign each video 𝑣𝑖 to a cluster from an unknown collection 𝐶= [𝑐1, . . . , 𝑐𝑘 ], where |𝐶 | is not known a priori. The solution to this task will be 𝑘 = |𝐶| and a |𝑉 | x |𝐶| binary matrix 𝐴, such that 𝐴𝑖𝑗 represents if video 𝑣𝑖 belongs to cluster 𝑐𝑗. 

Segmentation Module 

The segmentation module solves the following segmentation tasks:

Given a video 𝑉 = ⟨𝑇,𝑆 = ∅⟩, and a collection of consecutive sentences 𝑇 = [𝑇1,...,𝑇𝑁], split 𝑉 into a collection of disjoint segments 𝑆 = [𝑆1,...,𝑆𝑀]. Each segment 𝑆𝑖 = ⟨𝑇𝑖, 𝑙𝑖⟩ contains a sequence of sentences 𝑇𝑖 ⊆ 𝑇 and a subtopic label 𝑙𝑖 that describes the topic of the sentences. 

Following the intuition from the topic module and leveraging the task-agnostic ability of GPT-4, we define the segmentation task as a chain of natural language prompts. 

Subtopic Extraction Module 

The subtopic module solves the clustering tasks at the segment level. The clustering task at the segment level requires solving the task for each topic 𝑇𝑖 to identify subtopics 𝑆𝑖 = [𝑆1𝑖, . . . , 𝑆𝑘𝑖] for all 𝑖 in set 𝑇. 

Summarize and Sort Module

The summarization module generates a bulleted summary for each video clip. The sorting module organizes video clips in a semantic manner, where videos with similar embeddings are organized near each other. 

Use Liquid News