AI & ML Engineering

Cue

I built Cue to take a search result beyond a video name into an inspectable time interval. Each result shows a frame, transcript and the ranking contributions from image and text channels.

Fixture visual searchExplore the interface ↘
Project
Local moment retrieval from images and timed transcripts
My role
I built windowing, two-channel retrieval, rank fusion, the CLIP comparison and the local results interface.
Year
2026
Contents

Finding a moment someone can inspect

I built Cue to return a useful place inside a recording. A search result needs a source and a time interval, with enough evidence to decide whether opening it is worthwhile. The relevant information might be visible in a frame or appear in the words spoken during that interval.

I wanted to examine those two routes separately before combining them. A combined score can hide which channel contributed to a result. Cue keeps the result tied to a decoded frame, transcript text and channel ranks. The current implementation works locally with frame sequences and supplied timed transcripts, which gave me control over the retrieval experiment without requiring a recording service.

Asking for a blue rectangle

The supplied example begins with the query blue rectangle. In fixture mode, the result points to clip-10 between zero and three seconds. The visual cosine is 1; text contributes no evidence. Its reciprocal-rank score comes from the first visual rank, giving 1/61.

I show the decoded frame at the start of the interval, with the source identifier and timestamps above it. The evidence panel retains the scores and channel contributions. I can ask whether the picture supplied the match or a transcript phrase brought the interval into the ranking. The displayed frame is one actual frame from the interval, while the segment representation summarizes its frames.

A creature aligning a film frame and paper strip at the same moment
I drew visual and text evidence meeting at the same moment.

Putting the result before the score

I kept the Streamlit search surface small. The query field sits above controls for corpus split, retrieval method and result count. A search uses the saved local index. Each hit has its rank, video identifier and interval, followed by the image and transcript. More detailed ranking evidence stays available in an expander.

The interface also names the active backend and data provenance. This matters when the same query is tried with fixture descriptors and the optional model. Their supported meanings and rejection behavior differ, even though the controls are the same. I display recorded evaluation in a separate panel. A person can inspect a current search without confusing it with the earlier experiment, then open that experiment when they want the comparison and its sample counts.

Application screens

1 / 3
Fixture visual search

I opened the blue rectangle result with its source and interval in the generated frame sequence.

Open full-size image

Giving the recording a time structure

I turn each input sequence into three-second windows beginning every second, clamping the last windows to the clip duration. Intervals are half open, so a boundary belongs to the next interval rather than both. A transcript span enters a window when their time ranges overlap positively, and its words retain their chronological order.

The importer checks source IDs and group splits before indexing. It also checks frame decoding, dimensions and timing. A bad record therefore has a named failure before it becomes a search hit. I keep the source group with every overlapping segment, so windows from one clip cannot quietly migrate between training and test. Indexing and prediction use the input content; the gold relevant intervals are read only by evaluation.

A visual baseline with a limited vocabulary

I started the visual path with descriptors I could inspect directly. They measure color-mask areas and simple shape properties in the frames, then average those values across a window. The query grammar supports a small set of color and shape words, including blue and rectangle. A query outside that vocabulary can return no supported visual evidence.

This baseline makes its assumptions visible. The circle coordinate is a bounding-box-fill proxy, so it can also respond to other shapes with similar fill. Words such as in or on do not create spatial relationships. I use these descriptors to establish whether the content reaches the correct time intervals and to provide a numerical reference for a broader visual representation.

Keeping transcript fitting on the training side

For words I use TF-IDF fitted only on the 36 training segments. The resulting vocabulary and inverse document frequencies then transform all 72 segments without another fit. This gives transcript retrieval a simple, inspectable basis and keeps held-out text from influencing its preparation.

That choice has a visible cost. Project identifiers appearing only in held-out clips are absent from the vocabulary. Shared closing or logo words can then tie across several sources, with deterministic ordering deciding which appears first. I retain this behavior in the comparison. Adding held-out identifiers after seeing the mistake would change the experiment's meaning. The text channel searches supplied transcripts; speech recognition and audio extraction are outside the implemented input path.

Changing the visual representation

I added LAION CLIP ViT-B/32 with matched image/text towers to examine a broader visual search space. Frame vectors are normalized, averaged inside the existing window and normalized again. The query uses the matching text tower. I kept the windows and transcript method unchanged, so the comparison has a clear change in representation.

The adapter runs from verified local assets on CPU. Missing or incompatible assets produce an error instead of starting a download during inference. The saved index records backend and preprocessing identities as well as input hashes. When its source content changes, Cue asks for a new index. The checkpoint uses native English tokenization with 77-token truncation. Turkish retrieval needs separate evaluation.

Combining ranks without mixing score scales

I combine the channels through reciprocal-rank fusion with a constant of 60. A first-place result contributes 1/61, a second-place result 1/62. The contributions are added when both channels return the same segment. Raw cosine values remain separate, because a visual score and a transcript score do not automatically share a meaningful scale.

After fusion I suppress directly overlapping intervals from the same source before choosing the requested number of hits. That keeps the output manageable, although adjacent intervals can still describe one moment. I expose the contributions so I can distinguish two outcomes: covering a query through one useful channel, and having both channels support one returned interval. The evaluation checks whether either produces a better ranking.

Keeping the comparison on a common corpus

I generated 12 six-second clips as 144 PNGs with timed transcripts. The source groups divide into six training, three validation and three test clips. There are 27 queries and 30 relevant-interval annotations overall. The test comparison has six answerable queries and one unknown, evaluated separately. Repeated scenes and only four distinct frame contents make this a small integration fixture.

The paired descriptor and model runs are recorded exploratory comparisons on already known queries. I kept the eligible corpus and retrieval policy shared, then measured separate channels and fusion. A relevant result needs the correct source and temporal overlap with the annotated interval. One-to-one matching prevents several returned windows from receiving repeated credit for the same annotated moment.

Where the second channel helped

In the recorded test, fixture visual retrieval found three of the six answerable queries in its top three. Text found two. Their fused ranking found five, with MRR 0.755556. No retained fixture hit received both contributions, so that improvement came from covering different query types.

CLIP visual and its fused ranking both found all six in the top three, with the same MRR of 0.805556. Fusion moved one speech result from third to second and another from second to third. I kept that mixed outcome visible. Across 20 warmed operations, median retrieval was 0.5842 ms for fixture fusion and 72.0735 ms for model fusion. Those timings exclude UI and initialization; the interface creates a new searcher for each request, so interactive delay can include loading again.

Keeping the unknown and duplicate moments visible

The unknown query purple elephant in snow exposes an important model failure. Although the fixture has no relevant moment, model visual retrieval returns positive similarity and fusion keeps results. Both have one false positive out of one unknown. A positive cosine therefore cannot serve as a reliable answerability decision.

I also retained the closing-moment error. Adjacent intervals [2,5) and [5,6) can both match the same closing scene while surviving overlap suppression. They occupy result slots before a relevant clip appears at rank five. Separately, a stale index or corrupt frame produces a named preparation error, with guidance to rebuild the selected backend. Optional malformed evaluation reports warn without disabling search. Empty evidence and broken preparation remain different states in the interface.

What I would change before a broader evaluation

Cue helped me separate coverage from agreement and ranking improvement. I can inspect a returned moment and follow the contribution that brought it there. The saved CLI and UI share that path, and index identity checks keep a changed input from borrowing an old representation without notice.

My next evaluation would use independently licensed real clips and a larger held-out query set, including conflicting evidence and deliberately unknown scenes. I would develop moment grouping and rejection calibration separately before scoring untouched examples. The current project has no executed MP4 decoder, ASR or motion-analysis path. Its local frame-sequence workflow gives me a place to test those additions later while preserving the original reports and keeping each new experiment's source and configuration identifiable.

Available for work