# Handle images, speech, and video inside an application

Canonical page: https://yishto.com/en/learn/ai-foundations/multimodal-image-and-speech/

Author: [Yishto](https://yishto.com/en/about/)

Separate teaching-media processing into modality tasks, input checks, timing, assets, and review, with explicit evidence limits for sampling, generation, and recognition.

Published: 2026-11-10
Updated: 2026-11-10

A teaching-material assistant accepts a recording, images, and a short video to create readable notes. Additional modalities also add format checks, alignment, jobs, asset storage, and review responsibilities. Fluent output alone does not establish that the source information was preserved.

Two synthetic subtitle segments, hello and world, demonstrate timing checks. The example performs no recognition or model call. It focuses on observable stages between source media and published notes.

## Separate recognition, conversion, and generation

Vision recognition extracts information from images; generation creates images from conditions. ASR transcribes observed sound, while TTS synthesizes speech from text. Translation, speaker separation, and timestamps are independent outputs that can fail even when transcription succeeds.

Specify terminology preservation, source-time navigation, the purpose of generated illustrations, and review for uncertain material. Do not infer speaker identities when unnecessary. Mark summaries as omitting detail instead of presenting them as complete transcripts.

The Whisper paper concerns speech processing and transcription, not current TTS interfaces. Provider formats, limits, parameters, and output types require their own documentation; one task's example does not establish another capability.

## Validate actual media inputs

Check permitted formats, size, duration, resolution, and processing rights. A request's MIME declaration does not prove its content type; use trusted parsing too. Mismatched encoding, damaged media, and unsupported conditions need explicit errors.

Sources can contain voices, faces, and background information. Keep only necessary assets and define storage, access, and retention. Deleting originals does not automatically remove subtitles, thumbnails, or notes, so include derived assets in lifecycle rules.

Preprocessing changes information. Resizing can obscure small text, audio conversion can lose detail, and sampled video frames can miss brief events. Record transformations and time mapping so reviewers know what the model actually received.

## Distinguish mechanisms from interfaces

A class of diffusion models learns recovery from noisy representations and samples iteratively. Latent diffusion operates in a compressed representation. These mechanisms do not establish that every image API uses the same algorithm or step count.

Generated images may follow a prompt while containing incorrect text, counts, structures, or details. Check their relationship to the lesson, not only appearance. Recognition needs checks for small text, table order, and omissions. Generation and recognition require different acceptance rules.

OpenAI image documentation distinguishes generation and editing across interfaces. Audio documentation distinguishes file and live transcription, translation, and speech generation. This lesson uses those capability categories without unexecuted SDK instructions or claims about account access.

## State the alignment assumptions

Subtitle boundaries must fit inside the media duration. This example assumes one ordered, nonoverlapping text track. Simultaneous speakers may legitimately overlap and need a different contract.

```python
segments = [{"start": 0.0, "end": 1.2, "text": "hello"},
            {"start": 1.2, "end": 2.8, "text": "world"}]
def valid(items, duration):
    previous = 0.0
    for item in items:
        if not (previous <= item["start"] < item["end"] <= duration):
            return False
        previous = item["end"]
    return True
assert valid(segments, 3.0)
assert not valid(segments, 2.0)
print(valid(segments, 3.0))
```

The actual output is `True`. Segments cover 0 to 1.2 and 1.2 to 2.8 seconds within three seconds. A two-second source is rejected because the second segment exceeds it. Gaps are permitted, so validity does not prove complete coverage or matching speech.

Also validate units, finite numbers, required fields, and empty text. Valid segment boundaries do not establish accurate word timing. Review source audio alongside subtitles and preserve original-media offsets when processing long recordings in pieces.

## Give jobs and assets recoverable states

Large media may need queued, processing, review_required, succeeded, and failed states with a returned task ID. Confirm durable output storage before marking success, or users may receive a successful task with missing files.

Reuse task keys during retries and distinguish temporary service errors from unsupported files. Avoid infinite retries for damaged input. Record asset versions, transformations, model settings, and output identities to distinguish reused results from new assets.

Cancellation has limits: work already sent to an external service may not stop immediately. Late output must not revive a cancelled task. Cleanup should handle orphaned assets while retaining necessary audit records, without exposing source URLs or personal content in logs.

## Review specific error types

Noise, accents, overlapping speech, and specialized terms can cause transcription errors. Silence may also produce unwanted text. A sampled video that contains no observed event does not prove the full video lacks that event.

Model-reported confidence needs evaluation against actual errors, and scores across tasks are not interchangeable. Useful review triggers include out-of-range times, abnormal repetition, terminology conflicts, inconsistent summaries, and insufficient frame evidence.

Reviewers need source times, candidate content, and uncertainty markers. Version edits and distinguish generated assets from originals. Voice synthesis and depictions of people also require appropriate authorization and disclosure for the scenario.

## Exercise: validate the processing chain

Design states for input, preprocessing, transcription, summary, and review for three seconds of media. Add insufficient duration, overlapping segments, empty text, late results, and output after cancellation. Define rejection and review paths.

Accept the design when timestamps stay in bounds, offsets map back to sources, successful jobs have existing assets, invalid inputs do not retry forever, and review retains provenance. The timing function checks one-track ordering only. Recognition quality, image meaning, video coverage, and real interface execution need separate evidence.

## Sources & further reading

- [Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://arxiv.org/abs/2212.04356)
- [Image generation APIs](https://developers.openai.com/api/docs/guides/image-generation)
- [Audio and speech APIs](https://developers.openai.com/api/docs/guides/audio)

Sources checked on 2026-11-10. Examples are teaching data, not production measurements.

## Related guides

- [Project: a teaching-illustration job queue and asset library](https://yishto.com/en/learn/project-labs/lab-illustration-job-queue/)
- [Isolate file and interface privileges in an Electron tool](https://yishto.com/en/learn/application-platforms/electron-transcript-workbench/)