Run Microsoft's Florence-2 vision model live on video with NVIDIA DeepStream. It does object detection, OCR, captioning, dense-region caption, region proposal and phrase grounding, and you can switch between them on the fly while the stream is playing. Output goes to an MP4 file, an RTSP stream, or an on-screen window.

Florence-2 isn't a one-shot detector. It writes its answer out one token at a time (it's an encoder–decoder), so the pipeline is split in two: Gst-nvinfer runs the image vision encoder, and the text-generation loop runs right after it on TensorRT with a KV cache for speed. You don't really need to know any of that to use it; just run the scripts below.

Each task is a --task flag (or a live command). Rough speed on a Tesla T4 (base model, fp16):

object detection	OCR	grounding
caption	dense region	per-class colors

• An NVIDIA GPU with a recent driver (DeepStream 9 is CUDA 13, so driver 580+ is recommended).
• Docker and the nvidia-container-toolkit, so containers can use the GPU.
• Python 3.10 or newer on the host.

Everything heavy (DeepStream, CUDA, TensorRT, the app itself) runs inside the Docker container, so nothing gets installed on your machine. The only thing that runs on the host is a small Python venv, used to download the model and prepare a few files.

git clone https://github.com/Vishnu-RM-2001/Florence-2-deepstream.git
cd Florence-2-deepstream

bash scripts/setup_venv.sh        # one time: host Python setup (.venv310)
bash scripts/00_get_model.sh      # download the Florence-2 model
bash scripts/get_test_videos.sh   # grab the demo clips
bash scripts/run.sh               # build everything + run (grounding by default)  ->  out_florence_base.mp4

The first run.sh does all the building (the Docker image, the TensorRT engines and the app) and caches it, so later runs start quickly. Pick a different task with the 3rd argument:

bash scripts/run.sh '' '' caption                                   # one-line caption
bash scripts/run.sh file:///work/data/ocr_open.mp4 '' ocr           # OCR on the OPEN sign
bash scripts/run.sh '' '' 'ground car, person, street sign, palm tree'   # grounding

Signature: run.sh [VIDEO] [OUTPUT_MP4] [TASK] [-- extra options]. Use '' to keep a default. Paths are container paths: the repo is mounted at /work, so /work/data/x.mp4 is ./data/x.mp4 on the host.

Pass the task name as the 3rd argument. The car clip (sample_qHD.mp4) is the default input; OCR uses the OPEN-sign clip. Each run writes an annotated MP4 (e.g. out_florence_base.mp4).

bash scripts/run.sh '' '' od                      # object detection — boxes + class names
bash scripts/run.sh '' '' dense_region            # dense region caption — boxes + short labels
bash scripts/run.sh '' '' region_proposal         # region proposal — boxes, no labels
bash scripts/run.sh '' '' caption                 # caption — one short line
bash scripts/run.sh '' '' detailed_caption        # detailed caption — a sentence or two
bash scripts/run.sh '' '' more_detailed_caption   # more detailed caption — a full paragraph
bash scripts/run.sh file:///work/data/ocr_open.mp4 '' ocr        # OCR — text + boxes
bash scripts/run.sh file:///work/data/ocr_open.mp4 '' ocr_text   # OCR — plain text, no boxes
bash scripts/run.sh '' '' 'ground a car, a person, a palm tree'   # grounding — a box per phrase

Use your own video as the 1st argument (file:///work/data/your.mp4), or an rtsp://… URL.

While a run is playing, open another shell and send a command. It switches over straight away:

bash scripts/florence_cmd.sh od
bash scripts/florence_cmd.sh ocr
bash scripts/florence_cmd.sh caption
bash scripts/florence_cmd.sh dense_region
bash scripts/florence_cmd.sh "ground the dog"     # grounding takes any phrase you type

# MP4 file (the default):
bash scripts/run.sh '' out.mp4 od

# RTSP — watch live from another machine (low latency):
bash scripts/run.sh '' '' od -- --sink rtsp
ffplay rtsp://<host-ip>:8554/florence

# On-screen window (needs a display), looping the clip forever:
bash scripts/run.sh '' '' od -- --sink display --loop

Stopping a run with Ctrl-C finalizes any MP4 it was writing, so recordings stay playable even when looping.

Running Florence on every frame makes a clip play in slow motion, since the model is the bottleneck. Add --infer-interval 2 to run it on every 3rd frame and reuse the last boxes in between; the video then plays close to real time while detections still refresh several times a second.

Add these after a -- in run.sh (e.g. bash scripts/run.sh '' '' od -- --max-area 0.6):

• --classes "car, person, traffic light" keeps only the labels you care about.
• --max-area <f> drops boxes bigger than f of the frame. Defaults to 0.95.
• --infer-interval <N> runs Florence every N+1 frames for smoother video. Default 0 is every frame.
• --loop restarts a file clip when it ends (handy for the display/rtsp live view).
• --kafka also streams detections as JSON to Kafka (see below).

For the full list, run build/florence-app --help.

Add --kafka to any run to also publish each detection as JSON (label, bbox, confidence, timestamp) to a Kafka topic. The video output keeps working at the same time.

# 1) start a local test broker (KRaft, no ZooKeeper) -> localhost:9092, topic 'florence-detections'
bash scripts/kafka_up.sh up

# 2) run any task with --kafka  (here detection-only via --sink fake; combine with rtsp/file freely)
bash scripts/run.sh '' '' od -- --sink fake --kafka

# 3) in another shell, watch the messages stream in while step 2 runs
bash scripts/kafka_up.sh consume

# 4) stop the broker when done
bash scripts/kafka_up.sh down

To point at a real broker, pass --kafka-conn host;port and --kafka-topic <topic>.

MODEL=large bash scripts/run.sh '' '' 'ground street sign' switches to Florence-2-large, which is more accurate but slower. The build uses bf16 for the large decoder automatically (fp16 overflows it) on Ampere and newer GPUs; on an older Turing T4, add DEC_PREC=fp32.