Access the raw frames#63
Conversation
Co-authored-by: Sarah Gershuni <sarah556726@gmail.com> Co-authored-by: Chani Orlinski <c9992946@gmail.com>
Co-authored-by: Sarah Gershuni <sarah556726@gmail.com> Co-authored-by: Chani Orlinski <c9992946@gmail.com>
Co-authored-by: Sarah Gershuni <sarah556726@gmail.com> Co-authored-by: Chani Orlinski <c9992946@gmail.com>
Sarah5567
force-pushed
the
access_the_raw_frames
branch
from
a95ddda to
5db7977
Compare
| print("Pipeline stopped") | ||
|
|
||
| if __name__ == "__main__": | ||
| main() No newline at end of file |
| return Gst.PadProbeReturn.OK | ||
|
|
||
| # Retrieve NvBufSurface from GPU memory | ||
| surface = pyds.get_nvds_buf_surface(hash(buf), 0) |
There was a problem hiding this comment.
Avoid using hash(buf) to access the surface. It’s unreliable across DeepStream versions and may cause invalid memory access.
There was a problem hiding this comment.
deepstream sample apps show that it's ok to use it:
https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/blob/9b27f02ffea46a3ded2ad26b3eea27ef3e2dfded/apps/deepstream-imagedata-multistream/deepstream_imagedata-multistream.py#L121
| surface = pyds.get_nvds_buf_surface(hash(buf), 0) | ||
|
|
||
| # Copy to CPU (RGBA format) | ||
| frame = np.array(surface, copy=True, order='C') |
There was a problem hiding this comment.
This may not return a valid NumPy array — the surface might be a pointer, not the raw frame data.
| # frame = (frame * 0.7).astype(np.uint8) | ||
|
|
||
| # Copy back to GPU | ||
| np.copyto(np.array(surface, copy=False, order='C'), frame) |
There was a problem hiding this comment.
Copying back to GPU may fail if the surface isn’t in matching format (RGBA).
There was a problem hiding this comment.
I don't know how that works internally, but since we were able to see the output frames manipulated, I suppose that worked properly
| elems["nvvideoconvert1"].link(elems["capsfilter"]) | ||
| sinkpad = elems["streammux"].get_request_pad("sink_0") | ||
| srcpad = elems["capsfilter"].get_static_pad("src") | ||
| srcpad.link(sinkpad) |
There was a problem hiding this comment.
streammux should receive raw video, not post-RGBA capsfilter. This link order may cause performance loss.
There was a problem hiding this comment.
I think that in this case, RGBA conversion before nvstreammux is intentional, since the pad probe requires RGBA frames for CPU-side access and NumPy processing. The small performance cost is acceptable for this workflow.
| def on_pad_added(src, new_pad): | ||
| sink_pad = elems["depay"].get_static_pad("sink") | ||
| if not sink_pad.is_linked(): | ||
| new_pad.link(sink_pad) |
There was a problem hiding this comment.
Missing pad type check — if RTSP includes audio, it’ll try linking it too and fail.
There was a problem hiding this comment.
In general, yes - but here we don't care about audio stream in video
|
|
||
| def main(): | ||
| Gst.init(None) | ||
| os.makedirs(OUTPUT_DIR, exist_ok=True) |
There was a problem hiding this comment.
Create the output directory before building the pipeline, otherwise multifilesink may not find it.
Co-authored-by: Sarah Gershuni <sarah556726@gmail.com> Co-authored-by: Chani Orlinski <c9992946@gmail.com>
Co-authored-by: Sarah Gershuni <sarah556726@gmail.com> Co-authored-by: Chani Orlinski <c9992946@gmail.com>
This reverts commit ce8089f.
3 participants
Description
This PR adds the ability to access and modify raw frames inside a DeepStream pipeline.
A pad probe is attached after the
nvvideoconvert(RGBA conversion) element, allowing direct access to GPU buffers viapyds.get_nvds_buf_surface.Key Features
np.copytoto write back modified data.nvinferor OSD).multifilesinkoutput.Files Added
src/access_raw_frames.py- main implementation.env/arm/pyds-1.1.10-py3-none-linux_aarch64.whl- for installing thepydslibrary, which is required for the code.docs/access_raw_frames.md- documentation describing mechanism and usage.closes #57