Introduction

Above each of the sheep pens at my parents farm I've mounted fisheye cameras to provide visibility. However, fisheye cameras have quite a lot of distortion making it harder to see the edges of the pens. I'm currently using Shinobi to view the sheep pens, which relies strongly on many of the features offered by ffmpeg to efficiently restream the cameras. FFmpeg itself has options for fisheye correction, which I intend to use. alt text

Setup

To conduct the dewarping, I want to use the v360 filter offered in newer versions of ffmpeg (currently not in the Ubuntu 20.04 repos). As this is quite an intensive operation, requiring first decoding the RTSP h264 stream from the camera, then applying the filter and then re-encoding, I've opted to not do this on my Shinobi box so that its resources are dedicated to the basic streaming tasks. This means I cannot use the native Shinobi functionality to give flags to ffmpeg, but will have to restream from my dewarping instance back to Shinobi. For the restreaming, I use rtsp-simple-server, to which I can easily stream with ffmpeg and then consume it again with Shinobi, similar to the normal IP cams. I do no configuration on the rtsp server, just start it using ./rtsp-simple-server.

FFmpeg dewarp

The full command:

./ffmpeg -use_wallclock_as_timestamps 1 -analyzeduration 1000000 -probesize 1000000
-fflags +igndts -rtsp_transport tcp -i 'rtsp://<user>:<pw>@<ip>:554' -vf " fps=10,
v360=input=fisheye:ih_fov=179:iv_fov=179:output=e:roll=-57,crop=1820:1820,scale=2048:2048"
-g 20 -c:v libx264 -b:v 4500k -strict 2 -f rtsp -rtsp_transport tcp rtsp://localhost:8554/overhead

Going over each of the options, some of which I understand, some I don't -analyzeduration 1000000 -probesize 1000000: Allows ffmpeg to analyse the incoming rtsp stream to figure out codecs and such

-vf "fps=10,v360=input=fisheye:ih_fov=179:iv_fov=179:output=e:roll=-57,crop=1820:1820,scale=2048:2048 are all the chained video filters.
- fps=10, trying to limit the fps as the cpu can't keep up anyways.
- v360=input=fisheye:ih_fov=179:iv_fov=179: Actual fisheye correction, I limited the fov to 179 as it gave a bit less distorted images than true 180 by 180.
- output=e:roll=-57, rotate the fisheye output, this could be fixed in the physical world by just rotating the camera as well.
crop=1820:1820: fully image is 2048x2048, slightly cropping in as the edges are quite distorted anyways
scale=2048:2048: scaling back up to a nice round number
-g 20 -c:v libx264 -b:v 4500k -strict 2
- -g 20: I believe this helps with increasing the amount of keyframes
- -c:v libx264 CPU encoded h264, for compatibility reasons. mjpeg might be better here as I'm dealing with low framerate
- -b:v 4500k: Default bitrate is only 200k, my IP cameras do about 4000k, so I'm trying to maintain maximum detail as bandwidth is of little concern to me
- -strict 2: Strict following of standards, mostly used to hopefully maximise compatibilty with Shinobi for streaming over HLS and all the clients
-f rtsp -rtsp_transport tcp rtsp://localhost:8554/overhead: makes the output an rtsp stream, targeting the running rtsp server running on the same host

The resulting image: alt text

Problems

Main current issue is the performance. Directly taking in the native IP camera RTSP stream in Shinobi has negliable performance impact, but decoding, filtering and encoding with ffmpeg has considerable overhead. The server on which I've conducted this experimentation has several Nvidia graphics cards which I should be able to use with ffmpeg using -hwaccel cuda flags. However, I've not yet been able to properly install the drivers and CUDA toolkit needed to do this. https://trac.ffmpeg.org/wiki/HWAccelIntro

CPU Usage with the above ffmpeg command: alt text

References

https://ffmpeg.org/ffmpeg-filters.html#toc-crop
https://ffmpeg.org/ffmpeg-filters.html#toc-v360
Unofficial builds of ffmpeg
https://write.corbpie.com/ffmpeg-preset-comparison-x264-2019-encode-speed-and-file-size/
ffmpeg streaming guide