Slow and unusable live stream and trying to view recordings

[Mark Simpson]

Video quality now set to medium and frame rate to max on all cameras. It's like a different system now. Previously the menus in the web interface could take 30 seconds to become responsive.. now it's instant again. Thank you so much for the help.

The hit on the viewing and playback performance of the tweaked encoding to save space was the key. We didn't need the extra retention so turning this off saved our bacon it seems.

 

[Digilec]

Great News!

 

[Magic919]

Good to see this is sorted. Much bigger system than I’ve ever seen.

I presume the web clients are watching the feeds on the NVR and this is producing extra load.

 

[Mark Simpson]

The strain on the NVR when viewing images is the same with client or web browser I imagine. Although with the client it's possible to drag all of the cameras into the thumbnail view at once. That works but slows down pretty quickly.
Odd though that even navigating the configuration screens in the web interface was painfully slow with video compression set to high, even when no one was viewing any video. I guess that was the effect of the NVR compressing all the streams in the background. We have been experiencing some drop out in the footage. We'll see now if relaxing the compression has resolved that.

 

[Magic919]

When you say video compression set to high, what do you mean? There are a few video settings and none of these explicitly state compression.

I don't think the NVR is compressing the camera streams (when nobody is watching) and it is merely writing them to disk. There is a chance it is unwrapping the streams and recording the essence, but this is low load.

I expect that Hikvision have some people that have a deep understanding of how the NVR system functions. I'm presuming they write it themselves. It's a shame you don't have some access to those. At work we'd usually raise a ticket with a vendor, their support guys would escalate and sometimes even the developers would get involved.

In terms of strain on the NVR, we can only really see CPU, perhaps memory. Without knowing more about the processes, number of instances running and other performance-related metrics it's all guesswork. They ought to have a button that will collate and compress a support package that captures this. You'd then upload and they would investigate. I'd expect this on the higher end NVR like you have, not my little one at home.

 

[Mark Simpson]

The + indicates a compression standard supported by the camera. It's a huge amount of compression that was designed by Hikvision themselves I believe, saving a fantastic amount of disk space.

See here:
Hikvision’s H.265+ Encoding Technology

Thinking back to recent footage with dropout.. The dropout occurred when there was a lot of movement. We lost about a minute at a crucial time.

The compression clearly doesn't happen at the camera, as it's a software setting. Therefore the work is being done by the NVR.

 

[Magic919]

So you mean you have switched from H.265+ to H.265?

I am familiar with HEVC encoding and have read their white paper on the topic. I don't think they have done anything revolutionary as they claim to have stuck with the HEVC standards. It is more likely they are optimising this for storage and that doesn't necessarily favour the clients.

Here is the document - https://www.hikvision.com/upload/20161209211521855.pdf

Their strategy is best suited to largely static scenes with the odd moving object that traverses the scene. A lot of the effort goes into efficiently encoding the static background and minimising repeats of that data.

As you can see they have done work to push the I frame repetition rate into the 8-12 second range. This, like any long GOP technology, is great for storage, but not for clients due to start up delays. They have offset this by using an R (refresh) frame to act as an I frame for the clients. I think this bit is Hikvision specific as it's not terminology I've seen elsewhere.

They have put in some work regarding bitrate allocation and you can see they assess it over an unusually long period. I expect they virtually eliminate any stream stuffing by allowing a true VBR output.

 

[Mark Simpson]

Reading around it seems that the bottleneck is the processing power of the NVR. Exaggerated of course with compression set to maximum. I'm going to advise our operators to view footage during quiet times when the load will be at its least.
Not good news for our sports centre who just want a constant live view. I'm going to have to restrict the number of streams available to them.

I've switched from H265+ to H264.

You need much beefier hardware to run H265+. Unfortunately the model we bought isn't up to the job.

 

[Magic919]

For one source to many viewers it’s better to multicast if your network supports it. Ideally have a separate address per camera. Then the viewer can access the multicast that suits them.

Essentially one instance of stream comes from each camera to the first switch. Any clients perform an IGMP join to access the particular stream. No NVR overhead and network traffic only increases after the local access switches.

If that is not possible, then use a streaming server in between, like a proxy. Hikvision make one. I use Zoneminder and it picks up a single copy of each camera stream. Even if 20 people watch, there is only a single stream load on the NVR.

H.264 remains the best codec for viewers. Still too early days for H.265/HEVC in my opinion.

 

[Mark Simpson]

Hmm that's an idea. But the cameras are on a separate network hosted by the NVR. I notice though that the individual links to the cameras are via ports on the NVR IP address.
It's easier for the sports centre to use the iVMS-4500 client software. I can set them up a view and there's no fiddling about for them like with the web view. Can you recommend an alternative?

 

[Digilec]

I Would say that the cameras are capturing, processing and compressing the video to deliver the stream (H264 or H265) to whichever client (PC or NVR) is requesting it.

Although it’s a software setting, because it’s all Hikvision software/protocol the camera settings will be dictated by the NVR/VMS.

I would definitely agree that the NVR is lacking in performance and hikvision hardware datasheets Can be quite ambiguous.

For me Axis is the only Choice, especially for a Sytem of this size.
We can view and download a full server report from the cameras should there be any issues, we have all the specs available such as RAM memory, Flash and Processor.
Professional grade servers for storage such as the axis S1048 MKII.

We have axis chat support available 24 hours a day 5 days a week if needed and they are always extremely helpful and really know their stuff.

Sure, it comes at a cost compared to hikvision but in terms of what you get from Axis - quality, proven reliability, simplicity and in my opinion, best support available. I wouldn’t say it’s expensive, only that hikvision is cheap in comparison and the price you pay is a lack in core performance.

 

[Magic919]

I wouldn’t normally multicast over Wi-fi so I’d suggest have a look at Hikvision Stream Media Server to take the load off the NVR.

 

[Mark Simpson]

I spoke too soon. The system has gotten unusably slow again this morning. It had been a little slower in the last few days but I was willing to give it the benefit of the doubt.

A reboot just now has brought everything back to life again.

Symptoms: in the web backend streams were slow to load a thumbnail view. When you double clicked to access a full screen view it'd take 10 seconds to come up.
Navigating to the configuration tab, the menu's took 15 seconds to be active/ for the page to fully load.
Viewing past footage is impossible.
I'm on the latest firmware on the NVR, web client and Windows client.

 

[Phil]

Hi Mark,
Something makes your NVR struggle!
Was it OK until the staff returned to school?
Have they opened lots of client sessions?
Can you see any measure of how hard the NVR is now working?
Do you have a direct connected monitor, and if so can you check used/available bandwidth?

 

[Mark Simpson]

Was it OK until the staff returned to school?
Umm.. we may have had an extended uptime but this is the 1st time since the improved config so I've no benchmark to measure against.

Have they opened lots of client sessions?
No. I told them about this and they only connect when needed.

Can you see any measure of how hard the NVR is now working?

Do you have a direct connected monitor, and if so can you check used/available bandwidth?
Did you mean this:
Net Send Idle: 256mbps
Net Receive Idle: 41mbps

What is this screen and why do we need it?? Record Schedule and capture are both on.. the record schedule is working as we need it.

 

[Digilec]

That looks to me like local sd storage settings.

 

[Mark Simpson]

I presume those are settings for image capture. What about this schedule tho??

 

[Digilec]

Is this the Nvr or one of the camera web interfaces?

If not onboard storage of a camera, it looks like your set up for continuous recording of still images.. JPEG at 3 second intervals and a pretty low resolution?

 

[Mark Simpson]

It's the NVR