freeing mlocked memory in linux

I'm ruing caffe deep leaing library on a graphics card and the library does some fancy stuff like mmap'ing some huge files into memory and passing buffers from ram to graphics card back and forth. After some time I noticed quite significant memory usage when nothing heavy is ruing on the system (with no X server, something like 10 processes: getty, sshd, syslog-ng, bash, ...):

MemTotal:       24688288 kB
MemFree:        19112788 kB
MemAvailable:   19102240 kB
Buffers:            6632 kB
Cached:            14892 kB
SwapCached:            0 kB

To be noted, I drop caches using echo 3 > /proc/sys/vm/drop_caches. So, that is roughly 5GiB used for something. And a very close number is accounted here:

Active:          4658852 kB        <-- here
Inactive:           2312 kB 
Active(anon):    4644112 kB        <--- and here
Inactive(anon):      760 kB
Active(file):      14740 kB
Inactive(file):     1552 kB
Unevictable:        6352 kB
Mlocked:        17111149713616 kB  <-- that is also strange

How can I track origins of those anonymous buffers/caches? How do I free it?

And, there are no processes left to hold the locked memory. Why is it still present?