[squid-users] FATAL: xcalloc: Unable to allocate 1 blocks of 4112 bytes!

[Pranav Desai]

Hello,

I am doing some performance testing on FC5 with squid 2.6-S2

I am getting the following error when I use cache_mem 4 GB

FATAL: xcalloc: Unable to allocate 1 blocks of 4112 bytes!
Squid Cache (Version 2.6.STABLE2): Terminated abnormally.

And it restarts after that, thereby completely killing the test.

I have 16GB of memory.

Is there a better way to ask squid to use as much memory as possible,
since I have ample of it.

Thanks

-- Pranav

/proc/meminfo
--------------------
MemTotal: 16378848 kB
MemFree: 6608296 kB
Buffers: 398928 kB
Cached: 8159824 kB
SwapCached: 0 kB

Version
-----------
Squid Cache: Version 2.6.STABLE2
configure options: '--prefix=/usr/squid' '--exec-prefix=/usr/squid'
'--sysconfdir=/usr/squid/etc' '--enable-snmp'
'--enable-err-languages=English' '--enable-linux-netfilter'
'--enable-dlmalloc' '--enable-async-io=24' '--enable-linux-tproxy'
'--enable-gnuregex' '--enable-internal-dns' '--enable-epoll'
'--with-maxfd=32768' 'CC=/usr/local/bin/gcc'
'CFLAGS=-I/usr/local/include -g'


-- Pranav

------------------------------
http://pd.dnsalias.org

[Pranav Desai]

Forgot to mention, I am testing at 700 req/s on polymix-4.

--Pranav

Post by Pranav Desai
Hello,
I am doing some performance testing on FC5 with squid 2.6-S2
I am getting the following error when I use cache_mem 4 GB
FATAL: xcalloc: Unable to allocate 1 blocks of 4112 bytes!
Squid Cache (Version 2.6.STABLE2): Terminated abnormally.
And it restarts after that, thereby completely killing the test.
I have 16GB of memory.
Is there a better way to ask squid to use as much memory as possible,
since I have ample of it.
Thanks
-- Pranav
/proc/meminfo
--------------------
MemTotal: 16378848 kB
MemFree: 6608296 kB
Buffers: 398928 kB
Cached: 8159824 kB
SwapCached: 0 kB
Version
-----------
Squid Cache: Version 2.6.STABLE2
configure options: '--prefix=/usr/squid' '--exec-prefix=/usr/squid'
'--sysconfdir=/usr/squid/etc' '--enable-snmp'
'--enable-err-languages=English' '--enable-linux-netfilter'
'--enable-dlmalloc' '--enable-async-io=24' '--enable-linux-tproxy'
'--enable-gnuregex' '--enable-internal-dns' '--enable-epoll'
'--with-maxfd=32768' 'CC=/usr/local/bin/gcc'
'CFLAGS=-I/usr/local/include -g'
-- Pranav
------------------------------
http://pd.dnsalias.org

[Adrian Chadd]

Post by Pranav Desai
Hello,
I am doing some performance testing on FC5 with squid 2.6-S2
I am getting the following error when I use cache_mem 4 GB
FATAL: xcalloc: Unable to allocate 1 blocks of 4112 bytes!
Squid Cache (Version 2.6.STABLE2): Terminated abnormally.

It sounds like you've hit a process limit.

Try 'ulimit -a' to see whats going on.




Adrian

[Pranav Desai]

Post by Adrian Chadd

Post by Pranav Desai
Hello,
I am doing some performance testing on FC5 with squid 2.6-S2
I am getting the following error when I use cache_mem 4 GB
FATAL: xcalloc: Unable to allocate 1 blocks of 4112 bytes!
Squid Cache (Version 2.6.STABLE2): Terminated abnormally.

It sounds like you've hit a process limit.
Try 'ulimit -a' to see whats going on.

I checked it, it looks fine.

cat /proc/sys/fs/file-nr
9210 0 32768

ulimit -a
----------
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 131072
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 131072
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

when I run it without cache_mem (i.e. default value), it runs fine,
without crashing at the same load.

Let me know if you need any other details.

-- Pranav

Post by Adrian Chadd
Adrian

[Adrian Chadd]

Post by Pranav Desai
Hello,
I am doing some performance testing on FC5 with squid 2.6-S2
I am getting the following error when I use cache_mem 4 GB

Ah - which platform are you running this on? Is it being compiled
in proper 64 bit mode?

I've never run squid with 4 gigabytes of RAM for cache_mem
and i've never had a computer with enough memory to let me do so. :)



adrian

[Pranav Desai]

Post by Adrian Chadd

Post by Pranav Desai
Hello,
I am doing some performance testing on FC5 with squid 2.6-S2
I am getting the following error when I use cache_mem 4 GB

Ah - which platform are you running this on? Is it being compiled
in proper 64 bit mode?

How would I check if its being compiled correctly or not? I think it is.

#include <stdio.h>
main()
{
printf("Sizeof pointer = %d\n", sizeof(*int));
}
It should say 8 bytes for a 64 bit platform and 4 bytes for a 32 bit platform.

Post by Adrian Chadd
I've never run squid with 4 gigabytes of RAM for cache_mem
and i've never had a computer with enough memory to let me do so. :)

Same here, this is first time I have ever seen 16GB of memory in a machine
:-)

Heh. I've got access to a solaris machine with >4 gig of RAM I could
try running Squid on. This'll tell me whether there's any reason Squid
can't grow its cache_mem usage past four gigabytes.

Did you have any luck with higher value of cache_mem ?

On further testing it seems like the process fails when it hits ~1GB
limit (in top). I read in the FAQ that the msgs could be due lack of
swap space or per process data seg size limit.
I have checked both using and neither seems to be the case.

The other thing that the FAQ mentioned was that there is limit set by
the kernel also. I am not sure how to change that. Does anyone know
how to do that ?

Part of my ulimit output

core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 131072
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 32768
pipe size (512 bytes, -p) 8

It is set to unlimited and my swap space is about 750MB free.

line from top command.

Swap: 1959888k total, 1200580k used, 759308k free, 6514844k cached

Thanks

-- Pranav

Adrian

[Henrik Nordstrom]

Post by Pranav Desai
On further testing it seems like the process fails when it hits ~1GB
limit (in top). I read in the FAQ that the msgs could be due lack of
swap space or per process data seg size limit.
I have checked both using and neither seems to be the case.

Is your Squid compiled 64-bit or 32-bit?

file /path/to/sbin/squid

32-bit apps have all sorts of architectural size limitations. Exact
limits varies with the OS and OS version..


Regards
Henrik

[Pranav Desai]

Post by Henrik Nordstrom

Post by Pranav Desai
On further testing it seems like the process fails when it hits ~1GB
limit (in top). I read in the FAQ that the msgs could be due lack of
swap space or per process data seg size limit.
I have checked both using and neither seems to be the case.

Is your Squid compiled 64-bit or 32-bit?

64-bit.

Post by Henrik Nordstrom
file /path/to/sbin/squid

[***@pdesai:/usr/squid]$ file ./sbin/squid
./sbin/squid: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV),
for GNU/Linux 2.4.0, dynamically linked (uses shared libs), not
stripped

[***@pdesai:/usr/squid]$ uname -a
Linux pdesai 2.6.14newkernel64 #5 SMP Fri Mar 3 15:12:25 PST 2006 x86_64 unknown

Thanks

-- Pranav

Post by Henrik Nordstrom
32-bit apps have all sorts of architectural size limitations. Exact
limits varies with the OS and OS version..
Regards
Henrik

[Henrik Nordstrom]

Post by Pranav Desai
64-bit.

Then I have no idea. Have 64-bit Squid's being many GB in size..

Regards
Henrik

[Pranav Desai]

Post by Henrik Nordstrom

Post by Pranav Desai
64-bit.

Then I have no idea. Have 64-bit Squid's being many GB in size..

Ok thanks. Let me try to find some more information.

-- Pranav

Post by Henrik Nordstrom
Regards
Henrik

[Pranav Desai]

Post by Pranav Desai

Post by Henrik Nordstrom

Post by Pranav Desai
64-bit.

Then I have no idea. Have 64-bit Squid's being many GB in size..

Ok thanks. Let me try to find some more information.

I just tried a small calloc loop to find out if there is any system
imposed limitation. This program can increase in mem size much higher
that 1G, at which squid seems to fail.

while (1) {
fprintf(stdout, "allocating %d\n", i++);
if ((p = calloc(1, atoi(argv[1]))) == NULL) {
fprintf(stdout, "calloc failed ... %d\n", i);
perror("calloc");
return -1;
}
usleep(atoi(argv[2])*1000);
}

From the error string and related code in utils.c it looks like that
calloc is failing. Can someone help me debug this further. I dont know
how to debug things like malloc, so I kindof stuck here.

The other thing I found is in the strace. Last brk() before it failed.
It seems like some 24bit limit. Any ideas ?

brk(0x40000000) = 0x3ffff000
socket(PF_FILE, SOCK_DGRAM, 0) = 13
fcntl(13, F_SETFD, FD_CLOEXEC) = 0
connect(13, {sa_family=AF_FILE, path="/dev/log"}, 16) = 0
sendto(13, "out of memory [1785]", 20, MSG_NOSIGNAL, NULL, 0) = 20
write(4, "FATAL: xmalloc: Unable to alloca"..., 105) = 105

I would appreciate any help I can get ...

Thanks

-- Pranav

Post by Pranav Desai
-- Pranav

Post by Henrik Nordstrom
Regards
Henrik

--
------------------------------
http://pd.dnsalias.org

[Henrik Nordstrom]

Post by Pranav Desai
The other thing I found is in the strace. Last brk() before it failed.
It seems like some 24bit limit. Any ideas ?

I have only seen such limits on 32-bit systems. Never on an 64-bit
system running 64-bit software.

So triple-check that ulimit is set like you expect for the Squid
process. Remember that ulimit is per process so only because your shell
says something large does not neccesarily mean that Squid is running
under the same ulimit.

The Squid I am currently watching is now close to 4GB in size.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20701 squid 15 0 3931m 3.8g 1532 S 4 32.4 528:02.21 squid

and still climbing (cache not full yet).

This is on a stock Ubuntu dapper x86_64 with a custom Squid-2.6.STABLE3
build (some Squid patches, none relevant to memory limits).

Regards
Henrik

[Pranav Desai]

Post by Henrik Nordstrom

Post by Pranav Desai
The other thing I found is in the strace. Last brk() before it failed.
It seems like some 24bit limit. Any ideas ?

I have only seen such limits on 32-bit systems. Never on an 64-bit
system running 64-bit software.
So triple-check that ulimit is set like you expect for the Squid
process. Remember that ulimit is per process so only because your shell
says something large does not neccesarily mean that Squid is running
under the same ulimit.

Interestingly, disabling dlmalloc solved all the problems. Now the
process size can grow in excess of 2 GB.

Other than maybe some performance gain, do I lose anything by
disabling dlmalloc ?

Thanks for all your input.

-- Pranav

Post by Henrik Nordstrom
The Squid I am currently watching is now close to 4GB in size.
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20701 squid 15 0 3931m 3.8g 1532 S 4 32.4 528:02.21 squid
and still climbing (cache not full yet).
This is on a stock Ubuntu dapper x86_64 with a custom Squid-2.6.STABLE3
build (some Squid patches, none relevant to memory limits).
Regards
Henrik

[Henrik Nordstrom]

Post by Pranav Desai
Interestingly, disabling dlmalloc solved all the problems. Now the
process size can grow in excess of 2 GB.

Right..

Post by Pranav Desai
Other than maybe some performance gain, do I lose anything by
disabling dlmalloc ?

Rule of thumb: Don't use configure options you don't know you need.

dlmalloc is shipped with Squid for the unfortunate ones where the OS
provided malloc is real crap. You'll know if you are one of those
unfortunate from Squid growing and growing in size even if the allocated
memory stays the same ("Free memory" increasing a lot over time in
cachemgr general runtime info).

Regards
Henrik