Posted by bennos on February 25, 2010 at 10:20am
Hi
Have found Gear6 www.gear6.com.
They provide hosted memcache with automatic scaling. Sounds interesting.
Does anybody know, how to do it with memcache?
bennos
Autoscale of memcache bins
Hi
Have found Gear6 www.gear6.com.
They provide hosted memcache with automatic scaling. Sounds interesting.
Does anybody know, how to do it with memcache?
bennos
Comments
You could install your
You could install your memcached on its own instance which could then be individually scalable from the rest of the stack.
what I mean, is an automatic
what I mean, is an automatic scaling on a server. Normaly you define a memcache size like 128 MB and if the instance is full, old objects are dropped.
with autoscale the instance will grow.
this does memcache not normaly
to get that effect you can
to get that effect you can scale an instance up, say from a small to a high ram instance .. and then back down again when load lowers, the tricky part will be your bootup scripts that will alter memcache's allotted ram. Or perhaps since memcache can be distributed you can add instances in sm increments as needed. Im no memcache guru tho.
well, I think you didn't read
well, I think you didn't read descriptions.
they provide key based API for memcached - this mean that you will need to implement cache by your self and can not use existing memcached option for cacheroter module
Memcached by itself need to be restarted to get more RAM(for site with high load total cache rebuild can be a problem) and memcached have not authentication option - so, they need to filter your usage by ip->port relation.
Your life will be much more easy if you will get VPS/VDS or dedicated with options to increase RAM limit.
From reading their site it
From reading their site it appears they are running their own, modified version of memcached:
http://www.gear6.com/memcached-product/100-memcached-compatible
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.
mercury's memcached
mercury's memcached implementation doesnt use cacherouter module it uses memcache module.
There are surely ways to scale memcached in aws .. gear6 that bennos mentioned in the original post above uses aws to do just that.
http://www.gear6.com/memcached-product/gear6-webcache-aws-ec2-getting-st...
My point was that if they are doing it so could you, but really they seem to have a pretty nice package actually, nice find bennos. They seem to be adding/removing instances as needed like I suspected, which enables altering ram without needing a reboot or do full cache rebuild. They also seem to have some added optimizations that maximize ram usage in the instances.
With aws you can go from instances with 1.7 GB memory up to 68.4 GB of memory, and can spin up as many as needed when needed.
yeah their runnig their own
yeah their runnig their own memcache version, but this was the question. Does anybody know a solution like the gear6 solution, were memcache grows automatic without a restart of memcached daemons?
Not that I have seen. It
Not that I have seen. It wouldn't be that trivial to add either given Memcached's slab memory management. To get an idea check this out:
http://code.google.com/p/memcached/wiki/FAQ#How_does_memcached%27s_memor...
Oh and I really like gear6 now. They are powered by Drupal :D
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.
They are adding more
They are adding more instances of memcached servers and linking them together on the fly. This means you dont have to restart memcached or rebuild the whole cache.
EDIT: You could conceivably do this with cloudwatch, loadbalancer & bootup scritps + maybe some tweaks to memcached but probably not much since memcached allows distributed cache natively.
That's how I figured they
That's how I figured they were doing it, but their documentation says that it doesn't require any changes to the client app, so my guess is they are running each bin your app connects to as a virtual bin and then their software automatically monitors your actual bins and adds them in automatically to your virtual one. So your app doesn't actually connect to memcached, but rather a proxy.
HollyIT - Grab the Netbeans Drupal Development Tool at GitHub.
Someone from Gear6
Hi!
I'm with Gear6.
To answer some of the confusion about what our memcached on EC2 product does:
A memcached cluster is a set of individual running memcached server daemon, usually one per machine. This set of nodes is organized into a distributed hash table, by taking the the lookup key for an item, hashing it, and using that hash to calculate which node that that item is stored in. Normally, the nodes are completely unaware of the HDT process.
What Gear6 has done is make the nodes aware of the DHT, and aware of their peers in the cluster. So if a client asks the "wrong" node for an item, that node can proxy the memcached request to the node where the data actually is.
One of the things this means is that you can add nodes to the cluster without having to instantly reconfigure all the clients with the new list of nodes at the same moment. Which makes it very easy to add nodes. In the EC2 environment, the Gear6 UI lets just say "add more nodes, expand the memcached cluster", and it will spin up additional EC2 instances and then add them, thus allowing "push button" "grow the memcached".
The individual memcached nodes do not grow. They are whatever size they are when they are started. EC2 has number of options for different sizes of instances. As they get larger, they cost more to run, but interestingly, for the 64 bit high memory instances, they actually cost less per gigabyte-month then the "cheaper" 32bit small instances.
Now, I talk about all this stuff in the present tense, which is not completely accurate. As of today (2010-02-25), the AMIs with the "Dynamic Services" feature have not yet been released, but they will be next week.
When it is, do try it out.
And if you need an "ordinary" memcached today, try out our current AMIs. They act like normal memcached nodes, except they are much less wasteful of memory (improvements on the slab allocator), and they have a nice REST and Web UI management interface.
You guys have an awesome
You guys have an awesome setup. I wish you the best of luck and will likely try it out someday soon too.
Thanks for sharing some of the details with us.
Cheers!
thx Mark
thx Mark
Jobsuche