...
 
Commits (75)
......@@ -4,9 +4,10 @@ node_js:
- "0.8"
- "0.10"
# Create a travis enabled environment for the test suite to run in.
script: "make travisci"
# Create a travis memcached enabled environment for the test suite to run in and
# ensure that we test against localhost on travis-ci
services: memcache
env: MEMCACHED__HOST=localhost
# the `sevices: memcache` will start a memcached service on localhost
# and on the default port, but in order to test against multiple memcache
......
### 0.2.5
- Fix for two bugs in Issuelog #137 and #141
- Add new `failuresTimeout` option
### 0.2.4
- Tons of fixes have been made to the way we do error handling and failover,
this includes better reconnect, server failure detection, timeout handling
and much more.
- Introduction of a new `idle` timeout option.
- Documentation improvements.
### 0.2.3
- Added documentation for public api's
- new namespace option added that namespaces all your keys.
- minor parser fixes and some thrown errors.
### 0.2.2
- Support for touch command #86
- Fix for chunked responses from the server #84
### 0.2.1
- Supports for a queued callback limit so it would crash the process when we queue
to much callbacks. #81
### 0.2.0
- [breaking] We are now returning Error instances instead of strings for errors
- Dependency bump for a critical bug in our connection pool.
### 0.1.5
- Don't execute callbacks multiple times if the connection fails
- Parser fix for handling server responses that contain Memcached Protocol
keywords
- Make sure that the retry option is set correctly
### 0.1.4
- Added missing error listener to the 3rd-Eden/jackpot module, this prevents crashes
when it's unable to connect to a server.
### 0.1.3
- Handle Memcached responses that contain no value.
- Travis CI integration.
### 0.1.2
- Returning an error when the Memcached server issues a `NOT_STORED` response.
### 0.1.1
- Now using 3rd-Eden/jackpot as connection pool, this should give a more stable
connection.
### 0.1.0
- Storing numeric values are now returned as numeric values, they are no
longer strings.
### 0.0.12
- Added `.setEncoding` to the connections, this way UTF-8 chars will not be
chopped in to multiple pieces and breaking binary stored data or UTF-8 text
### 0.0.11
- Added more useful error messages instead of returning false. Please note
that they are still strings instead of Error instances (legacy)
### 0.0.10
- Compatibility with Node.js 0.8
- Don't saturate the Node process by retrying to connect if pool is full #43
- Minor code formatting
### 0.0.9
- Code style refactor, named the functions, removed tabs
- Added Mocha test suite
ALL_TESTS = $(shell find test -name '*.test.js')
test:
@./node_modules/.bin/mocha $(ALL_TESTS)
travisci:
MEMCACHED__HOST=localhost $(MAKE) test
doc:
dox --title "node-memcached" lib/* > doc/index.html
.PHONY: test doc
# Memcached [![Build Status](https://secure.travis-ci.org/3rd-Eden/node-memcached.png?branch=master)](http://travis-ci.org/3rd-Eden/node-memcached)
`memcached` is a fully featured Memcached client for Node.js. `memcached` is
build with scaling, high availability and exceptional performance in mind. We
built with scaling, high availability and exceptional performance in mind. We
use consistent hashing to store the data across different nodes. Consistent
hashing is a scheme that provides a hash table functionality in a way that
adding or removing a server node does not significantly change the mapping of
......@@ -18,18 +18,16 @@ When these issues occur the `memcached` client will emit different events where
you can subscribe to containing detailed information about the issues.
The client is configurable on different levels. There's a global configuration
that you update so all you Memcached clusters will use the same failure
that you update so all your Memcached clusters will use the same failure
configuration for example, but it's also possible to overwrite these changes per
`memcached` instance.
### protocol
This module uses the ASCII protocol to communicate with the server, this makes
it easier to debug for you are user as you can see what is send over the wire
but also for me as developer. But this also means that SASL auth is not
supported in this driver as that requires the use of the binary protocol. The
ASCII protocol not only used by memcached but also by other databases and
message queues, so that is a nice extra.
As in other databases and message queues, this module uses the ASCII protocol to communicate with
the server, which means that you can see what is send over the wire. For debugging this is easier
for both the users and the developers however this also means that SASL auth is not supported
because it demands the binary protocol.
## Setting up the client
......@@ -47,74 +45,53 @@ The server locations is designed to work with different formats. These formats
are all internally parsed to the correct format so our consistent hashing scheme
can work with it. You can either use:
1. **String**, this only works if you have are running a single server instance
1. **String**, this only works if you are running a single server instance
of Memcached. It's as easy a suppling a string in the following format:
`hostname:port`. For example `192.168.0.102:11212` This would tell the client
to connect to host `192.168.0.102` on port number `11212`.
`hostname:port`. For example `192.168.0.102:11211` This would tell the client
to connect to host `192.168.0.102` on port number `11211`.
2. **Array**, if you are running a single server you would only have to supply
one item in the array. The array format is particularly useful if you are
running a cluster of Memcached servers. This will allow you to spread the keys
and load between the different servers. Giving you higher availability for
and load between the different servers. Giving you higher availability
when one of your Memcached servers goes down.
3. **Object**, when you are running a cluster of Memcached servers it could
happen to not all server can allocate the same amount of memory. You might
have a Memcached server with 128mb, 512, 128mb. If you would the array
structure all servers would have the same weight in the consistent hashing
scheme. Spreading the keys 33/33/33 over the servers. But as server 2 has
more memory available you might want to give it more weight so more keys get
stored on that server. When you are using a object, the `key` should
represent the server location syntax and the value the weight of the server.
By default all servers have a weight of 1. `{ '192.168.0.102:11212': 1,
'192.168.0.103:11212': 2, '192.168.0.104:11212': 1 }` would generate a
25/50/25 distribution of the keys.
If you would implement one of the above formats, your constructor would
something like this:
3. **Object**, when running a cluster of Memcached servers, some servers may allocate different amounts of memory, e.g. 128, 512, and 128mb. While by default all servers are equally important and dispatch consistently the keys between the servers (33/33/33%), it is possible to send more keys in servers having more memory. To do so, define an object whose `key` represents the server location and whose value represents a server weight, the default weight for a server being 1; so, for instance `{ '192.168.0.102:11211': 1, '192.168.0.103:11211': 2, '192.168.0.104:11211': 1 }` distributes 50% of the keys on server 103, but only 25% on 104 and 25% on 102.
To implement one of the above formats, your constructor would look like this:
```js
var memcached = new Memcached({ '192.168.0.102:11212': 1, '192.168.0.103:11212': 2, '192.168.0.104:11212': 1 });
var memcached = new Memcached([ '192.168.0.102:11212', '192.168.0.103:11212', '192.168.0.104:11212' ]);
var memcached = new Memcached('192.168.0.102:11212');
var memcached = new Memcached({ '192.168.0.102:11211': 1, '192.168.0.103:11211': 2, '192.168.0.104:11211': 1 });
var memcached = new Memcached([ '192.168.0.102:11211', '192.168.0.103:11211', '192.168.0.104:11211' ]);
var memcached = new Memcached('192.168.0.102:11211');
```
### Options
There 2 kinds of options that can be configured. A global configuration that
will be inherited by all Memcached servers instances and a client specific
configuration that can be used to overwrite the globals. The options should be
formatted in an JavaScript `object`. They both use the same object structure:
* `maxKeySize`: *250*, the max size of they key allowed by the Memcached server.
* `maxExpiration`: *2592000*, the max expiration of keys by the Memcached server
in milliseconds.
* `maxValue`: *1048576*, the max size of a value that is allowed by the
Memcached server.
* `poolSize`: *10*, the maximum connections we can allocate in our connection pool.
* `algorithm`: *crc32*, the hashing algorithm that should be used to generate
the hashRing values.
* `reconnect`: *18000000*, when the server is marked as dead we will attempt to
reconnect every x milliseconds.
* `timeout`: *5000*, after x ms the server should send a timeout if we can't
connect. This will also be used close the connection if we are idle.
* `retries`: *5*, How many times to retry socket allocation for given request
* `failures`: *5*, Number of times a server may have issues before marked dead.
* `retry`: *30000*, time to wait between failures before putting server back in
service.
* `remove`: *false*, when the server is marked as dead you can remove it from
the pool so all other will receive the keys instead.
* `failOverServers`: *undefined*, the ability use these servers as failover when
the dead server get's removed from the consistent hashing scheme. This must be
an array of servers confirm the server_locations specification.
* `keyCompression`: *true*, compress keys using md5 if they exceed the
maxKeySize option.
Memcached accepts two option schemes. The first one inherits of all Memcached server instances
while the second one is client specific and overwrites the globals. To define these options,
Memcached server uses the same properties:
* `maxKeySize`: *250*, the maximum key size allowed.
* `maxExpiration`: *2592000*, the maximum expiration time of keys (in seconds).
* `maxValue`: *1048576*, the maximum size of a value.
* `poolSize`: *10*, the maximum size of the connection pool.
* `algorithm`: *md5*, the hashing algorithm used to generate the `hashRing` values.
* `reconnect`: *18000000*, the time between reconnection attempts (in milliseconds).
* `timeout`: *5000*, the time after which Memcached sends a connection timeout (in milliseconds).
* `retries`: *5*, the number of socket allocation retries per request.
* `failures`: *5*, the number of failed-attempts to a server before it is regarded as 'dead'.
* `retry`: *30000*, the time between a server failure and an attempt to set it up back in service.
* `remove`: *false*, if *true*, authorizes the automatic removal of dead servers from the pool.
* `failOverServers`: *undefined*, an array of `server_locations` to replace servers that fail and
that are removed from the consistent hashing scheme.
* `keyCompression`: *true*, whether to use `md5` as hashing scheme when keys exceed `maxKeySize`.
* `idle`: *5000*, the idle timeout for the connections.
Example usage:
```js
var memcached = new Memcached('localhost:11212', {retries:10,retry:10000,remove:true,failOverServers:['192.168.0.103:11212']});
var memcached = new Memcached('localhost:11211', {retries:10,retry:10000,remove:true,failOverServers:['192.168.0.103:11211']});
```
If you wish to configure the options globally:
......@@ -129,30 +106,20 @@ Memcached.config.poolSize = 25;
### Public methods
#### memcached.touch(key, lifetime, callback);
Touches the given key.
**Arguments**
**memcached.touch** Touches the given key.
`key`: **String** The key
`lifetime`: **Number** After how long should the key expire.
`callback`: **Function**
* `key`: **String** The key
* `lifetime`: **Number** After how long should the key expire measured in `seconds`
* `callback`: **Function**
```js
memcached.touch('key', 10, function (err) {
// stuff
});
memcached.touch('key', 10, function (err) { /* stuff */ });
```
#### memcached.get(key, callback);
Get the value for the given key.
**memcached.get** Get the value for the given key.
**Arguments**
`key`: **String**, the key
`callback`: **Function**, the callback.
* `key`: **String**, the key
* `callback`: **Function**, the callback.
```js
memcached.get('foo', function (err, data) {
......@@ -160,14 +127,10 @@ memcached.get('foo', function (err, data) {
});
```
#### memcached.gets(key, callback);
Get the value and the CAS id.
**memcached.gets** Get the value and the CAS id.
**Arguments**
`key`: **String**, the key
`callback`: **Function**, the callback.
* `key`: **String**, the key
* `callback`: **Function**, the callback.
```js
memcached.gets('foo', function (err, data) {
......@@ -177,15 +140,10 @@ memcached.gets('foo', function (err, data) {
// Please note that the data is stored under the name of the given key.
});
```
**memcached.getMulti** Retrieves a bunch of values from multiple keys.
#### memcached.getMulti(keys, callback);
Retrieves a bunch of values from multiple keys.
**Arguments**
`keys`: **Array**, all the keys that needs to be fetched
`callback`: **Function**, the callback.
* `keys`: **Array**, all the keys that needs to be fetched
* `callback`: **Function**, the callback.
```js
memcached.getMulti(['foo', 'bar'], function (err, data) {
......@@ -194,226 +152,160 @@ memcached.getMulti(['foo', 'bar'], function (err, data) {
});
```
#### memcached.set(key, value, lifetime, callback);
**memcached.set** Stores a new value in Memcached.
Stores a new value in Memcached.
**Arguments**
`key`: **String** the name of the key
`value`: **Mixed** Either a buffer, JSON, number or string that you want to store.
`lifetime`: **Number**, how long the data needs to be stored
`callback`: **Function** the callback
* `key`: **String** the name of the key
* `value`: **Mixed** Either a buffer, JSON, number or string that you want to store.
* `lifetime`: **Number**, how long the data needs to be stored measured in `seconds`
* `callback`: **Function** the callback
```js
memcached.set('foo', 'bar', 10, function (err) {
// stuff
});
memcached.set('foo', 'bar', 10, function (err) { /* stuff */ });
```
#### memcached.replace(key, value, lifetime, callback);
**memcached.replace** Replaces the value in memcached.
Replaces the value in memcached.
**Arguments**
`key`: **String** the name of the key
`value`: **Mixed** Either a buffer, JSON, number or string that you want to store.
`lifetime`: **Number**, how long the data needs to be replaced
`callback`: **Function** the callback
* `key`: **String** the name of the key
* `value`: **Mixed** Either a buffer, JSON, number or string that you want to store.
* `lifetime`: **Number**, how long the data needs to be replaced measured in `seconds`
* `callback`: **Function** the callback
```js
memcached.replace('foo', 'bar', 10, function (err) {
// stuff
});
memcached.replace('foo', 'bar', 10, function (err) { /* stuff */ });
```
#### memcached.add(key, value, lifetime, callback);
Add the value, only if it's in memcached already.
**memcached.add** Add the value, only if it's not in memcached already.
**Arguments**
`key`: **String** the name of the key
`value`: **Mixed** Either a buffer, JSON, number or string that you want to store.
`lifetime`: **Number**, how long the data needs to be replaced
`callback`: **Function** the callback
* `key`: **String** the name of the key
* `value`: **Mixed** Either a buffer, JSON, number or string that you want to store.
* `lifetime`: **Number**, how long the data needs to be replaced measured in `seconds`
* `callback`: **Function** the callback
```js
memcached.add('foo', 'bar', 10, function (err) {
// stuff
});
memcached.add('foo', 'bar', 10, function (err) { /* stuff */ });
```
#### memcached.cas(key, value, cas, lifetime, callback);
Add the value, only if it matches the given CAS value.
**Arguments**
**memcached.cas** Add the value, only if it matches the given CAS value.
`key`: **String** the name of the key
`value`: **Mixed** Either a buffer, JSON, number or string that you want to store.
`lifetime`: **Number**, how long the data needs to be replaced
`cas`: **String** the CAS value
`callback`: **Function** the callback
* `key`: **String** the name of the key
* `value`: **Mixed** Either a buffer, JSON, number or string that you want to store.
* `lifetime`: **Number**, how long the data needs to be replaced measured in `seconds`
* `cas`: **String** the CAS value
* `callback`: **Function** the callback
```js
memcached.gets('foo', function (err, data) {
memcached.cas('foo', 'bar', data.cas, 10, function (err) {
// stuff
});
memcached.cas('foo', 'bar', data.cas, 10, function (err) { /* stuff */ });
});
```
#### memcached.append(key, value, callback);
**memcached.append** Add the given value string to the value of an existing item.
Add the given value string to the value of an existing item.
**Arguments**
`key`: **String** the name of the key
`value`: **Mixed** Either a buffer, JSON, number or string that you want to store.
`callback`: **Function** the callback
* `key`: **String** the name of the key
* `value`: **Mixed** Either a buffer, JSON, number or string that you want to store.
* `callback`: **Function** the callback
```js
memcached.append('foo', 'bar', function (err) {
// stuff
});
memcached.append('foo', 'bar', function (err) { /* stuff */ });
```
#### memcached.prepend(key, value, callback);
Add the given value string to the value of an existing item.
**memcached.prepend** Add the given value string to the value of an existing item.
**Arguments**
`key`: **String** the name of the key
`value`: **Mixed** Either a buffer, JSON, number or string that you want to store.
`callback`: **Function** the callback
* `key`: **String** the name of the key
* `value`: **Mixed** Either a buffer, JSON, number or string that you want to store.
* `callback`: **Function** the callback
```js
memcached.preprend('foo', 'bar', function (err) {
// stuff
});
memcached.preprend('foo', 'bar', function (err) { /* stuff */ });
```
#### memcached.incr(key, amount, callback);
Increment a given key.
**memcached.incr** Increment a given key.
**Arguments**
`key`: **String** the name of the key
`amount`: **Number** The incremention
`callback`: **Function** the callback
* `key`: **String** the name of the key
* `amount`: **Number** The increment
* `callback`: **Function** the callback
```js
memcached.incr('foo', 10, function (err) {
// stuff
});
memcached.incr('foo', 10, function (err) { /* stuff */ });
```
#### memcached.decr(key, amount, callback);
Decrement a given kehy.
**Arguments**
**memcached.decr** Decrement a given key.
`key`: **String** the name of the key
`amount`: **Number** The incremention
`callback`: **Function** the callback
* `key`: **String** the name of the key
* `amount`: **Number** The increment
* `callback`: **Function** the callback
```js
memcached.incr('foo', 10, function (err) {
// stuff
});
memcached.decr('foo', 10, function (err) { /* stuff */ });
```
#### memcached.del(key, callback);
Remove the key from memcached.
**memcached.del** Remove the key from memcached.
**Arguments**
`key`: **String** the name of the key
`callback`: **Function** the callback
* `key`: **String** the name of the key
* `callback`: **Function** the callback
```js
memcached.del('foo', function (err) {
// stuff
});
memcached.del('foo', function (err) { /* stuff */ });
```
#### memcached.version(callback);
**memcached.version** Retrieves the version number of your server.
Retrieves the version number of your server.
* `callback`
#### memcached.flush(callback);
**memcached.flush** Flushes the memcached server.
Flushes the memcached server.
* `callback`
#### memcached.stats(callback);
**memcached.stats** Retrieves stats from your memcached server.
Retrieves stats from your memcached server.
* `callback`
#### memcached.settings(callback);
**memcached.settings** Retrieves your `stats settings`.
Retrieves your `stats settings`.
* `callback`
#### memcached.slabs(callback);
**memcached.slabs** Retrieves `stats slabs` information.
Retrieves `stats slabs` information.
* `callback`
#### memcached.items(callback);
**memcached.items** Retrieves `stats items` information.
Retrieves `stats items` information.
* `callback`
#### memcached.cachedump(server, slabid, number, callback);
**memcached.cachedump** Inspect cache, see examples for a detailed explanation.
Inspect cache, see examples for a detailed explaination.
* `server`
* `slabid`
* `number`
* `callback`
#### memcached.end();
Closes all active memcached connections.
**memcached.end** Closes all active memcached connections.
### Private methods
The following methods are intended for private usage:
---------------------------------------
#### .connect
Fetches or generates a connection for the given server. The supplied callback
The following methods are intended for private usage
**.connect** Fetches or generates a connection for the given server. The supplied callback
function will receive a reference to the connection as argument.
If there are issues with the server connection, we are going to respond with cache-miss pattern.
**Arguments**
`server`: *String*, The server that needs a connection, the format must be
* `server`: *String*, The server that needs a connection, the format must be
confirm the server_locations specification.
`callback`: *Function*, The callback function that receives the net.Stream
connection. It will be called with 2 arguments `error` and `connection`.
Example:
* `callback`: *Function*, The callback function that receives the net.Stre
``` js
memcached.connect( '192.168.0.103:11212', function( err, conn ){
memcached.connect( '192.168.0.103:11211', function( err, conn ){
if( err ) throw new Error( err );
console.log( conn.server );
});
```
---------------------------------------
#### .multi
A small wrapper function that makes it easier to query multiple Memcached
**.multi** A small wrapper function that makes it easier to query multiple Memcached
servers. It will return the location for each key or the complete list of
servers.
**Arguments**
`keys`: *Array* **(optional)**, They keys that needs to be converted to a server.
`callback`: *Function*, The callback function for the data, it will be called
* `keys`: *Array* **(optional)**, They keys that needs to be converted to a server.
* `callback`: *Function*, The callback function for the data, it will be called
for **each** key. It will be called with 4 arguments:
1. `server`: *String*, The server location.
......@@ -422,36 +314,26 @@ for **each** key. It will be called with 4 arguments:
3. `index`: *Number*, The current index of the loop
4. `total`: *Number*, The total amount server retrieved.
Example:
``` js
memcached.multi( false, function( server, key, index, totals ){
if( err ) throw new Error( err );
this.connect( server, function( err, conn ){
console.log( "connection ready" )
})
});
```
---------------------------------------
#### .command
This is the core functionality of the `memcached` client. All public API's are
**.command** This is the core functionality of the `memcached` client. All public API's are
routed through this function. It takes care of the argument validations Server
retrieval ( If the server argument isn't specified ). After all data ready a
connection is asked for the private `connect` method and the command is written
to the Memcached server.
**Arguments**
`query`: *Object*, The metaData object, see the `Callbacks` section for the
* `query`: *Object*, The metaData object, see the `Callbacks` section for the
specification.
`server`: *String*, The server the to connect. This is only needed when the
metaData object doesn't contain a key property to retrieve the server from.
Example:
* `server`: *String*, The server the to connect. This is only needed when the
metaData object doesn't contain a key property to retrieve the server from.
``` js
memcached.command({
......@@ -466,26 +348,17 @@ memcached.command({
});
```
---------------------------------------
#### .connectionIssue
A internal function for logging issues with connections. As there can be various
**.connectionIssue** A internal function for logging issues with connections. As there can be various
of ways that an error occurs we need solid issue manager to handle all these
cases. For example server could crash or the Memcached server could respond with
`SERVER ERROR <broken>`.
**Arguments**
`error`: *String*, The actual error message.
`Stream`: *net.Stream*, A reference to the connection stream where the error
* `error`: *String*, The actual error message.
* `Stream`: *net.Stream*, A reference to the connection stream where the error
occurred on.
`callback`: *Function* **(optional)**, The callback function of a potential
* `callback`: *Function* **(optional)**, The callback function of a potential
request, it will be marked as cache miss if it was provided
Example:
``` js
memcached.connectionIssue( "Server down", connectionReference );
```
......@@ -507,7 +380,7 @@ shift to a metaData object. The metaData object contains all information that we
used to generate the request for the Memcached server. The metaData object
contains the following properties:
* `start`: Date in milliseconds when the request was received
* `start`: Date in milliseconds when the request was received
* `execution`: Total execution time for the request, including response parsing.
* `callback`: Reference to the callback function
* `type`: The type of Memcached command
......@@ -515,14 +388,14 @@ contains the following properties:
* `validate`: The properties of metaData object that needs type validation.
And all the arguments you have send to the method, this depends on the method
you have called.
you have called.
## Events
When connection issues occur we send out different notifications using the
`EventEmitter` protocol. This can be useful for logging, notification and
debugging purposes. Each event will receive details Object containing detailed
information about the issues that occurred.
information about the issues that occurred.
### Details Object
......@@ -555,7 +428,7 @@ the success and failure combined.
### Events
There are `5` different events that the `memcached` client emits when connection
issues occur.
issues occur.
* `issue`: a issue occurred on one a server, we are going to attempt a retry next.
* `failure`: a server has been marked as failure or dead.
......@@ -566,11 +439,18 @@ issues occur.
Example implementations:
```js
var memcached = new Memcached([ '192.168.0.102:11212', '192.168.0.103:11212' ]);
var memcached = new Memcached([ '192.168.0.102:11211', '192.168.0.103:11211' ]);
memcached.on('failure', function( details ){ sys.error( "Server " + details.server + "went down due to: " + details.messages.join( '' ) ) });
memcached.on('reconnecting', function( details ){ sys.debug( "Total downtime caused by server " + details.server + " :" + details.totalDownTime + "ms")});
```
# Compatibility
For compatibility with other [libmemcached](http://libmemcached.org/Clients.html) clients they need to have the behavior
`ketama_weighted` set to true and the `hash` set to the same as `node-memcached`'s
`algorithm`.
Due to client dependent type flags it is unlikely that any types other than `string` will work.
# Contributors
This project wouldn't be possible without the hard work of our amazing
......
<html>
<head>
<title>node-memcached</title>
<script src="http://ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js"></script>
<style>body {
margin: 0;
padding: 0;
font: 14px/1.5 'Palatino Linotype', 'Book Antiqua', Palatino, FreeSerif, serif;
color: #252519;
}
a {
color: #252519;
}
a:hover {
text-decoration: underline;
color: #19469D;
}
p {
margin: 12px 0;
}
h1, h2, h3 {
margin: 0;
padding: 0;
}
table#source {
width: 100%;
border-collapse: collapse;
}
table#source td:first-child {
padding: 30px 40px 30px 40px;
vertical-align: top;
}
table#source td:first-child,
table#source td:first-child pre {
width: 450px;
}
table#source td:last-child {
padding: 30px 0 30px 40px;
border-left: 1px solid #E5E5EE;
background: #F5F5FF;
}
table#source tr {
border-bottom: 1px solid #E5E5EE;
}
table#source tr.filename {
padding-top: 40px;
border-top: 1px solid #E5E5EE;
}
table#source tr.filename td:first-child {
text-transform: capitalize;
}
table#source tr.filename td:last-child {
font-size: 12px;
}
table#source tr.filename h2 {
margin: 0;
padding: 0;
cursor: pointer;
}
table#source tr.code h1,
table#source tr.code h2,
table#source tr.code h3 {
margin-top: 30px;
font-family: "Lucida Grande", "Helvetica Nueue", Arial, sans-serif;
font-size: 18px;
}
table#source tr.code h2 {
font-size: 16px;
}
table#source tr.code h3 {
font-size: 14px;
}
table#source tr.code ul {
margin: 15px 0 15px 35px;
padding: 0;
}
table#source tr.code ul li {
margin: 0;
padding: 1px 0;
}
table#source tr.code ul li p {
margin: 0;
padding: 0;
}
table#source tr.code td:first-child pre {
padding: 20px;
}
#ribbon {
position: fixed;
top: 0;
right: 0;
}
code .string { color: #219161; }
code .regexp { color: #219161; }
code .keyword { color: #954121; }
code .number { color: #19469D; }
code .comment { color: #bbb; }
code .this { color: #19469D; }</style>
<script>
$(function(){
$('tr.code').hide();
$('tr.filename').toggle(function(){
$(this).nextUntil('.filename').fadeIn();
}, function(){
$(this).nextUntil('.filename').fadeOut();
});
});
</script>
</head>
<body>
<table id="source"><tbody><tr><td><h1>node-memcached</h1></td><td></td></tr><tr class="filename"><td><h2 id="lib/memcached.js"><a href="#">memcached</a></h2></td><td>lib/memcached.js</td></tr><tr class="code">
<td class="docs">
<p>var EventEmitter = require('events').EventEmitter
, Stream = require('net').Stream
, Buffer = require('buffer').Buffer;</p>
<p>var HashRing = require('hashring')
, Connection = require('./connection')
, Utils = require('./utils')
, Manager = Connection.Manager
, IssueLog = Connection.IssueLog;</p>
<p>// The constructor
function Client(args, options){
if(!(this &amp;&amp; this.hasOwnProperty &amp;&amp; (this instanceof Client))) this = new Client();</p>
<p> var servers = []
, weights = {}
, key;</p>
<p> // Parse down the connection arguments<br></br> switch (Object.prototype.toString.call(args)){
case '[object String]':
servers.push(args);
break;
case '[object Object]':
weights = args;
servers = Object.keys(args);
case '[object Array]':
default:
servers = args;
break;
}</p>
<p> if (!servers.length) throw new Error('No servers where supplied in the arguments');</p>
<p> // merge with global and user config
Utils.merge(this, Client.config);
Utils.merge(this, options);
EventEmitter.call(this);</p>
<p> this.servers = servers;
this.HashRing = new HashRing(args, this.algorithm);
this.connections = {};
this.issues = [];
};</p>
<p>// Allows users to configure the memcached globally or per memcached client
Client.config = {
maxKeySize: 251 // max key size allowed by Memcached
, maxExpiration: 2592000 // max expiration duration allowed by Memcached
, maxValue: 1048576 // max length of value allowed by Memcached</p>
<p>, algorithm: 'crc32' // hashing algorithm that is used for key mapping </p>
<p>, poolSize: 10 // maximal parallel connections
, reconnect: 18000000 // if dead, attempt reconnect each xx ms
, timeout: 5000 // after x ms the server should send a timeout if we can't connect
, retries: 5 // amount of retries before server is dead
, retry: 30000 // timeout between retries, all call will be marked as cache miss
, remove: false // remove server if dead if false, we will attempt to reconnect
, redundancy: false // allows you do re-distribute the keys over a x amount of servers
, keyCompression: true // compress keys if they are to large (md5)
, debug: false // Output the commands and responses
};</p>
<p>// There some functions we don't want users to touch so we scope them
(function(nMemcached){
const LINEBREAK = '\r\n'
, NOREPLY = ' noreply'
, FLUSH = 1E3
, BUFFER = 1E2
, CONTINUE = 1E1
, FLAG<em>JSON = 1&lt;&lt;1
, FLAG</em>BINARY = 2&lt;&lt;1;</p>
<p> var memcached = nMemcached.prototype = new EventEmitter
, private = {}
, undefined;</p>
<p> // Creates or generates a new connection for the give server, the callback will receive the connection
// if the operation was successful
memcached.connect = function connect(server, callback){
// server is dead, bail out
if (server in this.issues &amp;&amp; this.issues[server].failed) return callback(false, false);</p>
<pre><code>// fetch from connection pool
if (server in this.connections) return this.connections[server].allocate(callback);
// No connection factory created yet, so we must build one
var serverTokens = /(.*):(\d+){1,}$/.exec(server).reverse()
, memcached = this;
serverTokens.pop();
this.connections[server] = new Manager(server, this.poolSize, function(callback){
var S = new Stream
, Manager = this;
// config the Stream
S.setTimeout(memcached.timeout);
S.setNoDelay(true);
S.metaData = [];
S.responseBuffer = "";
S.bufferArray = [];
S.server = server;
S.tokens = serverTokens;
// Add the event listeners
Utils.fuse(S, {
connect: function streamConnect(){ callback(false, this) }
, close: function streamClose(){ Manager.remove(this) }
, error: function streamError(err){ memcached.connectionIssue(err, S, callback) }
, data: Utils.curry(memcached, private.buffer, S)
, timeout: function streamTimeout(){ Manager.remove(this) }
, end: S.end
});
// connect the net.Stream [port, hostname]
S.connect.apply(S, serverTokens);
return S;
});
// now that we have setup our connection factory we can allocate a new connection
this.connections[server].allocate(callback);</code></pre>
<p> };</p>
<p> // Creates a multi stream, so it's easier to query agains
// multiple memcached servers.
memcached.multi = function memcachedMulti(keys, callback){
var map = {}
, memcached = this
, servers
, i;</p>
<pre><code>// gets all servers based on the supplied keys,
// or just gives all servers if we don't have keys
if (keys){
keys.forEach(function fetchMultipleServers(key){
var server = memcached.HashRing.getNode(key);
if (map[server]){
map[server].push(key);
} else {
map[server] = [key];
}
});
// store the servers
servers = Object.keys(map);
} else {
servers = this.servers;
}
i = servers.length;
while(i--){
callback.call(this, servers[i], map[servers[i]], i, servers.length);
}</code></pre>
<p> };</p>
<p> // Executes the command on the net.Stream, if no server is supplied it will use the query.key to get
// the server from the HashRing
memcached.command = function memcachedCommand(queryCompiler, server){</p>
<pre><code>// generate a regular query,
var query = queryCompiler()
, redundancy = this.redundancy &amp;&amp; this.redundancy &lt; this.servers.length
, queryRedundancy = query.redundancyEnabled
, memcached = this;
// validate the arguments
if (query.validation &amp;&amp; !Utils.validateArg(query, this)) return;
// fetch servers
server = server ? server : redundancy &amp;&amp; queryRedundancy ? (redundancy = this.HashRing.createRange(query.key, (this.redundancy + 1), true)).shift() : this.HashRing.getNode(query.key);
// check if the server is still alive
if (server in this.issues &amp;&amp; this.issues[server].failed) return query.callback &amp;&amp; query.callback(false, false);
Client.config.debug &amp;&amp; console.log(query.command);
this.connect(server, function allocateMemcachedConnection(error, S){
// check for issues
if (!S) return query.callback &amp;&amp; query.callback(false, false);
if (error) return query.callback &amp;&amp; query.callback(error);
if (S.readyState !== 'open') return query.callback &amp;&amp; query.callback('Connection readyState is set to ' + S.readySate);
// used for request timing
query.start = Date.now();
S.metaData.push(query);
S.write(query.command + LINEBREAK);
});
// if we have redundancy enabled and the query is used for redundancy, than we are going loop over
// the servers, check if we can reach them, and connect to the correct net connection.
// because all redundancy queries are executed with "no reply" we do not need to store the callback
// as there will be no value to parse.
if (redundancy &amp;&amp; queryRedundancy){
queryRedundancy = queryCompiler(queryRedundancy);
redundancy.forEach(function(server){
if (server in memcached.issues &amp;&amp; memcached.issues[server].failed) return;
memcached.connect(server, function allocateMemcachedConnection(error, S){
if (!S || error || S.readyState !== 'open') return;
S.write(queryRedundancy.command + LINEBREAK);
});
})
}</code></pre>
<p> };</p>
<p> // Logs all connection issues, and handles them off. Marking all requests as cache misses.
memcached.connectionIssue = function connectionIssue(error, S, callback){
// end connection and mark callback as cache miss
if (S &amp;&amp; S.end) S.end();
if (callback) callback(false, false);</p>
<pre><code>var issues
, server = S.server
, memcached = this;
// check for existing issue logs, or create a new log
if (server in this.issues){
issues = this.issues[server];
} else {
issues = this.issues[server] = new IssueLog({
server: server
, tokens: S.tokens
, reconnect: this.reconnect
, retries: this.retries
, retry: this.retry
, remove: this.remove
});
// proxy the events
Utils.fuse(issues, {
issue: function(details){ memcached.emit('issue', details) }
, failure: function(details){ memcached.emit('failure', details) }
, reconnecting: function(details){ memcached.emit('reconnecting', details) }
, reconnected: function(details){ memcached.emit('reconnect', details) }
, remove: function(details){
// emit event and remove servers
memcached.emit('remove', details);
memcached.connections[server].end();
if (this.failOverServers &amp;&amp; this.failOverServers.length){
memcached.HashRing.replaceServer(server, this.failOverServers.shift());
} else {
memcached.HashRing.removeServer(server);
}
}
});
}
// log the issue
issues.log(error);</code></pre>
<p> };</p>
<p> // Kills all active connections
memcached.end = function endMemcached(){
var memcached = this;
Object.keys(this.connections).forEach(function closeConnection(key){
memcached.connections[key].free(0)
});
};</p>
<p> // These do not need to be publicly available as it's one of the most important
// parts of the whole client, the parser commands:
private.parsers = {
// handle error responses
'NOT<em>FOUND': function(tokens, dataSet, err){ return [CONTINUE, false] }
, 'NOT</em>STORED': function(tokens, dataSet, err){ return [CONTINUE, false] }
, 'ERROR': function(tokens, dataSet, err){ err.push('Received an ERROR response'); return [FLUSH, false] }
, 'CLIENT<em>ERROR': function(tokens, dataSet, err){ err.push(tokens.splice(1).join(' ')); return [CONTINUE, false] }
, 'SERVER</em>ERROR': function(tokens, dataSet, err, queue, S, memcached){ memcached.connectionIssue(tokens.splice(1).join(' '), S); return [CONTINUE, false] }</p>
<pre><code>// keyword based responses</code></pre>
<p> , 'STORED': function(tokens, dataSet){ return [CONTINUE, true] }
, 'DELETED': function(tokens, dataSet){ return [CONTINUE, true] }
, 'OK': function(tokens, dataSet){ return [CONTINUE, true] }
, 'EXISTS': function(tokens, dataSet){ return [CONTINUE, false] }
, 'END': function(tokens, dataSet, err, queue){ if (!queue.length) queue.push(false); return [FLUSH, true] }</p>
<pre><code>// value parsing:</code></pre>
<p> , 'VALUE': function(tokens, dataSet, err, queue){
var key = tokens[1]
, flag = +tokens[2]
, expire = tokens[3]
, cas = tokens[4]
, multi = this.metaData[0] &amp;&amp; this.metaData[0].multi || cas ? {} : false
, tmp;</p>
<pre><code> switch (flag){
case FLAG_JSON:
dataSet = JSON.parse(dataSet);
break;
case FLAG_BINARY:
tmp = new Buffer(dataSet.length);
tmp.write(dataSet, 0, 'binary');
dataSet = tmp;
break;
}
// Add to queue as multiple get key key key key key returns multiple values
if (!multi){
queue.push(dataSet);
} else {
multi[key] = dataSet;
if (cas) multi.cas = cas;
queue.push(multi);
}
return [BUFFER, false]
}</code></pre>
<p> , 'INCRDECR': function(tokens){ return [CONTINUE, +tokens[1]] }
, 'STAT': function(tokens, dataSet, err, queue){
queue.push([tokens[1], /^\d+$/.test(tokens[2]) ? +tokens[2] : tokens[2]]);
return [BUFFER, true]
}
, 'VERSION': function(tokens, dataSet){
var versionTokens = /(\d+)(?:.)(\d+)(?:.)(\d+)$/.exec(tokens.pop());</p>
<pre><code> return [CONTINUE, {
server: this.server
, version: versionTokens[0]
, major: versionTokens[1] || 0
, minor: versionTokens[2] || 0
, bugfix: versionTokens[3] || 0
}];
}</code></pre>
<p> , 'ITEM': function(tokens, dataSet, err, queue){
queue.push({
key: tokens[1]
, b: +tokens[2].substr(1)
, s: +tokens[4]
});
return [BUFFER, false]
}
};</p>
<p> // Parses down result sets
private.resultParsers = {
// combines the stats array, in to an object
'stats': function(resultSet){
var response = {};</p>
<pre><code> // add references to the retrieved server
response.server = this.server;
// Fill the object
resultSet.forEach(function(statSet){
response[statSet[0]] = statSet[1];
});
return response;
}
// the settings uses the same parse format as the regular stats</code></pre>
<p> , 'stats settings': function(){ return private.resultParsers.stats.apply(this, arguments) }
// Group slabs by slab id
, 'stats slabs': function(resultSet){
var response = {};</p>
<pre><code> // add references to the retrieved server
response.server = this.server;
// Fill the object
resultSet.forEach(function(statSet){
var identifier = statSet[0].split(':');
if (!response[identifier[0]]) response[identifier[0]] = {};
response[identifier[0]][identifier[1]] = statSet[1];
});
return response;
}</code></pre>
<p> , 'stats items': function(resultSet){
var response = {};</p>
<pre><code> // add references to the retrieved server
response.server = this.server;
// Fill the object
resultSet.forEach(function(statSet){
var identifier = statSet[0].split(':');
if (!response[identifier[1]]) response[identifier[1]] = {};
response[identifier[1]][identifier[2]] = statSet[1];
});
return response;
}</code></pre>
<p> };</p>
<p> // Generates a RegExp that can be used to check if a chunk is memcached response identifier
private.allCommands = new RegExp('^(?:' + Object.keys(private.parsers).join('|') + '|\d' + ')');
private.bufferedCommands = new RegExp('^(?:' + Object.keys(private.parsers).join('|') + ')');</p>
<p> // When working with large chunks of responses, node chunks it in to pieces. So we might have
// half responses. So we are going to buffer up the buffer and user our buffered buffer to query
// against. Also when you execute allot of .writes to the same stream, node will combine the responses
// in to one response stream. With no indication where it had cut the data. So it can be it cuts inside the value response,
// or even right in the middle of a line-break, so we need to make sure, the last piece in the buffer is a LINEBREAK
// because that is all what is sure about the Memcached Protocol, all responds end with them.
private.buffer = function BufferBuffer(S, BufferStream){
S.responseBuffer += BufferStream;</p>
<pre><code>Client.config.debug &amp;&amp; console.log(S.responseBuffer);
// only call transform the data once we are sure, 100% sure, that we valid response ending
if (S.responseBuffer.substr(S.responseBuffer.length - 2) === LINEBREAK){
var chunks = S.responseBuffer.split(LINEBREAK);
S.responseBuffer = ""; // clear!
this.rawDataReceived(S, S.bufferArray = S.bufferArray.concat(chunks));
} </code></pre>
<p> };</p>
<p> // The actual parsers function that scan over the responseBuffer in search of Memcached response
// identifiers. Once we have found one, we will send it to the dedicated parsers that will transform
// the data in a human readable format, deciding if we should queue it up, or send it to a callback fn.
memcached.rawDataReceived = function rawDataReceived(S){
var queue = []
, token
, tokenSet
, dataSet = ''
, resultSet
, metaData
, err = []
, tmp;</p>
<pre><code>while(S.bufferArray.length &amp;&amp; private.allCommands.test(S.bufferArray[0])){
token = S.bufferArray.shift();
tokenSet = token.split(' ');
// special case for digit only's these are responses from INCR and DECR
if (/\d+/.test(tokenSet[0])) tokenSet.unshift('INCRDECR');
// special case for value, it's required that it has a second response!
// add the token back, and wait for the next response, we might be handling a big
// ass response here.
if (tokenSet[0] == 'VALUE' &amp;&amp; S.bufferArray.indexOf('END') == -1){
return S.bufferArray.unshift(token);
}
// check for dedicated parser
if (private.parsers[tokenSet[0]]){
// fetch the response content
while(S.bufferArray.length){
if (private.bufferedCommands.test(S.bufferArray[0])) break;
dataSet += S.bufferArray.shift();
};
resultSet = private.parsers[tokenSet[0]].call(S, tokenSet, dataSet || token, err, queue, this);
// check how we need to handle the resultSet response
switch(resultSet.shift()){
case BUFFER:
break;
case FLUSH:
metaData = S.metaData.shift();
resultSet = queue;
// if we have a callback, call it
if (metaData &amp;&amp; metaData.callback){
metaData.execution = Date.now() - metaData.start;
metaData.callback.call(
metaData, err.length ? err : err[0],
// see if optional parsing needs to be applied to make the result set more readable
private.resultParsers[metaData.type] ? private.resultParsers[metaData.type].call(S, resultSet, err) :
!Array.isArray(queue) || queue.length &gt; 1 ? queue : queue[0]
);
}
queue.length = err.length = 0;
break;
case CONTINUE:
default:
metaData = S.metaData.shift();
if (metaData &amp;&amp; metaData.callback){
metaData.execution = Date.now() - metaData.start;
metaData.callback.call(metaData, err.length &gt; 1 ? err : err[0], resultSet[0]);
}
err.length = 0;
break;
}
} else {
// handle unkown responses
metaData = S.metaData.shift();
if (metaData &amp;&amp; metaData.callback){
metaData.execution = Date.now() - metaData.start;
metaData.callback.call(metaData, 'Unknown response from the memcached server: "' + token + '"', false);
}
}
// cleanup
dataSet = ''
tokenSet = metaData = undefined;
// check if we need to remove an empty item from the array, as splitting on /r/n might cause an empty
// item at the end..
if (S.bufferArray[0] === '') S.bufferArray.shift();
};</code></pre>
<p> };</p>
<p> // Small wrapper function that only executes errors when we have a callback
private.errorResponse = function errorResponse(error, callback){
if (typeof callback == 'function') callback(error, false);</p>
<pre><code>return false;</code></pre>
<p> };</p>
<p> // This is where the actual Memcached API layer begins:
memcached.get = function get(key, callback){
if (Array.isArray(key)) return this.getMulti.apply(this, arguments);</p>
<pre><code>this.command(function getCommand(noreply){ return {
key: key
, callback: callback
, validate: [['key', String], ['callback', Function]]
, type: 'get'
, command: 'get ' + key
}});</code></pre>
<p> };</p>
<p> // the difference between get and gets is that gets, also returns a cas value
// and gets doesn't support multi-gets at this moment.
memcached.gets = function get(key, callback){
this.command(function getCommand(noreply){ return {
key: key
, callback: callback
, validate: [['key', String], ['callback', Function]]
, type: 'gets'
, command: 'gets ' + key
}});
};</p>
<p> // Handles get's with multiple keys
memcached.getMulti = function getMulti(keys, callback){
var memcached = this
, responses = {}
, errors = []
, calls</p>
<pre><code> // handle multiple responses and cache them untill we receive all.
, handle = function(err, results){
if (err) errors.push(err);
// add all responses to the array
(Array.isArray(results) ? results : [results]).forEach(function(value){ Utils.merge(responses, value) });
if (!--calls) callback(errors.length ? errors : false, responses);
};
this.multi(keys, function(server, key, index, totals){
if (!calls) calls = totals;
memcached.command(function getMultiCommand(noreply){ return {
callback: handle
, multi:true
, type: 'get'
, command: 'get ' + key.join(' ')
}},
server
);
});</code></pre>
<p> };</p>
<p> // As all command nearly use the same syntax we are going to proxy them all to this
// function to ease maintenance. This is possible because most set commands will use the same
// syntax for the Memcached server. Some commands do not require a lifetime and a flag, but the
// memcached server is smart enough to ignore those.
private.setters = function setters(type, validate, key, value, lifetime, callback, cas){
var flag = 0
, memcached = this
, valuetype = typeof value
, length;</p>
<pre><code>if (Buffer.isBuffer(value)){
flag = FLAG_BINARY;
value = value.toString('binary');
} else if (valuetype !== 'string' &amp;&amp; valuetype !== 'number'){
flag = FLAG_JSON;
value = JSON.stringify(value);
} else {
value = value.toString();
}
length = Buffer.byteLength(value);
if (length &gt; memcached.maxValue) return private.errorResponse('The length of the value is greater than ' + memcached.maxValue, callback);
memcached.command(function settersCommand(noreply){ return {
key: key
, callback: callback
, lifetime: lifetime
, value: value
, cas: cas
, validate: validate
, type: type
, redundancyEnabled: true
, command: [type, key, flag, lifetime, length].join(' ') +
(cas ? ' ' + cas : '') +
(noreply ? NOREPLY : '') +
LINEBREAK + value
}});</code></pre>
<p> };</p>
<p> // Curry the function and so we can tell the type our private set function
memcached.set = Utils.curry(false, private.setters, 'set', [['key', String], ['lifetime', Number], ['value', String], ['callback', Function]]);
memcached.replace = Utils.curry(false, private.setters, 'replace', [['key', String], ['lifetime', Number], ['value', String], ['callback', Function]]);
memcached.add = Utils.curry(false, private.setters, 'add', [['key', String], ['lifetime', Number], ['value', String], ['callback', Function]]);</p>
<p> memcached.cas = function checkandset(key, value, cas, lifetime, callback){
private.setters.call(this, 'cas', [['key', String], ['lifetime', Number], ['value', String], ['callback', Function]], key, value, lifetime, callback, cas);