...
 
Commits (39)
# ensure that we only run ci test on the Node.js enabled servers
# Ensure that we only run ci test on the Node.js enabled servers.
language: node_js
node_js:
- 0.8
- "0.8"
- "0.10"
# create a travis enabled environment for the test suite to run in
# Create a travis enabled environment for the test suite to run in.
script: "make travisci"
services: memcache
......
# Authors and contributors of the node Memcached driver project in alphabetical order.
- Alfonso Boza
- Anton Onyshchenko
- Arek Flinik
- Arnout Kazemier
- Jan Krems
- Jason Pearlman
- Joseph Mordetsky
- Kinya TERASAKA
- Near Privman
- Nebojsa Sabovic
- René van Sweeden
- Ron Korving
- Sebastian Seilund
- Tobias Müllerleile
0.2.3
### 0.2.4
- Tons of fixes have been made to the way we do error handling and failover,
this includes better reconnect, server failure detection, timeout handling
and much more.
- Introduction of a new `idle` timeout option.
- Documentation improvements.
### 0.2.3
- Added documentation for public api's
- new namespace option added that namespaces all your keys.
- minor parser fixes and some thrown errors.
0.2.2
### 0.2.2
- Support for touch command #86
- Fix for chunked responses from the server #84
0.2.1
### 0.2.1
- Supports for a queued callback limit so it would crash the process when we queue
to much callbacks. #81
0.2.0
### 0.2.0
- [breaking] We are now returning Error instances instead of strings for errors
- Dependency bump for a critical bug in our connection pool.
0.1.5
### 0.1.5
- Don't execute callbacks multiple times if the connection fails
- Parser fix for handling server responses that contain Memcached Procotol
- Parser fix for handling server responses that contain Memcached Protocol
keywords
- Make sure that the retry option is set correctly
0.1.4
### 0.1.4
- Added missing error listener to the 3rd-Eden/jackpot module, this prevents crashes
when it's unable to connect to a server.
0.1.3
### 0.1.3
- Handle Memcached responses that contain no value.
- Travis CI integration.
0.1.2
- Returning an error when the Memcached server issues a NOT_STORED response.
### 0.1.2
- Returning an error when the Memcached server issues a `NOT_STORED` response.
0.1.1
### 0.1.1
- Now using 3rd-Eden/jackpot as connection pool, this should give a more stable
connection.
0.1.0
### 0.1.0
- Storing numeric values are now returned as numeric values, they are no
longer strings.
0.0.12
- Added .setEncoding to the connections, this way UTF-8 chars will not be
### 0.0.12
- Added `.setEncoding` to the connections, this way UTF-8 chars will not be
chopped in to multiple pieces and breaking binary stored data or UTF-8 text
0.0.11
### 0.0.11
- Added more useful error messages instead of returning false. Please note
that they are still strings instead of Error instances (legacy)
0.0.10
### 0.0.10
- Compatibility with Node.js 0.8
- Don't saturate the Node process by retrying to connect if pool is full #43
- Minor code formatting
0.0.9
- Codestyle refactor, named the functions, removed tabs
### 0.0.9
- Code style refactor, named the functions, removed tabs
- Added Mocha test suite
#Memcached [![Build Status](https://secure.travis-ci.org/3rd-Eden/node-memcached.png?branch=master)](http://travis-ci.org/3rd-Eden/node-memcached)
# Memcached [![Build Status](https://secure.travis-ci.org/3rd-Eden/node-memcached.png?branch=master)](http://travis-ci.org/3rd-Eden/node-memcached)
`memcached` is a fully featured Memcached client for Node.js. `memcached` is
build with scaling, high availability and exceptional performance in mind. We
......@@ -98,8 +98,10 @@ formatted in an JavaScript `object`. They both use the same object structure:
reconnect every x milliseconds.
* `timeout`: *5000*, after x ms the server should send a timeout if we can't
connect. This will also be used close the connection if we are idle.
* `retries`: *5*, amount of tries before we mark the server as dead.
* `retry`: *30000*, timeout between each retry in x milliseconds.
* `retries`: *5*, How many times to retry socket allocation for given request
* `failures`: *5*, Number of times a server may have issues before marked dead.
* `retry`: *30000*, time to wait between failures before putting server back in
service.
* `remove`: *false*, when the server is marked as dead you can remove it from
the pool so all other will receive the keys instead.
* `failOverServers`: *undefined*, the ability use these servers as failover when
......@@ -107,6 +109,7 @@ formatted in an JavaScript `object`. They both use the same object structure:
an array of servers confirm the server_locations specification.
* `keyCompression`: *true*, compress keys using md5 if they exceed the
maxKeySize option.
* `idle`: *5000*, the idle timeout for the connections.
Example usage:
......@@ -191,53 +194,53 @@ memcached.getMulti(['foo', 'bar'], function (err, data) {
});
```
#### memcached.set(key, lifetime, value, callback);
#### memcached.set(key, value, lifetime, callback);
Stores a new value in Memcached.
**Arguments**
`key`: **String** the name of the key
`lifetime`: **Number**, how long the data needs to be stored
`value`: **Mixed** Either a buffer, JSON, number or string that you want to store.
`lifetime`: **Number**, how long the data needs to be stored
`callback`: **Function** the callback
```js
memcached.set('foo', 10, 'bar', function (err) {
memcached.set('foo', 'bar', 10, function (err) {
// stuff
});
```
#### memcached.replace(key, lifetime, value, callback);
#### memcached.replace(key, value, lifetime, callback);
Replaces the value in memcached.
**Arguments**
`key`: **String** the name of the key
`lifetime`: **Number**, how long the data needs to be replaced
`value`: **Mixed** Either a buffer, JSON, number or string that you want to store.
`lifetime`: **Number**, how long the data needs to be replaced
`callback`: **Function** the callback
```js
memcached.replaces('foo', 10, 'bar', function (err) {
memcached.replace('foo', 'bar', 10, function (err) {
// stuff
});
```
#### memcached.add(key, lifetime, value, callback);
#### memcached.add(key, value, lifetime, callback);
Add the value, only if it's in memcached already.
**Arguments**
`key`: **String** the name of the key
`lifetime`: **Number**, how long the data needs to be replaced
`value`: **Mixed** Either a buffer, JSON, number or string that you want to store.
`lifetime`: **Number**, how long the data needs to be replaced
`callback`: **Function** the callback
```js
memcached.add('foo', 10, 'bar', function (err) {
memcached.add('foo', 'bar', 10, function (err) {
// stuff
});
```
......@@ -535,9 +538,9 @@ following 3 will always be present in all error events:
The following properties depend on the type of event that is send. If we are
still in our retry phase the details will also contain:
* `retries`: the amount of retries left before we mark the server as dead.
* `totalRetries`: the total amount of retries we did on this server, as when the
server has been reconnected after it's dead the `retries` will be rest to
* `failures`: the amount of failures left before we mark the server as dead.
* `totalFailures`: the total amount of failures that occurred on this server, as when the
server has been reconnected after it's dead the `failures` will be rest to
defaults and messages will be removed.
If the server is dead these details will be added:
......@@ -567,3 +570,16 @@ var memcached = new Memcached([ '192.168.0.102:11212', '192.168.0.103:11212' ]);
memcached.on('failure', function( details ){ sys.error( "Server " + details.server + "went down due to: " + details.messages.join( '' ) ) });
memcached.on('reconnecting', function( details ){ sys.debug( "Total downtime caused by server " + details.server + " :" + details.totalDownTime + "ms")});
```
# Contributors
This project wouldn't be possible without the hard work of our amazing
contributors. See the contributors tab in Github for an up to date list of
[contributors](/3rd-Eden/node-memcached/graphs/contributors).
Thanks for all your hard work on this project!
# License
The driver is released under the MIT license. See the
[LICENSE](/3rd-Eden/node-memcached/blob/master/LICENSE) for more information.
......@@ -25,8 +25,10 @@ function IssueLog (args) {
this.config = args;
this.messages = [];
this.failed = false;
this.locked = false;
this.isScheduledToReconnect = false;
this.totalRetries = 0;
this.totalFailures = 0;
this.retry = 0;
this.totalReconnectsAttempted = 0;
this.totalReconnectsSuccess = 0;
......@@ -42,33 +44,36 @@ issues.log = function log (message) {
this.failed = true;
this.messages.push(message || 'No message specified');
if (this.retries) {
if (this.failures && !this.locked) {
this.locked = true;
setTimeout(issue.attemptRetry.bind(issue), this.retry);
return this.emit('issue', this.details);
}
if (this.remove) return this.emit('remove', this.details);
setTimeout(issue.attemptReconnect.bind(issue), this.reconnect);
if (!this.isScheduledToReconnect) {
this.isScheduledToReconnect = true;
setTimeout(issue.attemptReconnect.bind(issue), this.reconnect);
}
};
Object.defineProperty(issues, 'details', {
get: function getDetails () {
var res = {};
res.server = this.serverAddress;
res.server = this.server;
res.tokens = this.tokens;
res.messages = this.messages;
if (this.retries) {
res.retries = this.retries;
res.totalRetries = this.totalRetries;
if (this.failures) {
res.failures = this.failures;
res.totalFailures = this.totalFailures;
} else {
res.totalReconnectsAttempted = this.totalReconnectsAttempted;
res.totalReconnectsSuccess = this.totalReconnectsSuccess;
res.totalReconnectsFailed = this.totalReconnectsAttempted - this.totalReconnectsSuccess;
res.totalDownTime = (res.totalReconnectsFailed * this.reconnect) + (this.totalRetries * this.retry);
res.totalDownTime = (res.totalReconnectsFailed * this.reconnect) + (this.totalFailures * this.retry);
}
return res;
......@@ -76,9 +81,10 @@ Object.defineProperty(issues, 'details', {
});
issues.attemptRetry = function attemptRetry () {
this.totalRetries++;
this.retries--;
this.totalFailures++;
this.failures--;
this.failed = false;
this.locked = false;
};
issues.attemptReconnect = function attemptReconnect () {
......@@ -99,6 +105,7 @@ issues.attemptReconnect = function attemptReconnect () {
issue.totalReconnectsSuccess++;
issue.messages.length = 0;
issue.failed = false;
issue.isScheduledToReconnect = false;
// we connected again, so we are going through the whole cycle again
Utils.merge(issue, JSON.parse(JSON.stringify(issue.config)));
......
......@@ -3,8 +3,7 @@
/**
* Node's native modules
*/
var EventEmitter = require('events').EventEmitter
, Stream = require('net').Stream
var Stream = require('net').Stream
, Socket = require('net').Socket;
/**
......@@ -58,7 +57,6 @@ function Client (args, options) {
// merge with global and user config
Utils.merge(this, Client.config);
Utils.merge(this, options);
EventEmitter.call(this);
this.servers = servers;
this.HashRing = new HashRing(args, this.algorithm);
......@@ -76,10 +74,17 @@ Client.config = {
, algorithm: 'crc32' // hashing algorithm that is used for key mapping
, poolSize: 10 // maximal parallel connections
, retries: 5 // Connection pool retries to pull connection from pool
, factor: 3 // Connection pool retry exponential backoff factor
, minTimeout: 1000 // Connection pool retry min delay before retrying
, maxTimeout: 60000 // Connection pool retry max delay before retrying
, randomize: false // Connection pool retry timeout randomization
, reconnect: 18000000 // if dead, attempt reconnect each xx ms
, timeout: 5000 // after x ms the server should send a timeout if we can't connect
, retries: 5 // amount of retries before server is dead
, retry: 30000 // timeout between retries, all call will be marked as cache miss
, failures: 5 // Number of times a server can have an issue before marked dead
, retry: 30000 // When a server has an error, wait this amount of time before retrying
, idle: 5000 // Remove connection from pool when no I/O after `idle` ms
, remove: false // remove server if dead if false, we will attempt to reconnect
, redundancy: false // allows you do re-distribute the keys over a x amount of servers
, keyCompression: true // compress keys if they are to large (md5)
......@@ -98,7 +103,9 @@ Client.config = {
, FLAG_BINARY = 1<<2
, FLAG_NUMERIC = 1<<3;
var memcached = nMemcached.prototype = new EventEmitter
nMemcached.prototype.__proto__ = require('events').EventEmitter.prototype;
var memcached = nMemcached.prototype
, privates = {}
, undefined;
......@@ -133,12 +140,25 @@ Client.config = {
manager = new Jackpot(this.poolSize);
manager.retries = memcached.retries;
manager.factor = memcached.factor;
manager.minTimeout = memcached.minTimeout;
manager.maxTimeout = memcached.maxTimeout;
manager.randomize = memcached.randomize;
manager.setMaxListeners(0);
manager.factory(function factory() {
var S = Array.isArray(serverTokens)
? new Stream
: new Socket
, Manager = this;
, Manager = this
, idleTimeout = function() {
Manager.remove(this);
}
, streamError = function(e) {
memcached.connectionIssue(e.toString(), S);
Manager.remove(this);
};
// config the Stream
S.streamID = sid++;
......@@ -158,8 +178,14 @@ Client.config = {
Manager.remove(this);
}
, data: curry(memcached, privates.buffer, S)
, timeout: function streamTimeout() {
Manager.remove(this);
, connect: function streamConnect() {
// Jackpot handles any pre-connect timeouts by calling back
// with the error object.
this.setTimeout(this.memcached.idle, idleTimeout);
// Jackpot handles any pre-connect errors, but does not handle errors
// once a connection has been made, nor does Jackpot handle releasing
// connections if an error occurs post-connect
this.on('error', streamError);
}
, end: S.end
});
......@@ -262,8 +288,10 @@ Client.config = {
}
}
// check if the server is still alive
if (server in this.issues && this.issues[server].failed) {
// check if any server exists or and if the server is still alive
// a server may not exist if the manager was never able to connect
// to any server.
if (!server || (server in this.issues && this.issues[server].failed)) {
return query.callback && memcached.makeCallback(query.callback,new Error('Server not available'));
}
......@@ -275,11 +303,21 @@ Client.config = {
}
// check for issues
if (error) return query.callback && memcached.makeCallback(query.callback,error);
if (!S) return query.callback && memcached.makeCallback(query.callback,new Error('Connect did not give a server'));
if (error) {
memcached.connectionIssue(error.toString(), S);
return query.callback && memcached.makeCallback(query.callback,error);
}
if (!S) {
var message = 'Connect did not give a server';
memcached.connectionIssue(message);
return query.callback && memcached.makeCallback(query.callback,new Error(message));
}
if (S.readyState !== 'open') {
return query.callback && memcached.makeCallback(query.callback,new Error('Connection readyState is set to ' + S.readySate));
var message = 'Connection readyState is set to ' + S.readyState;
memcached.connectionIssue(message, S);
return query.callback && memcached.makeCallback(query.callback,new Error(message));
}
// used for request timing
......@@ -326,7 +364,7 @@ Client.config = {
server: server
, tokens: S.tokens
, reconnect: this.reconnect
, retries: this.retries
, failures: this.failures
, retry: this.retry
, remove: this.remove
});
......@@ -354,6 +392,7 @@ Client.config = {
memcached.HashRing.replaceServer(server, this.failOverServers.shift());
} else {
memcached.HashRing.removeServer(server);
memcached.emit('failure', details);
}
}
});
......@@ -877,8 +916,8 @@ Client.config = {
, 'set'
, [
['key', String]
, ['lifetime', Number]
, ['value', String]
, ['lifetime', Number]
, ['callback', Function]
]
);
......@@ -887,8 +926,8 @@ Client.config = {
, 'replace'
, [
['key', String]
, ['lifetime', Number]
, ['value', String]
, ['lifetime', Number]
, ['callback', Function]
]
);
......@@ -897,8 +936,8 @@ Client.config = {
, 'add'
, [
['key', String]
, ['lifetime', Number]
, ['value', String]
, ['lifetime', Number]
, ['callback', Function]
]
);
......@@ -908,8 +947,8 @@ Client.config = {
, 'cas'
, [
['key', String]
, ['lifetime', Number]
, ['value', String]
, ['lifetime', Number]
, ['callback', Function]
]
, key
......@@ -925,8 +964,8 @@ Client.config = {
, 'append'
, [
['key', String]
, ['lifetime', Number]
, ['value', String]
, ['lifetime', Number]
, ['callback', Function]
]
, key
......@@ -941,8 +980,8 @@ Client.config = {
, 'prepend'
, [
['key', String]
, ['lifetime', Number]
, ['value', String]
, ['lifetime', Number]
, ['callback', Function]
]
, key
......
{
"name": "memcached"
, "version": "0.2.3"
, "version": "0.2.4"
, "author": "Arnout Kazemier"
, "description": "A fully featured Memcached API client, supporting both single and clustered Memcached servers through consistent hashing and failover/failure. Memcached is rewrite of nMemcached, which will be deprecated in the near future."
, "main": "index"
......@@ -35,7 +35,7 @@
}
, "dependencies": {
"hashring": "0.0.x"
, "jackpot": ">=0.0.2"
, "jackpot": ">=0.0.6"
}
, "devDependencies": {
"mocha": "*"
......
......@@ -33,4 +33,201 @@ describe('Memcached connections', function () {
done();
});
});
it('should remove a failed server', function(done) {
var memcached = new Memcached('127.0.1:1234', {
timeout: 1000,
retries: 0,
failures: 0,
retry: 100,
remove: true });
this.timeout(60000);
memcached.get('idontcare', function (err) {
function noserver() {
memcached.get('idontcare', function(err) {
throw err;
});
};
assert.throws(noserver, /Server not available/);
memcached.end();
done();
});
});
it('should rebalance to remaining healthy server', function(done) {
var memcached = new Memcached(['127.0.1:1234', common.servers.single], {
timeout: 1000,
retries: 0,
failures: 0,
retry: 100,
remove: true,
redundancy: true });
this.timeout(60000);
// 'a' goes to fake server. first request will cause server to be removed
memcached.get('a', function (err) {
// second request should be rebalanced to healthy server
memcached.get('a', function (err) {
assert.ifError(err);
memcached.end();
done();
});
});
});
it('should properly schedule failed server retries', function(done) {
var server = '127.0.0.1:1234';
var memcached = new Memcached(server, {
retries: 0,
failures: 5,
retry: 100 });
// First request will schedule a retry attempt, and lock scheduling
memcached.get('idontcare', function (err) {
assert.throws(function() { throw err }, /connect ECONNREFUSED/);
assert.deepEqual(memcached.issues[server].failures, 5);
assert.deepEqual(memcached.issues[server].locked, true);
assert.deepEqual(memcached.issues[server].failed, true);
// Immediate request should not decrement failures
memcached.get('idontcare', function(err) {
assert.throws(function() { throw err }, /Server not available/);
assert.deepEqual(memcached.issues[server].failures, 5);
assert.deepEqual(memcached.issues[server].locked, true);
assert.deepEqual(memcached.issues[server].failed, true);
// Once `retry` time has passed, failures should decrement by one
setTimeout(function() {
// Server should be back in action
assert.deepEqual(memcached.issues[server].locked, false);
assert.deepEqual(memcached.issues[server].failed, false);
memcached.get('idontcare', function(err) {
// Server should be marked healthy again, though we'll get this error
assert.throws(function() { throw err }, /connect ECONNREFUSED/);
assert.deepEqual(memcached.issues[server].failures, 4);
memcached.end();
done();
});
}, 100); // `retry` is 100 so wait 100
});
});
});
it('should properly schedule server reconnection attempts', function(done) {
var server = '127.0.0.1:1234'
, memcached = new Memcached(server, {
retries: 3,
minTimeout: 0,
maxTimeout: 100,
failures: 0,
reconnect: 100 })
, reconnectAttempts = 0;
memcached.on('reconnecting', function() {
reconnectAttempts++;
});
// First request will mark server dead and schedule reconnect
memcached.get('idontcare', function (err) {
assert.throws(function() { throw err }, /connect ECONNREFUSED/);
// Second request should not schedule another reconnect
memcached.get('idontcare', function (err) {
assert.throws(function() { throw err }, /Server not available/);
// Allow enough time to pass for a connection retries to occur
setTimeout(function() {
assert.deepEqual(reconnectAttempts, 1);
memcached.end();
done();
}, 400);
});
});
});
it('should reset failures after reconnecting to failed server', function(done) {
var server = '127.0.0.1:1234'
, memcached = new Memcached(server, {
retries: 0,
minTimeout: 0,
maxTimeout: 100,
failures: 1,
retry: 1,
reconnect: 100 })
this.timeout(60000);
// First request will mark server failed
memcached.get('idontcare', function(err) {
assert.throws(function() { throw err }, /connect ECONNREFUSED/);
// Wait 10ms, server should be back online
setTimeout(function() {
// Second request will mark server dead
memcached.get('idontcare', function(err) {
assert.throws(function() { throw err }, /connect ECONNREFUSED/);
// Third request should find no servers
memcached.get('idontcare', function(err) {
assert.throws(function() { throw err }, /Server not available/);
// Give enough time for server to reconnect
setTimeout(function() {
// Server should be reconnected, but expect ECONNREFUSED
memcached.get('idontcare', function(err) {
assert.throws(function() { throw err }, /connect ECONNREFUSED/);
assert.deepEqual(memcached.issues[server].failures,
memcached.issues[server].config.failures);
memcached.end();
done();
});
}, 150);
});
});
},10);
});
});
it('should return error on connection timeout', function(done) {
// Use a non routable IP
var server = '10.255.255.255:1234'
, memcached = new Memcached(server, {
retries: 0,
timeout: 100,
idle: 1000,
failures: 0 });
memcached.get('idontcare', function(err) {
assert.throws(function() { throw err }, /Timed out while trying to establish connection/);
memcached.end();
done();
});
});
it('should remove connection when idle', function(done) {
var memcached = new Memcached(common.servers.single, {
retries: 0,
timeout: 100,
idle: 100,
failures: 0 });
memcached.get('idontcare', function(err) {
assert.deepEqual(memcached.connections[common.servers.single].pool.length, 1);
setTimeout(function() {
assert.deepEqual(memcached.connections[common.servers.single].pool.length, 0);
memcached.end();
done();
}, 100);
});
});
it('should remove server if error occurs after connection established', function(done) {
var memcached = new Memcached(common.servers.single, {
poolSize: 1,
retries: 0,
timeout: 1000,
idle: 5000,
failures: 0 });
// Should work fine
memcached.get('idontcare', function(err) {
assert.ifError(err);
// Fake an error on the connected socket which should mark server failed
var S = memcached.connections[common.servers.single].pool.pop();
S.emit('error', new Error('Dummy error'));
memcached.get('idontcare', function(err) {
assert.throws(function() { throw err; }, /Server not available/);
done();
});
});
});
});