Commit 9cf8f0a8 authored by wanderer's avatar wanderer 💬

added posts

parent 2b1a9378
Pipeline #131237096 failed with stage
in 53 seconds
title: Exploring Ethereum's state trie with Node.js
date: 2014-05-21 13:22:09
number: 1
comments: true
If you not familiar with ethereum start [here]( In this post I will attempt to read [cpp-ethereum's]( state db give a root using Node.js and the [merkle-patricia-tree]( module. The merkle-patricia-tree module is hot off the press so there might be bugs. If you find any please let me know. The State DB is like a global ledger; It stores balances of all the accounts was well as the code for all of the contracts and their respective balances.
The global state is stored in a modified merkle patricia tree (trie), which you can read more about in the [yellow paper]( or on the [wiki]( The important part is understanding what it is suppose to do, which is "The core of the trie, and its sole requirement in terms of the protocol specification is to provide a single 32-byte value that identifies a given set of key-value pairs."
## Reading the db
cpp-ethereum stores the trie in a level db. On linux the path to the state db is ~/.ethereum/state. Let try just opening the database and seeing what we get. To do this we are going to need install the levelup module `npm install levelup`
We are going to look up one of the state roots. You can get them from althezero. In the example I'm using the genesis root hash from the latest cpp code from the dev branch. Looking up the root should give the top node in the trie.
var levelup = require('levelup');
var db = levelup('./.ethereum/state');
//the genesis state root
var root = '12582945fc5ad12c3e7b67c4fc37a68fc0d52d995bb7f7291ff41a2739a7ca16';
//Note: we are doing everything using binary encoding.
db.get(new Buffer(root, 'hex'), {
encoding: 'binary'
}, function (err, value) {
We should get a print out of a Buffer... not necessarily usefully. Each node in the trie is `rlp` encoded, so let decoded the node. To do that we are going to need to install the [rlp module]( `npm install rlp`. and add it the code
var rlp = require('rlp');
db.get(new Buffer(root, 'hex'), {
encoding: 'binary'
}, function (err, value) {
var decoded = rlp.decode(value);
Yay we got one node! (hopefully). Now the node decoded should print out an array of buffers. In our case we should get a branch node with is an array of the length 17. 1-16 are more nodes and 17 is a value.
## Looking up Values in the Trie
Now lets try to look up a key. After looking up the root node finding a key is as simple as following the key path to the right leaf. I'm skimming over the details, but you can read the specs [here]( First install `npm install merkle-patricia-tree`. This module allows you to read and manipulate the trie. Lets try looking up Gav's, our patron saint of cpp programming, account. His address is "8a40bfaa73256b60764c1bf40675a99083efb075".
var Trie = require('merkle-patricia-tree');
var rlp = require('rlp');
var levelup = require('levelup');
var db = levelup('/home/null/.ethereum/state');
//the genesis state root
var root = '12582945fc5ad12c3e7b67c4fc37a68fc0d52d995bb7f7291ff41a2739a7ca16';
var trie = new Trie(db, root);
//gav's address
var gav = new Buffer('8a40bfaa73256b60764c1bf40675a99083efb075', 'hex');
trie.get(gav, function (err, val) {
var decoded = rlp.decode(val);
The accounts are rlp encoded before they are stored so we need to decoded them after we retrieve them. After decoding them we should get an array of the length 4. Which are the four fields in an account, `balance`, `nonce`, `stateRoot` and `codeHash`
## Streaming the Trie
The Last thing we want to do is to read out the entire trie. This might be handy for debugging or just poking around for fun. To do this we are going to use `streams`. If your not familiar with node streams [here]( is a really fun place to get stated. Here the on `data` event returns an `object`that has two propeties; the `key` and the `value`. Both should be buffers.
var Trie = require('merkle-patricia-tree'),
rlp = require('rlp'),
levelup = require('levelup');
var db = levelup('/home/null/.ethereum/state');
var root = "12582945fc5ad12c3e7b67c4fc37a68fc0d52d995bb7f7291ff41a2739a7ca16"
var trie = new Trie(db, root);
//get a read stream object
var stream = trie.createReadStream();
stream.on('data', function (data) {
console.log('key:' + data.key.toString('hex'));
//accouts are rlp encoded
var decodedVal = rlp.decode(data.value);
stream.on('end', function (val) {
console.log('done reading!');
This should read out all the account's address and the account's contents in the trie.
## The End?
[merkle-patricia-tree]( module can do more than what I went over here. It can create, update and delete values too! amazing! And if you actually read all the way to the bottom, thanks for reading.
layout: post
title: Validating GeoJson from the command line
date: 2014-05-26 13:22:09
categories: geo cli nodejs
number: 2
This week I add a CLI to [geojson validation module]( and will be going over how to use it in this post. To start, install it `npm install geojson-validation -g`.
Next you need some geoJSON. We will use [this]( for the example. Now to check if it is valid simple run `curl | gjv`. If it was invalid `gjv` would print out the errors else it will just tell you its valid.
You can also validate local files like this `gjv file1 files2 ...`
title: Creating Ethereum contracts and verifying messages with node.js
date: 2014-06-14 23:39:25
categories: ethereum
number: 3
In this article I'll be covering ethereum transaction creation and verifying transactions. First of all; whats the rational for creating transaction with with javascript? Well I, as always just want to explore and play around with ethereum. But I imagine this would be usefull in testing compilers and even more usefull, create contracts programmatically. So lets get started! :D
First of all we will be using [ethereumjs-lib]( in this post. You can install it from npm `npm install ethereumjs-lib` or directly from git `npm install git+`
## Creating a transaction
After that let create a new file and put the following in it. For the those how are cut and paste handicap here is the [full example](
var Ethereum = require("ethereum-lib");
var Transaction = Ethereum.Transaction;
//create a blank transaction
var tx = new Transaction();
// So now we have created a blank transaction but Its not quiet valid yet. We
// need to add some things to it. Lets start with
tx.nonce = 0;
tx.gasPrice = 100;
tx.gasLimit = 1000; = 0; //Sending to 0, therefore we are creating a contract
tx.value = 0; = "7f4e616d65526567000000000000000000000000000000000000000000000000003057307f4e616d6552656700000000000000000000000000000000000000000000000000573360455760415160566000396000f20036602259604556330e0f600f5933ff33560f601e5960003356576000335700604158600035560f602b590033560f60365960003356573360003557600035335700";
Hopefull this is pretty self explanatory. `nonce` is the number of transaction that the sending account has sent. `gasPrice` is the amount you are will to pay for gas. `gasLimit` if the amount of gas you are will to spend. `to` is the address we are sending to, in this case the zero address which creates a new contract. If we where to put a valid address here this account would become a message instead of a contract. To check which type of transaction you have, you can do
tx.type === "message"
tx.type === "contract"
//type updates every time you change
`value` is the is the amount you are sending and `data` is data that you are sending in this case it is initialization code for a contract.
The data is a Name Registrar contract. You can use one of the compilers to generate the code. The two main ones are lllc which is compiled with [cpp-ethereum]( and [serpent]( Ok now we have a transaction with data. Now we need to sign it. To do this we will need a private key. There are several way to acquire one but for now we are just going to steal one from [AlethZero]( That way I know that I'm create valid transaction that actually has the ether that this transaction is says that it has. If you have AlethZero running. You can select `tools>export selected key`, and the copy the private key out. Here is mine.
![exporting private key](
var privateKey = new Buffer("e331b6d69882b4cb4ea581d88e0b604039a3de5967688d3dcffdd2270c0fd109");
Now we have a signed transaction, but for it to be valid, the account that we signed it with needs to have a certain amount of wei in to. To see how much that this account needs we can use `tx.getUpFrontCost`
console.log("Total Amount of wei needed:" + tx.getUpFrontCost());
this gives the amount in wei that the account needs to have. If your wondering how this is caculated it is
- data lenght in bytes * 5
- \+ 500 Default transaction fee
- \+ gasLimit * gasPrice
Lets serialize the transaction
console.log("---Serialized TX----");
console.log(tx.serialize().toString("hex")); //serialize returns a buffer
Now that we have the serialized transaction we can get AlethZero to except it by selecting `debug>inject transaction` and pasting in the transaction serialization. It should then show up in the pending transaction pane.
![inject transaction](
## Parsing & Validating transactions
If you have a transaction that you want to verify you can parse it. If you got it directly from the network it will be rlp encoded. You can decode it using the [rlp module]( After decoding it that you should have something like this.
var rawTx = [
var tx = new Transaction(rawTx);
Note: `rlp.decode` will actully produce an array of buffers `new Transaction` will take either and array of buffers or and array of hex strings. So assuming that you were able to parse the transaction, we will now get the sender's address
console.log("Senders Address" + tx.getSenderAddress());
Cool now we know who sent the tx! Lets verify the signature to make sure it not some poser.
console.log("Signature Checks out!");
And hopefull its verified. For the transaction to be total valid we would also need to check the account of the sender and see if they have at least `tx.getUpFrontCost()`.
And that is all for now I hope you have enjoyed the read. You can view the documentation for transaction [here]( to see all the exciting things it can do. Let me know if you have any questions or comments.
layout: post
title: How to run contracts and create traces with Node.js
date: 2014-08-12 00:00:00
categories: ethereum nodejs code
number: 4
A valid state transition is one which comes about through a transaction. Formally:
σ<sub>t+1</sub> ≡ Υ(σ<sub>t</sub> , T )
where Υ is the Ethereum state transition function. In ☰thereum, Υ, together with σ are <b>considerably more powerful then any existing comparable system;</b> Υ allows components to carry out arbitrary computation, while σ allows components to store arbitrary state between transactions.
- The Yellow Paper
At the core of Etheruem is the state transition function. Within the this function lies the ☰thereum ∇irtual Machine which upon ☰∇M code runs. In this post we will be running transactions thourgh the VM and looking at the results. A note on Terminology: the VM as descibed in the yellow paper is a subset of the state transition function. With an executioner that manipulates accounts and sets up the enviorment for the VM. Here we will just refer to the state transition function as the VM although this may not be techicanally correct. I did not like having an executioner in my code. It's a bit too barbaric for my taste. Altogether the VM in this sense handles all changes done to the state, which wholly resided in a trie.
This post will explore creating transactions and processing them with the VM. You can find all the code for this post [here](;
## Running Contracts
To get stated we will be using two libraries. `async` and [`ethereum-lib`](
`npm install async`
`npm install ethereum-lib`
First import the necessary libraries and initailize some varibles.
var async = require('async'),
Ethereum = require('ethereum-lib'),
VM = Ethereum.VM,
Account = Ethereum.Account,
Transaction = Ethereum.Transaction,
Trie = Ethereum.Trie,
rlp = Ethereum.rlp
utils = Ethereum.utils;
//creating a trie that just resides in memory
var stateTrie = new Trie();
//create a new VM instance
var vm = new VM(stateTrie);
//we will use this later
var storageRoot;
//the private/public key pare. used to sign the transactions and generate the addresses
var secretKey = '3cd7232cd6f3fc66a57a6bedc1a8ed6c228fff0a327e169c2bcc5e869ed49511';
var publicKey = '0406cc661590d48ee972944b35ad13ff03c7876eae3fd191e8a2f77311b0a3c6613407b5005e63d7d8d76b89d5f900cde691497688bb281e07a5052ff61edebdc0';
Lets set up the state trie. We need to give the sender's account enougth wei to send the transaction and run the code.
//sets up the initial state and runs the callback when complete
function setup(cb) {
//the address we are sending from
var address = utils.pubToAddress(new Buffer(publicKey, 'hex'));
//create a new account
var account = new Account();
//give the account some wei.
//This needs to be a `Buffer` or a string. all strings need to be in hex.
account.balance = 'f00000000000000000';
//store in the trie
stateTrie.put(address, account.serialize(), cb);
Next we will create a transaction and process it. If you want to know more about generating transaction see this [post]( In this example we will forgo generating a transaction from scratch and just use the some json I generated earlier. `Transaction` have a `toJSON` method if you want to do the same. This transaction contains the initializtion code for the [name register](
var rawTx = {
nonce: '00',
gasPrice: '09184e72a000',
gasLimit: '2710',
data: '7f4e616d65526567000000000000000000000000000000000000000000000000003055307f4e616d6552656700000000000000000000000000000000000000000000000000557f436f6e666967000000000000000000000000000000000000000000000000000073661005d2720d855f1d9976f88bb10c1a3398c77f5573661005d2720d855f1d9976f88bb10c1a3398c77f7f436f6e6669670000000000000000000000000000000000000000000000000000553360455560df806100c56000396000f3007f726567697374657200000000000000000000000000000000000000000000000060003514156053576020355415603257005b335415603e5760003354555b6020353360006000a233602035556020353355005b60007f756e72656769737465720000000000000000000000000000000000000000000060003514156082575033545b1560995733335460006000a2600033545560003355005b60007f6b696c6c00000000000000000000000000000000000000000000000000000000600035141560cb575060455433145b1560d25733ff5b6000355460005260206000f3'
Since running a transaction in the VM is asynchronous lets wrap this in a function which we will use later. Running this transaction should create a new name register contract. It is important to note that `vm.runTx()` would also need a `Block` a the second parameter if the EVM code where to use any opcodes that accessed the block propeties.
//runs a transaction through the vm
function runTx(raw, cb) {
//create a new transaction out of the json
var tx = new Transaction(raw);
tx.sign(new Buffer(secretKey, 'hex'));
//run the tx
vm.runTx(tx, function(err, results) {
var createdAddress = results.createdAddress;
//log some results
console.log('gas used: ' + results.gasUsed.toString());
if (createdAddress) console.log('address created: ' + createdAddress.toString('hex'));
`vm.runTx` 1) checks if the sending `account` has enough wei to run then 2) runs the code and lastly 3) saves the changed accounts in the trie. It uses a callback to returns the results of running the transaction. We are only going to log some of the results here but if you are curious george you can look at the [docs](
Now lets actully run this!
//run the asynchronous functions in series
async.apply(runTx, rawTx)
So you should see how much gas the transaction used and the addrres of the created contract `c8b97e77d29c5ccb5e9298519c707da5fb83c442`. But not too exciting yet. Lets make sure that things worked. The VM should have created a new account for the contract in the trie. Lets make sure.
function checkResults(cb) {
var createdAddress = new Buffer('692a70d2e424a56d2c6c27aa97d1a86395877b3a', 'hex');
//fetch the new account from the trie.
stateTrie.get(createdAddress, function(err, val) {
var account = new Account(val);
//we will use this later! :)
storageRoot = account.stateRoot;
console.log('nonce: ' + account.nonce.toString('hex'));
console.log('blance in wei: ' + account.balance.toString('hex'));
console.log('stateRoot: ' + storageRoot.toString('hex'));
console.log('codeHash:' + account.codeHash.toString('hex'));
Add this to `async.series` like so
async.apply(runTx, rawTx),
And run. We should see the details of the new created contract. Next thing we can do is send a message to our newly created contract. Lets create another transaction to do so. This transaction should register the sending address as "null_radix". The data field here is the RLP of pad_32_bytes('register') and pad_32_bytes('null_radix')
//This transaction should register the sending address as "null_radix"
var rawTx2 = {
nonce: '01',
gasPrice: '09184e72a000',
gasLimit: '2710',
to: '692a70d2e424a56d2c6c27aa97d1a86395877b3a',
data: '72656769737465720000000000000000000000000000000000000000000000006e756c6c5f726164697800000000000000000000000000000000000000000000'
//run everything
async.apply(runTx, rawTx),
async.apply(runTx, rawTx2),
So if everything went right we should have "null_radix" stored at "0x9bdf9e2cc4dfa83de3c35da792cdf9b9e9fcfabd". To see this we need to print out the name register's storage trie.
//reads and prints the storage of a contract
function readStorage(cb) {
//we need to create a copy of the state root
var storageTrie = stateTrie.copy();
//Since we are using a copy we won't change the root of `stateTrie`
storageTrie.root = storageRoot;
var stream = storageTrie.createReadStream();
//prints all of the keys and values in the storage trie
stream.on('data', function(data) {
console.log('key: ' + data.key.toString('hex'));
//remove the 'hex' if you want to see the ascii values
console.log('Value: ' + rlp.decode(data.value).toString('hex'));
stream.on('end', cb);
//and finally
//run everything
async.apply(runTx, rawTx),
async.apply(runTx, rawTx2),
After running you should see the contents of the contract's storage.
## Debugging with traces
Lastly lets create a trace of the EMV code. This can be very usefully for debugging (and deherbing, don't smoke and code m'kay?) contracts.
The VM provides a simple hook for each step the VM takes while running EVM code.
//runs on each opcode
vm.onStep = function (info, done) {
//prints the program counter, the current opcode and the amount of gas left
console.log('[vm] ' + info.pc + ' Opcode: ' + info.opcode + ' Gas: ' + info.gasLeft.toString());
//prints out the current stack
info.stack.forEach(function (item) {
console.log('[vm] ' + item.toString('hex'));
//important! call `done` when your done messing around
Now when you run you should see a complete trace. `onStep` provides an object that contians all the information on the current state of the `VM`.
the `vm` also provides a `runBlock` function that will process a `block` as well as a few other usefull functions.
And now I'm out of stuff to say.
title: Merklizing ASTs
date: 2015-12-29 13:22:09
categories: ethereum ast webassembly
number: 5
*Special thanks to [Juan Benet]( for mentioning this idea at Devcon 1.*
Recently I wrote a draft of [EIP 105]( which propose using a subset of webassembly as Ethereum’s VM. If you aren’t aware webassembly “is a new, portable, size- and load-time-efficient format suitable for compilation to the web.” One interesting note about Webassembly doesn’t compile to linear byte code. Instead it uses an [Abstract Syntax Tree (AST)]( This might not surprise you if you have an experience with LLVM IR. But for me it was a new concept to have bytecode in this form.
## What is an AST?
It is just a tree repesentation of some code. Each node of the AST represents an expression. Each function body consists of exactly one expression. Compared to the source code, an AST does not include certain elements, such as inessential punctuation and delimiters (braces, semicolons, parentheses, etc.).
To give you a better idea here is a textual representation of some webassembly code using [s-expressions](
;; Recursive factorial
(func $factorial (param $i i64) (result i64)
(if_else (i64.eq (get_local $i) (i64.const 0))
(i64.const 1)
(i64.mul (get_local $i) (call $factorial (i64.sub (get_local $i) (i64.const 1))))))
The AST of the above could be displayed like
![AST example](images/Merklizing%20ASTs.svg)
figure 1 - an AST
This can then be [serialized]( and sent across the wire.
## Why use an AST?
The [rationale that webassembly]( gives is
* Trees allow a smaller binary encoding: [JSZap][], [Slim Binaries][].
* [Polyfill prototype][] shows simple and efficient translation to asm.js.
[Slim Binaries]:
[Polyfill prototype]:
Some auxiliary reason might be:
* Effective to JIT
* Code Deduplication
The idea has a bit of an interesting history. It appears that [Michael Franz]( first used the idea to compress java bytecode in a paper on Slim Binaries. The slim binaries were also implemented in the [Oberon OS](
In addition to the Slim Binaries paper here are some more papers if you are interested in the subject.
* [Adaptive Compression of Syntax Trees andIterative Dynamic Code Optimization:Two Basic Technologies for Mobile-Object Systems](
* [A Tree-Based Alternative to Java Byte-Codes](
## Merklized ASTs
So another fun thing to do with AST is to merklize them! To review; a [merkle tree]( is just a tree which links its nodes together by using the cryptographic hashes of the nodes.
![a merkle tree](images/640px-Hash_Tree.svg.png)
The result is one root hash that points to the root node. This allows for efficient and secure verification of the contents of large data structures. To turn a AST into Merkle AST all you have walk from the leaf nodes up to the root hashing each node along the path.
## Efficiency
If you look at figure one you can count 19 nodes. This is a fairly small program and the number of nodes in a larger program can add up fast. The computation time needed to merklize larger AST might start to add up rather fast. One method to increase efficiency would be to store entire subroutines in a single node and only create merkle tree branches at the point where the AST has a branch condition. For example see figure 3.
figure 3 - nodes contain entire subroutines
In figure 3 the green blocks represent subroutines that are in a single node. Note how the `if else` still forms a block by itself since it is a branch condition. This also provide an nice opportunity for parallelization. The interpreter or JIT can run the first branch while the next branch is still being fetched.
## Why Merkle ASTs?
One reason is code security. It would make DLL hijack impossible. Of course how you would use DLL’s would be a bit different. Instead of open a dynamic link by a name you would reference the root hash of the routine you wanted to use. This way you have 100% confidence that you're getting the code that you want.
Have you ever had a software problem that you googled and found that 100s of forum posts with the same problem as you but none of their solutions worked for you? This is maybe in part caused by the fact you computer is in a different configuration or ‘state’ then thiers. If we had secure merklized code this would a lot less common since the state of any given piece of software could be immediately determined or set by the root hash.
Perhaps one the most convincing reason is bandwidth saving and massive code deduplication. How many nearly-the-same versions of libraries you have on your harddrive? Or how many time the same function and routine is duplicated through programs?
Let's say you have you already have the green nodes since they are common subroutines. You would only have to download the orange nodes. Where this bandwidth saving could be very important is things like Ethereum light clients but also for general computation. As Web Pages begin to more and more resemble apps the larger their code size becomes. In the age of ephemeral webapps code is repeatedly downloaded many times. How many times do you think you have downloaded jquery? Couple this with a peer-to-peer distribution method like [IPFS]( and I think you would have a very efficient system.
\ No newline at end of file
This diff is collapsed.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment