Commit b30da110 authored by David Bakin's avatar David Bakin
Browse files

Initial checkin of project development notes

parents
###############################################################################
# Set default behavior to automatically normalize line endings.
###############################################################################
* text=auto
###############################################################################
# Set default behavior for command prompt diff.
#
# This is need for earlier builds of msysgit that does not have it on by
# default for csharp files.
# Note: This is only used by command line
###############################################################################
#*.cs diff=csharp
###############################################################################
# Set the merge driver for project and solution files
#
# Merging from the command prompt will add diff markers to the files if there
# are conflicts (Merging from VS is not affected by the settings below, in VS
# the diff markers are never inserted). Diff markers may cause the following
# file extensions to fail to load in VS. An alternative would be to treat
# these files as binary and thus will always conflict and require user
# intervention with every merge. To do so, just uncomment the entries below
###############################################################################
#*.sln merge=binary
#*.csproj merge=binary
#*.vbproj merge=binary
#*.vcxproj merge=binary
#*.vcproj merge=binary
#*.dbproj merge=binary
#*.fsproj merge=binary
#*.lsproj merge=binary
#*.wixproj merge=binary
#*.modelproj merge=binary
#*.sqlproj merge=binary
#*.wwaproj merge=binary
###############################################################################
# behavior for image files
#
# image files are treated as binary by default.
###############################################################################
#*.jpg binary
#*.png binary
#*.gif binary
###############################################################################
# diff behavior for common document formats
#
# Convert binary document formats to text before diffing them. This feature
# is only available from the command line. Turn it on by uncommenting the
# entries below.
###############################################################################
#*.doc diff=astextplain
#*.DOC diff=astextplain
#*.docx diff=astextplain
#*.DOCX diff=astextplain
#*.dot diff=astextplain
#*.DOT diff=astextplain
#*.pdf diff=astextplain
#*.PDF diff=astextplain
#*.rtf diff=astextplain
#*.RTF diff=astextplain
## Ignore Visual Studio temporary files, build results, and
## files generated by popular Visual Studio add-ons.
##
## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore
# User-specific files
*.rsuser
*.suo
*.user
*.userosscache
*.sln.docstates
# User-specific files (MonoDevelop/Xamarin Studio)
*.userprefs
# Mono auto generated files
mono_crash.*
# Build results
[Dd]ebug/
[Dd]ebugPublic/
[Rr]elease/
[Rr]eleases/
x64/
x86/
[Ww][Ii][Nn]32/
[Aa][Rr][Mm]/
[Aa][Rr][Mm]64/
bld/
[Bb]in/
[Oo]bj/
[Oo]ut/
[Ll]og/
[Ll]ogs/
# Visual Studio 2015/2017 cache/options directory
.vs/
# Uncomment if you have tasks that create the project's static files in wwwroot
#wwwroot/
# Visual Studio 2017 auto generated files
Generated\ Files/
# MSTest test Results
[Tt]est[Rr]esult*/
[Bb]uild[Ll]og.*
# NUnit
*.VisualState.xml
TestResult.xml
nunit-*.xml
# Build Results of an ATL Project
[Dd]ebugPS/
[Rr]eleasePS/
dlldata.c
# Benchmark Results
BenchmarkDotNet.Artifacts/
# .NET Core
project.lock.json
project.fragment.lock.json
artifacts/
# ASP.NET Scaffolding
ScaffoldingReadMe.txt
# StyleCop
StyleCopReport.xml
# Files built by Visual Studio
*_i.c
*_p.c
*_h.h
*.ilk
*.meta
*.obj
*.iobj
*.pch
*.pdb
*.ipdb
*.pgc
*.pgd
*.rsp
*.sbr
*.tlb
*.tli
*.tlh
*.tmp
*.tmp_proj
*_wpftmp.csproj
*.log
*.vspscc
*.vssscc
.builds
*.pidb
*.svclog
*.scc
# Chutzpah Test files
_Chutzpah*
# Visual C++ cache files
ipch/
*.aps
*.ncb
*.opendb
*.opensdf
*.sdf
*.cachefile
*.VC.db
*.VC.VC.opendb
# Visual Studio profiler
*.psess
*.vsp
*.vspx
*.sap
# Visual Studio Trace Files
*.e2e
# TFS 2012 Local Workspace
$tf/
# Guidance Automation Toolkit
*.gpState
# ReSharper is a .NET coding add-in
_ReSharper*/
*.[Rr]e[Ss]harper
*.DotSettings.user
# TeamCity is a build add-in
_TeamCity*
# DotCover is a Code Coverage Tool
*.dotCover
# AxoCover is a Code Coverage Tool
.axoCover/*
!.axoCover/settings.json
# Coverlet is a free, cross platform Code Coverage Tool
coverage*.json
coverage*.xml
coverage*.info
# Visual Studio code coverage results
*.coverage
*.coveragexml
# NCrunch
_NCrunch_*
.*crunch*.local.xml
nCrunchTemp_*
# MightyMoose
*.mm.*
AutoTest.Net/
# Web workbench (sass)
.sass-cache/
# Installshield output folder
[Ee]xpress/
# DocProject is a documentation generator add-in
DocProject/buildhelp/
DocProject/Help/*.HxT
DocProject/Help/*.HxC
DocProject/Help/*.hhc
DocProject/Help/*.hhk
DocProject/Help/*.hhp
DocProject/Help/Html2
DocProject/Help/html
# Click-Once directory
publish/
# Publish Web Output
*.[Pp]ublish.xml
*.azurePubxml
# Note: Comment the next line if you want to checkin your web deploy settings,
# but database connection strings (with potential passwords) will be unencrypted
*.pubxml
*.publishproj
# Microsoft Azure Web App publish settings. Comment the next line if you want to
# checkin your Azure Web App publish settings, but sensitive information contained
# in these scripts will be unencrypted
PublishScripts/
# NuGet Packages
*.nupkg
# NuGet Symbol Packages
*.snupkg
# The packages folder can be ignored because of Package Restore
**/[Pp]ackages/*
# except build/, which is used as an MSBuild target.
!**/[Pp]ackages/build/
# Uncomment if necessary however generally it will be regenerated when needed
#!**/[Pp]ackages/repositories.config
# NuGet v3's project.json files produces more ignorable files
*.nuget.props
*.nuget.targets
# Microsoft Azure Build Output
csx/
*.build.csdef
# Microsoft Azure Emulator
ecf/
rcf/
# Windows Store app package directories and files
AppPackages/
BundleArtifacts/
Package.StoreAssociation.xml
_pkginfo.txt
*.appx
*.appxbundle
*.appxupload
# Visual Studio cache files
# files ending in .cache can be ignored
*.[Cc]ache
# but keep track of directories ending in .cache
!?*.[Cc]ache/
# Others
ClientBin/
~$*
*~
*.dbmdl
*.dbproj.schemaview
*.jfm
*.pfx
*.publishsettings
orleans.codegen.cs
# Including strong name files can present a security risk
# (https://github.com/github/gitignore/pull/2483#issue-259490424)
#*.snk
# Since there are multiple workflows, uncomment next line to ignore bower_components
# (https://github.com/github/gitignore/pull/1529#issuecomment-104372622)
#bower_components/
# RIA/Silverlight projects
Generated_Code/
# Backup & report files from converting an old project file
# to a newer Visual Studio version. Backup files are not needed,
# because we have git ;-)
_UpgradeReport_Files/
Backup*/
UpgradeLog*.XML
UpgradeLog*.htm
ServiceFabricBackup/
*.rptproj.bak
# SQL Server files
*.mdf
*.ldf
*.ndf
# Business Intelligence projects
*.rdl.data
*.bim.layout
*.bim_*.settings
*.rptproj.rsuser
*- [Bb]ackup.rdl
*- [Bb]ackup ([0-9]).rdl
*- [Bb]ackup ([0-9][0-9]).rdl
# Microsoft Fakes
FakesAssemblies/
# GhostDoc plugin setting file
*.GhostDoc.xml
# Node.js Tools for Visual Studio
.ntvs_analysis.dat
node_modules/
# Visual Studio 6 build log
*.plg
# Visual Studio 6 workspace options file
*.opt
# Visual Studio 6 auto-generated workspace file (contains which files were open etc.)
*.vbw
# Visual Studio LightSwitch build output
**/*.HTMLClient/GeneratedArtifacts
**/*.DesktopClient/GeneratedArtifacts
**/*.DesktopClient/ModelManifest.xml
**/*.Server/GeneratedArtifacts
**/*.Server/ModelManifest.xml
_Pvt_Extensions
# Paket dependency manager
.paket/paket.exe
paket-files/
# FAKE - F# Make
.fake/
# CodeRush personal settings
.cr/personal
# Python Tools for Visual Studio (PTVS)
__pycache__/
*.pyc
# Cake - Uncomment if you are using it
# tools/**
# !tools/packages.config
# Tabs Studio
*.tss
# Telerik's JustMock configuration file
*.jmconfig
# BizTalk build output
*.btp.cs
*.btm.cs
*.odx.cs
*.xsd.cs
# OpenCover UI analysis results
OpenCover/
# Azure Stream Analytics local run output
ASALocalRun/
# MSBuild Binary and Structured Log
*.binlog
# NVidia Nsight GPU debugger configuration file
*.nvuser
# MFractors (Xamarin productivity tool) working folder
.mfractor/
# Local History for Visual Studio
.localhistory/
# BeatPulse healthcheck temp database
healthchecksdb
# Backup folder for Package Reference Convert tool in Visual Studio 2017
MigrationBackup/
# Ionide (cross platform F# VS Code tools) working folder
.ionide/
# Fody - auto-generated XML schema
FodyWeavers.xsd
\ No newline at end of file
# CMakeList.txt : Top-level CMake project file, do global configuration
# and include sub-projects here.
#
cmake_minimum_required (VERSION 3.8)
project ("edu-bitcoin-staging")
# Include sub-projects.
add_subdirectory ("edu-bitcoin-staging")
{
"configurations": [
{
"name": "x64-Debug-MSVC",
"generator": "Ninja",
"configurationType": "Debug",
"inheritEnvironments": [ "msvc_x64_x64" ],
"buildRoot": "${projectDir}\\out\\build\\${name}",
"installRoot": "${projectDir}\\out\\install\\${name}",
"cmakeCommandArgs": "",
"buildCommandArgs": "",
"ctestCommandArgs": ""
},
{
"name": "x64-Release-MSVC",
"generator": "Ninja",
"configurationType": "RelWithDebInfo",
"buildRoot": "${projectDir}\\out\\build\\${name}",
"installRoot": "${projectDir}\\out\\install\\${name}",
"cmakeCommandArgs": "",
"buildCommandArgs": "",
"ctestCommandArgs": "",
"inheritEnvironments": [ "msvc_x64_x64" ],
"variables": []
},
{
"name": "x64-Debug-CLang",
"generator": "Ninja",
"configurationType": "Debug",
"buildRoot": "${projectDir}\\out\\build\\${name}",
"installRoot": "${projectDir}\\out\\install\\${name}",
"cmakeCommandArgs": "",
"buildCommandArgs": "",
"ctestCommandArgs": "",
"inheritEnvironments": [ "clang_cl_x64" ],
"variables": []
},
{
"name": "x64-Release-CLang",
"generator": "Ninja",
"configurationType": "Release",
"buildRoot": "${projectDir}\\out\\build\\${name}",
"installRoot": "${projectDir}\\out\\install\\${name}",
"cmakeCommandArgs": "",
"buildCommandArgs": "",
"ctestCommandArgs": "",
"inheritEnvironments": [ "clang_cl_x64" ],
"variables": []
}
]
}
\ No newline at end of file
MIT License
Copyright (c) 2021 David Bakin
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
# **BlockSifter**: Repository for an education-oriented Bitcoin Core "clone"
Current goal: **BlockSifter**, A flexible playpen for inspecting the Bitcoin Blockchain down to the most primitive data structure. Modular (plugin) architecture to make it easy to experiment with different implementations of each server-side "primitive".
## Ambient Services
List of _ambient_ services:
| Service | Lifetime (w.r.t. component) |
| :----------------------------------------------------------- | :----------------------------------------------------------- |
| Service Locator | Never change |
| Alerting (user messages) | Never change |
| Database (get connection) | Never change |
| Event/Task system | Never change |
| Workflow system | Never change |
| Heartbeat/Health | Never change |
| Logger | _Service_ never changes, but ambient behavior can be scoped |
| Metrics | _Service_ never changes, but ambient behavior can be scoped |
| Network (get connection, create socket pool, throttle | Never change |
| Memory Allocation (including heap creation) | Sometimes user may want to pass in heap to use (including for return value) |
| Thread pool (including thread pool creation, including long-lived thread creation) | Sometimes user may want to pass in the pool to use |
The "Never change" ones are _not_ ambient: You get an interface to a peer service and hold it, use it as necessary. (Except for the service locator of course, that you get when the module is initialized.)
The ambient ones can't actually be passed via thread-local to plugins (they can for built-ins, but built-ins are supposed to be the _same_ as plugins). What needs to be passed in is a kind of "caching proxy" that, when a thread's ambient changes, all the "caching proxys" are notified to update _their_ thread local. But in fact, this is a bad idea: that can be expensive, and it sounds unwieldy, and a source of bugs. Instead: make _all_ the ambient ones services.
* I was thinking of the memory allocator but there's no reason it can't be used as a service too.
Now the only tricky thing left are the "scoped" ones, e.g., scoped logger and metrics.
* For those, it seems expensive (and tedious) to pass them in at API call sites
* Maybe scoping does cross module boundaries? After all, module boundaries mostly use the task system?
* But then there's the tool/CLI case to consider ...
* So maybe, the logger/metrics systems _do_ have a way to communicate scope in some "global" way, global to thread or global to "work item".
* Work items are easier - they can _hold_ the scope they need, now that I think of it
* So maybe it's a hybrid: work item scoping can cross module boundaries but call stack scoping can't?
* **TODO:** How do _exception_ services work? Do we pass exceptions at all past module boundaries?? Maybe the answer is: **NO**, module boundaries pass only explicit status
* So modules _catch_ exceptions that would fly out of them, translate to status, and if needed, the main translates them back.
* And in fact it may be that the main routine, in this service oriented system, is so minimal and basic (like a message passing OS kernel) that it _never_ deals in exceptions and _always_ deals in status.
* By "status" I mean the exception monad.
* How does the network provider work w.r.t. different network types? E.g., you may want some services restricted by configuration/policy, e.g., bitcoin network via Tor only, while others allow other types but with _other_ restrictions, e.g., unlimited TCP/IP or TCP/IP open to localhost only)
\ No newline at end of file
# Async APIs
(Rough notes:)
Given the synchronous (usual) API to services _generate_ the async interface and implementation from it.
* Perhaps the sync interface is in some DSL (like the proto language)
* Perhaps it is simply a restricted subset of C\++, or an annotated subset of C++
* Demands simple and uniform synchronous APIs (so that the tool can be built) which is a good idea anyway
* **Ideally** should be able to generate wrappers that make the async API a lambda (cloud function)!
* And/or mix with the batch mode APIs ...
The async API will be task/event based.
* Each task will carry its cancellation token
* The tasks will have continuations - preferably not just one, but the ability to make a chain
* Secondary importance: memory efficient (not require a bunch of allocations)
* Continuations both for success _and_ failure
* Ability to _accumulate_ results, esp. failures
* Tracking ability of tasks (ability to visual current _and past_ task queues, who executed what when, who entered what into the queue with which continuation, etc.)
* Should have a way to look like a streaming API
* Batched synchronous APIs can turn into stream of task items
* Maybe not always desirable - sometimes a batch, specified with a single API call, can be worked on and results returned more efficiently than a stream. So it depends, both points-of-view are useful at different times.
\ No newline at end of file
## Available Blockchain Webservices
There may be interesting stuff available on the various websites besides blocks and transactions.
E.g., consider [blockchain.com](https://blockchain.com/api) which has available (besides a bunch of exchange information!):
* [Query API](https://www.blockchain.com/api/q) - which has some interesting singleton information (current block height, difficulty target, etc. etc.), also address and transaction lookups, price information, and some address⬌hash tools.
* [_websocket_ API](https://www.blockchain.com/api/api_websocket) which gives a stream of unconfirmed transactions and new blocks. Can also subscribe to all transactions on an address.
* typical [Blockchain Data API](https://www.blockchain.com/api/blockchain_api)
* block by hash, raw or decoded
* transaction by hash
* _blocks at a given height_
* hashes of _blocks for one specific day_
* address info, w/ or w/o transactions on that address (up to 100), singly or bulk
* also address unspent outputs and balance
* UTXOs
* [Exchange Rates API](https://www.blockchain.com/api/exchange_rates_api) - rates to⬌from fiat currency
\ No newline at end of file
# Block Fetch Schemes
Three ways to fetch blocks:
1. The genesis block is hardwired.
1. Fetch from a block storage provider
* May or may not be present
* Not fully synced
* Sparse block storage provider
1. Fetch from online service (e.g., `blockchain.info`).
* Electrum [source code](https://github.com/spesmilo/electrum/blob/4.1.5/electrum/util.py) lists a bunch of these - there may be more. Many are very similar to each other.
\ No newline at end of file
# Block Storage Schemes
Block storage can be
1. [Block Storage In File Systems](#Block-Storage-In-File-Systems), or
2. [Block Storage In Databases](#Block-Storage-In-Databases).
Furthermore, there are two ways to run such a store:
1. Complete (except for IBD).
1. Sparse (either meant to stay that way, or fill-on-demand).
* Might be meant to stay that way if you want to keep the last 10000 blocks on fast storage and the rest somewhere slow
* Or the difference might be blocks immediately ready for use vs. packed/compressed in some way.
Finally, there is an orthogonal way to run such a store:
1. All blocks always complete.
1. Some blocks pruned (to various degrees of pruning).
## Block Storage In File Systems
There are 697703 blocks on the Bitcoin (main) blockchain right now (2021-08-26). There are ~55500 blocks/year.
* (From interesting article ["The First Bitcoin Block Mined Each Year 2009-2020"](https://crypto.bi/years/).)
* Which points out that there were 67923 blocks mined in 2020-2021 - 7.75 blks/hr, or every 7m43s - due to the fast evolution of GPU and ASIC miners in this year.
* And also says least blocks mined was 2019-2020 with only 54239. "This indicates that the difficulty algorithm has adjusted ahead of mining technology advancements. During this same period, hashpower peaked at more than 110 exahashes per second, a 153% increase from one year before."
We'll reach 1,000,000 blocks in ~5+ years. And 10,000,000 blocks in ~160+ years ...
Even where we are now - ~700000 blocks - it would tax the capability of many file systems (and nearly all _tools_ like `dir`, `ls`, `rm`, etc. - not to mention archi