...
 
Commits (4)
......@@ -12,7 +12,7 @@ building:
script:
- mkdir -p $GOPATH/src/$REPO_NAME
- cp -rf $CI_PROJECT_DIR/* $GOPATH/src/$REPO_NAME/
- cd $GOPATH/src/$REPO_NAME
- cd $GOPATH/src/$REPO_NAME/app
- make build
publish:
......
Very much influences by [goenning](https://www.reddit.com/user/goenning)'s [comment](https://www.reddit.com/r/golang/comments/79wf23/where_can_i_learn_about_code_structure_and/) on [Reddit](https://reddit.com)
* [cmd](cmd): The file pertaining to running the program and creating binaries.
The codebase for Vee is organized according to the [Standard Package Layout](https://medium.com/@benbjohnson/standard-package-layout-7cdbc8391fc1)
that [Ben Johnson](https://medium.com/@benbjohnson) coined in 2016. Whenever something intentionally breaks
with that architecture, it is to be marked with a `FIXME(Name):` message
in the comments.
* [app](app): Code specific to running the application
## Testing
Vee uses the following guidelines for testing:
* [app/actions](app/actions): The input struct that is mapped from POST requests. Each action
knows how to validate itself and check whenever current user can perform that
action.
1. Unit and Integration tests are in the same folder as the code being tested.
2. Acceptance tests, when introduced, goes in a separate folder.
3. `_test.go` is for Unit tests that has their own package - Ex: http_test.go resides in the same folder as http.go, but belongs to the http_test package.
4. `_internal_test.go` is also for Unit tests belonging to the same package as the code being tested.
5. `_test.go` is prefered over `_internal_test.go` to allow us to change the internals as we like. See [this](https://medium.com/@matryer/5-simple-tips-and-tricks-for-writing-unit-tests-in-golang-619653f90742) post by Mat Ryer.
6. `_integration_test.go` is for Integration testing. To lighten the development workload, these are not run as often, as Unit testing covers _most_ of our needs.
7. Add an integration build constraint to the top of integration tests and call tests with "sudo -E /usr/local/go/bin/go test -v -cover -tags=integration ./..." to run integration tests
* [app/handlers](app/handlers): One handler per route. They receive a request and return a
response.
* [app/middlewares](app/middlewares): They wrap common functionality like authorization,
open/commit/rollback transaction, logging, fill the context with userfull
information for the handlers
### Acceptance vs Integration
To make sure we're referring to the same thing, we'll refer to Steve Freeman and Nat Pryce for the distinction between the two:
* [app/models](app/models): Just dummy models that are either used for Input, Output or
Internal
* [app/storage](app/storage): Interfaces and structs to access the storage the models. I
currently support only inmemory and postgres. InMemory is used only for unit
testing, while postgres package is used when the application is running. In
the future I plan to add mysql and mongo by just creating a new package and
implement the interfaces.
* [app/pkg/*](app/pkg/*): common packages that are not used by the storage, handlers,
middlewares, actions, etc. I also like to wrap most packages into my own
packages (example: web, jwt, oauth, dbx) so that I can expose a simpler API
to the handlers and easily exchange an internal package with another one.
> Acceptance: Does the whole system work?
>
> Integration: Does our code work against code we can’t change?
......@@ -2,6 +2,9 @@ If you're reading this, then great! It most likely means that you have healthy
interest in the project!
First thing to notice is that the project aims to use the Git Flow workflow
for working with Git. You can read more about it [here](http://nvie.com/posts/a-successful-git-branching-model/)
for working with Git. You can read more about it [here](http://nvie.com/posts/a-successful-git-branching-model/).
You would most likely also like to read about the way the code is organized,
which you canfind [here](CODE_ORGANIZATION.md).
Please feel free send a pull request!
# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.
[[projects]]
branch = "master"
name = "github.com/dustin/go-humanize"
packages = ["."]
revision = "bb3d318650d48840a39aa21a027c6630e198e626"
[[projects]]
name = "github.com/go-ini/ini"
packages = ["."]
revision = "6333e38ac20b8949a8dd68baa3650f4dee8f39f0"
version = "v1.33.0"
[[projects]]
name = "github.com/google/uuid"
packages = ["."]
......@@ -20,15 +32,71 @@
revision = "da3231b0b66e2e74cdb779f1d46c5e958ba8be27"
version = "v3.1.0"
[[projects]]
name = "github.com/minio/minio-go"
packages = [
".",
"pkg/credentials",
"pkg/encrypt",
"pkg/policy",
"pkg/s3signer",
"pkg/s3utils",
"pkg/set"
]
revision = "9e124ec59547551cb3f1324f73623bbb30650cf8"
version = "4.0.9"
[[projects]]
branch = "master"
name = "github.com/mitchellh/go-homedir"
packages = ["."]
revision = "b8bc1bf767474819792c23f32d8286a45736f1c6"
[[projects]]
name = "github.com/pkg/errors"
packages = ["."]
revision = "645ef00459ed84a119197bfb8d8205042c6df63d"
version = "v0.8.0"
[[projects]]
name = "github.com/sirupsen/logrus"
packages = ["."]
revision = "418b41d23a1bf978c06faea5313ba194650ac088"
version = "v0.8.7"
[[projects]]
branch = "release-branch.go1.10"
name = "golang.org/x/net"
packages = [
"idna",
"lex/httplex"
]
revision = "0ed95abb35c445290478a5348a7b38bb154135fd"
[[projects]]
name = "golang.org/x/text"
packages = [
"collate",
"collate/build",
"internal/colltab",
"internal/gen",
"internal/tag",
"internal/triegen",
"internal/ucd",
"language",
"secure/bidirule",
"transform",
"unicode/bidi",
"unicode/cldr",
"unicode/norm",
"unicode/rangetable"
]
revision = "f21a4dfb5e38f5895301dc265a8def02365cc3d0"
version = "v0.3.0"
[solve-meta]
analyzer-name = "dep"
analyzer-version = 1
inputs-digest = "a7ad5d22cb9589ee9f6f95d2e9f17fc143af1cfa17f6f0774048a9d231369cf0"
inputs-digest = "c22f216c185ce5145dfda60421346594ba498b121b7476d7760ab6a9d695d3e1"
solver-name = "gps-cdcl"
solver-version = 1
package fileservice
import (
"time"
"github.com/google/uuid"
vee "gitlab.com/MadsRC/vee/app"
)
//FileService represents an implementation of vee.FileService
type FileService struct {
DB vee.MetadataService
}
// File queries the metadata database and tries to retrieve a file
func (s *FileService) File(name uuid.UUID) (*vee.File, error) {
var f vee.File
_, err := s.DB.File(name)
if err != nil {
return nil, err
}
// Fetch the file metadata from the database
// Check that the file is still valid as per it's invalidat field
// Return error if invalid
// Fetch actual file from the object store
// Return file
return &f, nil
}
// CreateFile creates a file in the object store and creates the metadata.
func (s *FileService) CreateFile(name uuid.UUID, nonce [32]byte, createdat time.Time, invalidat time.Time) error {
// Upload file to object store
// Create file metadata in the database
return nil
}
package fileservice_test
import (
"errors"
"reflect"
"testing"
"github.com/google/uuid"
vee "gitlab.com/MadsRC/vee/app"
"gitlab.com/MadsRC/vee/app/fileservice"
"gitlab.com/MadsRC/vee/app/mock"
)
func TestFileService(t *testing.T) {
t.Run("File() calls", func(t *testing.T) {
// Inject our mock
var mbs mock.MetadataService
var fs fileservice.FileService
fs.DB = &mbs
//Mock MetadataService.File() call.
mbs.FileFn = func(name uuid.UUID) (*vee.File, error) {
return &vee.File{Name: name}, nil
}
_, err := fs.File(uuid.New())
if err != nil {
t.Fatalf("File() returned err: %v", err)
}
t.Run("MetadataService.File invocation", func(t *testing.T) {
//Mock MetadataService.File() call.
mbs.FileFn = func(name uuid.UUID) (*vee.File, error) {
return &vee.File{Name: name}, nil
}
_, _ = fs.File(uuid.New())
// Validate the mock
if !mbs.FileInvoked {
t.Fatal("expected File() to be invoked")
}
})
t.Run("MetadataService.File err returned back", func(t *testing.T) {
//Mock MetadataService.File() call.
theError := errors.New("DB Error")
mbs.FileFn = func(name uuid.UUID) (*vee.File, error) {
return nil, theError
}
_, err := fs.File(uuid.New())
if !reflect.DeepEqual(err, theError) {
t.Fatalf("Error was not as expected: %v", err)
}
})
})
}
// Package log implements a simple logging package, that adheres to
// the idea that there's only really 2 log levels. The idea is described well
// in Dave cheney's blog post here
// in Dave Cheney's blog post here
// https://dave.cheney.net/2015/11/05/lets-talk-about-logging
// The log package tries to make logging simple and provides a default logger
// much like Go's standard logger.
//
// By default, logs are send to os.Stdout and debugging is disbled.
// By default, logs are send to os.Stdout and debugging is disabled.
package log
import (
......
package minio
import (
"time"
"github.com/google/uuid"
"github.com/minio/minio-go"
vee "gitlab.com/MadsRC/vee/app"
)
//ObjectService represents a Minio implementation of vee.ObjectService
type ObjectService struct {
Store *minio.Client
Config vee.Config
Log vee.LogService
}
//File returns a file for a given name.
func (s *ObjectService) File(name uuid.UUID) (*vee.File, int64, error) {
var f vee.File
var err error
obj, err := s.Store.GetObject(s.Config.BucketName.String(), name.String(), minio.GetObjectOptions{})
if err != nil {
return nil, 0, err
}
f.Content = obj
stat, err := obj.Stat()
return &f, stat.Size, err
}
//CreateFile creates a file with the given values
//
// The provided file must have a correct Size field
// tells the size of the Content field. While minio
// supports setting the size as -1, S3 does not.
// S3 requires the size to be set in all usecases.
//
// Another drawback to uses -1 as size is that it
// will use multiples of 512MB RAM. Setting the size
// allows for more optimized upload.
// On underpowered hosts, or hosts with a lot of
// incoming files, using -1 as size can very easily
// lead to OOM.
func (s *ObjectService) CreateFile(f *vee.File) error {
var err error
err = s.Store.MakeBucket(s.Config.BucketName.String(), s.Config.Location)
if err != nil {
exists, err := s.Store.BucketExists(s.Config.BucketName.String())
if err == nil && exists {
s.Log.Debugf("We already own %s\n", s.Config.BucketName.String())
} else {
return err
}
} else {
s.Log.Debugf("Successfully created %s\n", s.Config.BucketName.String())
}
n, err := s.Store.PutObject(s.Config.BucketName.String(), f.Name.String(), f.Content, f.Size, minio.PutObjectOptions{})
if err != nil {
return err
}
f.Createdat = time.Now()
f.Invalidat = f.Createdat.Add(time.Second * time.Duration(s.Config.TTL))
s.Log.Debugf("Successfully uploaded %s of size %d \n", f.Name.String(), n)
return err
}
// +build integration
package minio_test
import (
"bytes"
"fmt"
"log"
"os"
"testing"
dc "github.com/fsouza/go-dockerclient"
"github.com/minio/minio-go"
"github.com/ory/dockertest"
vee "gitlab.com/MadsRC/vee/app"
veeMinio "gitlab.com/MadsRC/vee/app/minio"
"gitlab.com/MadsRC/vee/app/mock"
)
var minioClient *minio.Client
func TestMain(m *testing.M) {
pool, err := dockertest.NewPool("")
if err != nil {
log.Fatalf("Could not connect to docker: %s", err)
}
options := &dockertest.RunOptions{
Repository: "minio/minio",
Tag: "latest",
Cmd: []string{"server", "/data"},
PortBindings: map[dc.Port][]dc.PortBinding{
"9000": []dc.PortBinding{{HostPort: "9000"}},
},
Env: []string{"MINIO_ACCESS_KEY=MYACCESSKEY", "MINIO_SECRET_KEY=MYSECRETKEY"},
}
resource, err := pool.RunWithOptions(options)
if err != nil {
log.Fatalf("Could not start resource: %s", err)
}
endpoint := fmt.Sprintf("localhost:%s", resource.GetPort("9000/tcp"))
// exponential backoff-retry, because the application in the container might not be ready to accept connections yet
if err := pool.Retry(func() error {
minioClient, err = minio.New(endpoint, "MYACCESSKEY", "MYSECRETKEY", false)
if err != nil {
log.Println("Failed to create minio client:", err)
return err
}
_, err := minioClient.ListBuckets()
return err
}); err != nil {
log.Fatalf("Could not connect to docker: %s", err)
}
code := m.Run()
// You can't defer this because os.Exit doesn't care for defer
if err := pool.Purge(resource); err != nil {
log.Fatalf("Could not purge resource: %s\n", err)
}
os.Exit(code)
}
func TestCreateFile(t *testing.T) {
var logService mock.LoggerService
logService.DebugfFn = func(format string, v ...interface{}) {
return
}
obs := veeMinio.ObjectService{
Store: minioClient,
Config: vee.Config{},
Log: &logService,
}
fileData := bytes.NewBufferString("R29waGVycyBydWxlIQ==")
file := vee.File{
Content: fileData,
Size: int64(fileData.Len()),
}
err := obs.CreateFile(&file)
if err != nil {
t.Fatalf("Unexpected error: %s\n", err)
}
}
package mock
import (
"io"
"github.com/google/uuid"
vee "gitlab.com/MadsRC/vee/app"
)
......@@ -24,3 +26,69 @@ func (s *FileService) CreateFile(f *vee.File) error {
s.CreateFileInvoked = true
return s.CreateFileFn(f)
}
//MetadataService represents a mock implementation of vee.MetadataService
type MetadataService struct {
FileFn func(name uuid.UUID) (*vee.File, error)
FileInvoked bool
CreateFileFn func(f *vee.File) error
CreateFileInvoked bool
}
// File invokes the mock implementation and marks the function as invoked
func (s *MetadataService) File(name uuid.UUID) (*vee.File, error) {
s.FileInvoked = true
return s.FileFn(name)
}
// CreateFile invokes the mock implementation aand marks the function as invoked
func (s *MetadataService) CreateFile(f *vee.File) error {
s.CreateFileInvoked = true
return s.CreateFileFn(f)
}
type LoggerService struct {
PrintFn func(v ...interface{})
PrintInvoked bool
PrintfFn func(format string, v ...interface{})
PrintfInvoked bool
SetOutputFn func(w io.Writer)
SetOutputInvoked bool
GetDebugFn func() bool
GetDebugInvoked bool
SetDebugFn func(state bool)
SetDebugInvoked bool
DebugFn func(v ...interface{})
DebugInvoked bool
DebugfFn func(format string, v ...interface{})
DebugfInvoked bool
}
func (s *LoggerService) Print(v ...interface{}) {
s.PrintInvoked = true
s.PrintFn(v)
}
func (s *LoggerService) Printf(format string, v ...interface{}) {
s.PrintfInvoked = true
s.PrintfFn(format, v)
}
func (s *LoggerService) SetOutput(w io.Writer) {
s.SetOutputInvoked = true
s.SetOutputFn(w)
}
func (s *LoggerService) GetDebug() bool {
s.GetDebugInvoked = true
return s.GetDebug()
}
func (s *LoggerService) SetDebug(state bool) {
s.SetDebugInvoked = true
s.SetDebugFn(state)
}
func (s *LoggerService) Debug(v ...interface{}) {
s.DebugInvoked = true
s.DebugFn(v)
}
func (s *LoggerService) Debugf(format string, v ...interface{}) {
s.DebugfInvoked = true
s.DebugfFn(format, v)
}
package postgres
import (
"time"
"github.com/google/uuid"
"github.com/jackc/pgx"
"gitlab.com/MadsRC/vee/app"
vee "gitlab.com/MadsRC/vee/app"
)
//FileService represents a PostgreSQL implementation og vee.FileService
type FileService struct {
//MetadataService represents a PostgreSQL implementation og vee.MetadataService
type MetadataService struct {
DB *pgx.ConnPool
}
//File returns a file for a given name.
func (s *FileService) File(name uuid.UUID) (*vee.File, error) {
func (s *MetadataService) File(name uuid.UUID) (*vee.File, error) {
var f vee.File
var tmpNonce []byte
row := s.DB.QueryRow("SELECT name, nonce, createdat, invalidat FROM file WHERE name = $1", name)
......@@ -30,9 +28,9 @@ func (s *FileService) File(name uuid.UUID) (*vee.File, error) {
}
//CreateFile returns creates a file with the given values
func (s *FileService) CreateFile(name uuid.UUID, nonce [32]byte, createdat time.Time, invalidat time.Time) error {
func (s *MetadataService) CreateFile(f *vee.File) error {
_, err := s.DB.Exec("INSERT INTO file (name, nonce, createdat, invalidat) VALUES ($1, $2, $3, $4);", name, nonce[:], createdat, invalidat)
_, err := s.DB.Exec("INSERT INTO file (name, nonce, createdat, invalidat) VALUES ($1, $2, $3, $4);", f.Name, f.Nonce[:], f.Createdat, f.Invalidat)
return err
}
// +build integration
package postgres_test
import (
"log"
"os"
"strconv"
"testing"
"time"
"github.com/google/uuid"
"github.com/jackc/pgx"
"github.com/ory/dockertest"
vee "gitlab.com/MadsRC/vee/app"
"gitlab.com/MadsRC/vee/app/postgres"
)
var dbPool *pgx.ConnPool
func TestMain(m *testing.M) {
pool, err := dockertest.NewPool("")
if err != nil {
log.Fatalf("Could not connect to docker: %s", err)
}
// pulls an image, creates a container based on it and runs it
resource, err := pool.Run("postgres", "10", []string{"POSTGRES_PASSWORD=postgres"})
if err != nil {
log.Fatalf("Could not start resource: %s", err)
}
// exponential backoff-retry, because the application in the container might not be ready to accept connections yet
err = pool.Retry(func() error {
dbPort, err := strconv.ParseUint(resource.GetPort("5432/tcp"), 10, 16)
if err != nil {
log.Fatalf("Unable to parse %s as uint. Error: %s\n", resource.GetPort("5432/tcp"), err)
}
connPoolConfig := pgx.ConnPoolConfig{
ConnConfig: pgx.ConnConfig{
Host: "localhost",
User: "postgres",
Password: "postgres",
Database: "postgres",
Port: uint16(dbPort),
},
}
dbPool, err = pgx.NewConnPool(connPoolConfig)
return err
})
if err != nil {
log.Fatalf("Could not connect to docker: %s\n", err)
}
_, err = dbPool.Exec(`CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE TABLE file (
name uuid PRIMARY KEY,
nonce bytea,
createdat timestamp,
invalidat timestamp
);`)
if err != nil {
log.Fatalf("DB returned error: %s\n", err)
}
code := m.Run()
// You can't defer this because os.Exit doesn't care for defer
if err := pool.Purge(resource); err != nil {
log.Fatalf("Could not purge resource: %s\n", err)
}
os.Exit(code)
}
var testFile vee.File
func TestCreateFile(t *testing.T) {
mds := postgres.MetadataService{DB: dbPool}
var nonce [32]byte
testFile = vee.File{
Name: uuid.New(),
Nonce: nonce,
Createdat: time.Now(),
Invalidat: time.Now(),
}
err := mds.CreateFile(&testFile)
if err != nil {
t.Fatalf("Unable to create file. Error: %s\n", err)
}
}
func TestFile(t *testing.T) {
mds := postgres.MetadataService{DB: dbPool}
file, err := mds.File(testFile.Name)
if err != nil {
t.Fatalf("Unable to get file. Error: %s\n", err)
}
if file.Name != testFile.Name {
t.Fatalf("expected filename to be %v, got %v\n", testFile.Name, file.Name)
}
if file.Nonce != testFile.Nonce {
t.Fatalf("expected filename to be %v, got %v\n", testFile.Nonce, file.Nonce)
}
_, err = mds.File(uuid.New())
if err.Error() != "no rows in result set" {
t.Fatalf("Expected error, got: %s\n", err)
}
}
package vee
import (
"io"
"time"
"github.com/google/uuid"
"gitlab.com/MadsRC/vee/app/config"
)
type File struct {
......@@ -12,9 +14,47 @@ type File struct {
Key [32]byte
Createdat time.Time
Invalidat time.Time
Content io.Reader
Size int64
}
type FileService interface {
File(name uuid.UUID) (*File, error)
CreateFile(f *File) error
}
type MetadataService interface {
File(name uuid.UUID) (*File, error)
CreateFile(f *File) error
}
type ObjectService interface {
File(name uuid.UUID) (*File, int64, error)
CreateFile(f *File) error
}
// Config is the domain representation of a configuration object
// FIXME(MadsRC): Need to find someother way of making the config available. This way
// of making the fields of config.Obj available in the Config type forces
// us to make the root package dependent on the config package. This does
// not go well for with the Standard Package Layout.
type Config config.Obj
type LogService interface {
Print(v ...interface{})
Printf(format string, v ...interface{})
SetOutput(w io.Writer)
GetDebug() bool
SetDebug(state bool)
Debug(v ...interface{})
Debugf(format string, v ...interface{})
}
// CryptService is the domain representation of the cryptographic service
// available throughout the application.
type CryptService interface {
Nonce(n []byte) (int, error)
Key(n []byte) ([32]byte, error)
Encrypt(dst io.Writer, src io.Reader, config Config) (n int64, err error)
Decrypt(dst io.Writer, src io.Reader, config Config) (n int64, err error)
}
sudo: false
language: go
go:
- 1.3.x
- 1.5.x
- 1.6.x
- 1.7.x
- 1.8.x
- 1.9.x
- master
matrix:
allow_failures:
- go: master
fast_finish: true
install:
- # Do nothing. This is needed to prevent default install action "go get -t -v ./..." from happening here (we want it to happen inside script step).
script:
- go get -t -v ./...
- diff -u <(echo -n) <(gofmt -d -s .)
- go tool vet .
- go test -v -race ./...
Copyright (c) 2005-2008 Dustin Sallings <dustin@spy.net>
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
<http://www.opensource.org/licenses/mit-license.php>
# Humane Units [![Build Status](https://travis-ci.org/dustin/go-humanize.svg?branch=master)](https://travis-ci.org/dustin/go-humanize) [![GoDoc](https://godoc.org/github.com/dustin/go-humanize?status.svg)](https://godoc.org/github.com/dustin/go-humanize)
Just a few functions for helping humanize times and sizes.
`go get` it as `github.com/dustin/go-humanize`, import it as
`"github.com/dustin/go-humanize"`, use it as `humanize`.
See [godoc](https://godoc.org/github.com/dustin/go-humanize) for
complete documentation.
## Sizes
This lets you take numbers like `82854982` and convert them to useful
strings like, `83 MB` or `79 MiB` (whichever you prefer).
Example:
```go
fmt.Printf("That file is %s.", humanize.Bytes(82854982)) // That file is 83 MB.
```
## Times
This lets you take a `time.Time` and spit it out in relative terms.
For example, `12 seconds ago` or `3 days from now`.
Example:
```go
fmt.Printf("This was touched %s.", humanize.Time(someTimeInstance)) // This was touched 7 hours ago.
```
Thanks to Kyle Lemons for the time implementation from an IRC
conversation one day. It's pretty neat.
## Ordinals
From a [mailing list discussion][odisc] where a user wanted to be able
to label ordinals.
0 -> 0th
1 -> 1st
2 -> 2nd
3 -> 3rd
4 -> 4th
[...]
Example:
```go
fmt.Printf("You're my %s best friend.", humanize.Ordinal(193)) // You are my 193rd best friend.
```
## Commas
Want to shove commas into numbers? Be my guest.
0 -> 0
100 -> 100
1000 -> 1,000
1000000000 -> 1,000,000,000
-100000 -> -100,000
Example:
```go
fmt.Printf("You owe $%s.\n", humanize.Comma(6582491)) // You owe $6,582,491.
```
## Ftoa
Nicer float64 formatter that removes trailing zeros.
```go
fmt.Printf("%f", 2.24) // 2.240000
fmt.Printf("%s", humanize.Ftoa(2.24)) // 2.24
fmt.Printf("%f", 2.0) // 2.000000
fmt.Printf("%s", humanize.Ftoa(2.0)) // 2
```
## SI notation
Format numbers with [SI notation][sinotation].
Example:
```go
humanize.SI(0.00000000223, "M") // 2.23 nM
```
## English-specific functions
The following functions are in the `humanize/english` subpackage.
### Plurals
Simple English pluralization
```go
english.PluralWord(1, "object", "") // object
english.PluralWord(42, "object", "") // objects
english.PluralWord(2, "bus", "") // buses
english.PluralWord(99, "locus", "loci") // loci
english.Plural(1, "object", "") // 1 object
english.Plural(42, "object", "") // 42 objects
english.Plural(2, "bus", "") // 2 buses
english.Plural(99, "locus", "loci") // 99 loci
```
### Word series
Format comma-separated words lists with conjuctions:
```go
english.WordSeries([]string{"foo"}, "and") // foo
english.WordSeries([]string{"foo", "bar"}, "and") // foo and bar
english.WordSeries([]string{"foo", "bar", "baz"}, "and") // foo, bar and baz
english.OxfordWordSeries([]string{"foo", "bar", "baz"}, "and") // foo, bar, and baz
```
[odisc]: https://groups.google.com/d/topic/golang-nuts/l8NhI74jl-4/discussion
[sinotation]: http://en.wikipedia.org/wiki/Metric_prefix
package humanize
import (
"math/big"
)
// order of magnitude (to a max order)
func oomm(n, b *big.Int, maxmag int) (float64, int) {
mag := 0
m := &big.Int{}
for n.Cmp(b) >= 0 {
n.DivMod(n, b, m)
mag++
if mag == maxmag && maxmag >= 0 {
break
}
}
return float64(n.Int64()) + (float64(m.Int64()) / float64(b.Int64())), mag
}
// total order of magnitude
// (same as above, but with no upper limit)
func oom(n, b *big.Int) (float64, int) {
mag := 0
m := &big.Int{}
for n.Cmp(b) >= 0 {
n.DivMod(n, b, m)
mag++
}
return float64(n.Int64()) + (float64(m.Int64()) / float64(b.Int64())), mag
}
package humanize
import (
"fmt"
"math/big"
"strings"
"unicode"
)
var (
bigIECExp = big.NewInt(1024)
// BigByte is one byte in bit.Ints
BigByte = big.NewInt(1)
// BigKiByte is 1,024 bytes in bit.Ints
BigKiByte = (&big.Int{}).Mul(BigByte, bigIECExp)
// BigMiByte is 1,024 k bytes in bit.Ints
BigMiByte = (&big.Int{}).Mul(BigKiByte, bigIECExp)
// BigGiByte is 1,024 m bytes in bit.Ints
BigGiByte = (&big.Int{}).Mul(BigMiByte, bigIECExp)
// BigTiByte is 1,024 g bytes in bit.Ints
BigTiByte = (&big.Int{}).Mul(BigGiByte, bigIECExp)
// BigPiByte is 1,024 t bytes in bit.Ints
BigPiByte = (&big.Int{}).Mul(BigTiByte, bigIECExp)
// BigEiByte is 1,024 p bytes in bit.Ints
BigEiByte = (&big.Int{}).Mul(BigPiByte, bigIECExp)
// BigZiByte is 1,024 e bytes in bit.Ints
BigZiByte = (&big.Int{}).Mul(BigEiByte, bigIECExp)
// BigYiByte is 1,024 z bytes in bit.Ints
BigYiByte = (&big.Int{}).Mul(BigZiByte, bigIECExp)
)
var (
bigSIExp = big.NewInt(1000)
// BigSIByte is one SI byte in big.Ints
BigSIByte = big.NewInt(1)
// BigKByte is 1,000 SI bytes in big.Ints
BigKByte = (&big.Int{}).Mul(BigSIByte, bigSIExp)
// BigMByte is 1,000 SI k bytes in big.Ints
BigMByte = (&big.Int{}).Mul(BigKByte, bigSIExp)
// BigGByte is 1,000 SI m bytes in big.Ints
BigGByte = (&big.Int{}).Mul(BigMByte, bigSIExp)
// BigTByte is 1,000 SI g bytes in big.Ints
BigTByte = (&big.Int{}).Mul(BigGByte, bigSIExp)
// BigPByte is 1,000 SI t bytes in big.Ints
BigPByte = (&big.Int{}).Mul(BigTByte, bigSIExp)
// BigEByte is 1,000 SI p bytes in big.Ints
BigEByte = (&big.Int{}).Mul(BigPByte, bigSIExp)
// BigZByte is 1,000 SI e bytes in big.Ints
BigZByte = (&big.Int{}).Mul(BigEByte, bigSIExp)
// BigYByte is 1,000 SI z bytes in big.Ints
BigYByte = (&big.Int{}).Mul(BigZByte, bigSIExp)
)
var bigBytesSizeTable = map[string]*big.Int{
"b": BigByte,
"kib": BigKiByte,
"kb": BigKByte,
"mib": BigMiByte,
"mb": BigMByte,
"gib": BigGiByte,
"gb": BigGByte,
"tib": BigTiByte,
"tb": BigTByte,
"pib": BigPiByte,
"pb": BigPByte,
"eib": BigEiByte,
"eb": BigEByte,
"zib": BigZiByte,
"zb": BigZByte,
"yib": BigYiByte,
"yb": BigYByte,
// Without suffix
"": BigByte,
"ki": BigKiByte,
"k": BigKByte,
"mi": BigMiByte,
"m": BigMByte,
"gi": BigGiByte,
"g": BigGByte,
"ti": BigTiByte,
"t": BigTByte,
"pi": BigPiByte,
"p": BigPByte,
"ei": BigEiByte,
"e": BigEByte,
"z": BigZByte,
"zi": BigZiByte,
"y": BigYByte,
"yi": BigYiByte,
}
var ten = big.NewInt(10)
func humanateBigBytes(s, base *big.Int, sizes []string) string {
if s.Cmp(ten) < 0 {
return fmt.Sprintf("%d B", s)
}
c := (&big.Int{}).Set(s)
val, mag := oomm(c, base, len(sizes)-1)
suffix := sizes[mag]
f := "%.0f %s"
if val < 10 {
f = "%.1f %s"
}
return fmt.Sprintf(f, val, suffix)
}
// BigBytes produces a human readable representation of an SI size.
//
// See also: ParseBigBytes.
//
// BigBytes(82854982) -> 83 MB
func BigBytes(s *big.Int) string {
sizes := []string{"B", "kB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB"}
return humanateBigBytes(s, bigSIExp, sizes)
}
// BigIBytes produces a human readable representation of an IEC size.
//
// See also: ParseBigBytes.
//
// BigIBytes(82854982) -> 79 MiB
func BigIBytes(s *big.Int) string {
sizes := []string{"B", "KiB", "MiB", "GiB", "TiB", "PiB", "EiB", "ZiB", "YiB"}
return humanateBigBytes(s, bigIECExp, sizes)
}
// ParseBigBytes parses a string representation of bytes into the number
// of bytes it represents.
//
// See also: BigBytes, BigIBytes.
//
// ParseBigBytes("42 MB") -> 42000000, nil
// ParseBigBytes("42 mib") -> 44040192, nil
func ParseBigBytes(s string) (*big.Int, error) {
lastDigit := 0
hasComma := false
for _, r := range s {
if !(unicode.IsDigit(r) || r == '.' || r == ',') {
break
}
if r == ',' {
hasComma = true
}
lastDigit++
}
num := s[:lastDigit]
if hasComma {
num = strings.Replace(num, ",", "", -1)
}
val := &big.Rat{}
_, err := fmt.Sscanf(num, "%f", val)
if err != nil {
return nil, err
}
extra := strings.ToLower(strings.TrimSpace(s[lastDigit:]))
if m, ok := bigBytesSizeTable[extra]; ok {
mv := (&big.Rat{}).SetInt(m)
val.Mul(val, mv)
rv := &big.Int{}
rv.Div(val.Num(), val.Denom())
return rv, nil
}
return nil, fmt.Errorf("unhandled size name: %v", extra)
}
package humanize
import (
"fmt"
"math"
"strconv"
"strings"
"unicode"
)
// IEC Sizes.
// kibis of bits
const (
Byte = 1 << (iota * 10)
KiByte
MiByte
GiByte
TiByte
PiByte
EiByte
)
// SI Sizes.
const (
IByte = 1
KByte = IByte * 1000
MByte = KByte * 1000
GByte = MByte * 1000
TByte = GByte * 1000
PByte = TByte * 1000
EByte = PByte * 1000
)
var bytesSizeTable = map[string]uint64{
"b": Byte,
"kib": KiByte,
"kb": KByte,
"mib": MiByte,
"mb": MByte,
"gib": GiByte,
"gb": GByte,
"tib": TiByte,
"tb": TByte,
"pib": PiByte,
"pb": PByte,
"eib": EiByte,
"eb": EByte,
// Without suffix
"": Byte,
"ki": KiByte,
"k": KByte,
"mi": MiByte,
"m": MByte,
"gi": GiByte,
"g": GByte,
"ti": TiByte,
"t": TByte,
"pi": PiByte,
"p": PByte,
"ei": EiByte,
"e": EByte,
}
func logn(n, b float64) float64 {
return math.Log(n) / math.Log(b)
}
func humanateBytes(s uint64, base float64, sizes []string) string {
if s < 10 {
return fmt.Sprintf("%d B", s)
}
e := math.Floor(logn(float64(s), base))
suffix := sizes[int(e)]
val := math.Floor(float64(s)/math.Pow(base, e)*10+0.5) / 10
f := "%.0f %s"
if val < 10 {
f = "%.1f %s"
}
return fmt.Sprintf(f, val, suffix)
}
// Bytes produces a human readable representation of an SI size.
//
// See also: ParseBytes.
//
// Bytes(82854982) -> 83 MB
func Bytes(s uint64) string {
sizes := []string{"B", "kB", "MB", "GB", "TB", "PB", "EB"}
return humanateBytes(s, 1000, sizes)
}
// IBytes produces a human readable representation of an IEC size.
//
// See also: ParseBytes.
//
// IBytes(82854982) -> 79 MiB
func IBytes(s uint64) string {
sizes := []string{"B", "KiB", "MiB", "GiB", "TiB", "PiB", "EiB"}
return humanateBytes(s, 1024, sizes)
}
// ParseBytes parses a string representation of bytes into the number
// of bytes it represents.
//
// See Also: Bytes, IBytes.
//
// ParseBytes("42 MB") -> 42000000, nil
// ParseBytes("42 mib") -> 44040192, nil
func ParseBytes(s string) (uint64, error) {
lastDigit := 0
hasComma := false
for _, r := range s {
if !(unicode.IsDigit(r) || r == '.' || r == ',') {
break
}
if r == ',' {
hasComma = true
}
lastDigit++
}
num := s[:lastDigit]
if hasComma {
num = strings.Replace(num, ",", "", -1)
}
f, err := strconv.ParseFloat(num, 64)
if err != nil {
return 0, err
}
extra := strings.ToLower(strings.TrimSpace(s[lastDigit:]))
if m, ok := bytesSizeTable[extra]; ok {
f *= float64(m)
if f >= math.MaxUint64 {
return 0, fmt.Errorf("too large: %v", s)
}
return uint64(f), nil
}
return 0, fmt.Errorf("unhandled size name: %v", extra)
}
package humanize
import (
"bytes"
"math"
"math/big"
"strconv"
"strings"
)
// Comma produces a string form of the given number in base 10 with
// commas after every three orders of magnitude.
//
// e.g. Comma(834142) -> 834,142
func Comma(v int64) string {
sign := ""
// Min int64 can't be negated to a usable value, so it has to be special cased.
if v == math.MinInt64 {
return "-9,223,372,036,854,775,808"
}
if v < 0 {
sign = "-"
v = 0 - v
}
parts := []string{"", "", "", "", "", "", ""}
j := len(parts) - 1
for v > 999 {
parts[j] = strconv.FormatInt(v%1000, 10)
switch len(parts[j]) {
case 2:
parts[j] = "0" + parts[j]