Commit 08cb6dcf authored by Ines's avatar Ines Committed by Mat
Browse files

Import Searchitect

parents
*.sw?
.#*
*#
*~
.classpath
.project
.settings
bin
build
target
salt
dependency-reduced-pom.xml
*.sublime-*
/scratch
.gradle
README.html
.idea
*.iml
*/.classpath
*/.project
*/.settings
*/build
*/target
*/.mvn/wrapper
*/bin
*/gradle/wrapper
*/salt/saltInvIX
*/salt/*
*/*.log
*/*.gz
*/salt
# Eclipse artifacts, including WTP generated manifests
.classpath
*/.project
*/spring-*/src/main/java/META-INF/MANIFEST.MF
# IDEA artifacts and output dirs
*/*.iml
*/*.ipr
*/*.iws
*/.idea
*/out
*/test-output
*/atlassian-ide-plugin.xml
*/.gradletasknamecache
*.iml
*.ipr
*.iws
.idea
out
test-output
sudo: false
language: java
jdk:
- oraclejdk8
script:
- test/run.sh
notifications:
slack:
secure: IYB72v6BEnrBz0hD7Zist2EQqjdOdsM/uIuPqV91erPGk/EDg+E/A66t22nDKZW+bJbgZTx5n3TBPDod8u9adXI3OH4aOuwdbzEFCi6nBc/QJfwuDR97s8/F6Yb3/2H6JXrxwbuy05bBVmnscC9viw+MI4P8rDBRutwde5Pz6mc=
Other Code is licenced under GNU GPL v3
Original spring-boot license:
All code in this repository is:
=======================================================================
Copyright (c) 2013 GoPivotal, Inc. All Rights Reserved
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
# Searchitect-Searchable Encryption Framework
#### Description of the Searchitect Framework:
This framework enables the integration of SE schemes.
#### General description of SE
A SE scheme enables a server to search over an encrypted database on behalf of a client without revealing the content to the server.
A SE scheme provides 3 protocols:
1. Setup - First the client is indexing a document collection contained in a directory. This plaintext index gets encrypted by a specific implementation of an encryption scheme and uploaded to the server.
2. Search - After Setup the client is able to search over the data by passing the keyword to the search protocol, which computes a search token which is sent to the server. This search Token enables the server to search over the encrypted data and return the resulting document matches. In resource hiding schemes these are encrypted and therfore a second Resolve procedure at the client is needed to decrypt document identifiers.
3. Update - Dynamic schemes support a update of the documents contained in the encrypted index.
#### Framework Architecture:
* Client/server architecture based on microservices
* SOA (service oriented architecture) based on RESTful webservices
#### Implementation:
* Basic implementations
* searchitect-common - classes shared between server and client
* searchitect-client - common client implementation
* searchitect-gateway - external interface
* searchitect-backend-module - backend side of the implemented scheme
* Added schemes appear in the project forms of:
* searchitect-common-scheme
* searchitect-client-scheme-plugin
* searchitect-backend-scheme
#### The interface description of the gateway is after deployment available at:
https://localhost:8433/swagger-ui.html
### Compile
Run the following command in the top level directory
mvn clean install
### General deployment using Docker-compose
Docker enables a containerized easy deployment, the docker-compose configuration file is called docker-compose.yml.
docker-compose build
docker-compose up
### How to add a new scheme
1. Implement your scheme in a new searchitect-common-scheme project
2. Create a new searchitect-client-scheme-plugin project which implements the client plugin interface. This interface can be found in searchitect.common.client.ClientScheme
3. Create a new project which implements the searchitect-backend-scheme at the server side, take a look to the other implementations the interface of the controller needs to be similar
4. Adapt the Docker File in your searchitect.common.client.ClientScheme, take care with the port, choose one that is still available
5. Add your new Searchitect-backend implementation in the application properties of the searchitect-gate project to the list. The name needs to be the same as in the docker-compose file (docker-compose.yml).
6. Recompile the whole workspace and test.
# plot keywords and records in the realistic dataset
#set working directory
getwd()
#tabelle einlesen
testset= read.table("realtestsetlog", header=T, sep = ";")
Keywords=testset[["keywords"]]
Records=testset[["records"]]
plot.default(x=Records,y=Keywords, xlab="Number of Records", ylab="Number of Keywords", main="Keywords and Records of the Realistic Dataset")
text(x=10000,y=2100,"a_user")
text(x=250000,y=23500,"l_user")
text(x=600000,y=19500,"m_user")
text(x=600000,y=40000,"n_user")
#get ordered table
testset[order(testset$name),]
##########################################################
# plot synthetic setup dynrh2lev and dynrh2levrocks
sd= read.table("dynrh2levsynthetictest", hetader=T, sep = ";")
sdr = read.table("dynrh2levrockssynthetictest", header=T, sep = ";")
setuprec = sd[["records"]]
sdsetuptime = sd[["setuptime"]]
sdrsetuptime = sdr[["setuptime"]]
#plot(x=setuprec,y=sdrsetuptime, xlab="Number of Records", ylab="Running Time of Setup", main="Setup Time of Synthetic Test Set", type="l",col="red")
#lines(x=setuprec,y=sdsetuptime, type="i",col="blue")
matplot(x= setuprec, cbind(sdrsetuptime,sdsetuptime),type="l",col=c("red","blue"),lty=c(1,5), xlab="Number of Records", ylab="Running Time of Setup [ms]", main="Setup time of the Synthetic Test Set")
points(x=setuprec,y=sdrsetuptime,col="red",pch=1)
points(x=setuprec,y=sdsetuptime,col="blue",pch=4)
legend(10,120000, c("DynRH2LevRocks","DynRH2Lev"), lty=c(1,5), col=c("red","blue"), pch=c(1,4))
#################################################################
# plot synthetic search dynrh2lev and dynrh2levrocks
sd= read.table("dynrh2levsynthetictest", header=T, sep = ";")
sdr = read.table("dynrh2levrockssynthetictest", header=T, sep = ";")
setuprec = sd[["records"]]
sdsearchtime = sd[["searchtime"]]
sdrsearchtime = sdr[["searchtime"]]
matplot(x= setuprec, cbind(sdrsearchtime,sdsearchtime),type="l",col=c("red","blue"),lty=c(1,5), xlab="Number of Records", ylab="Running Time of Search [ms]", main="Search time of Synthetic Test Set")
points(x=setuprec,y=sdrsearchtime,col="red",pch=1)
points(x=setuprec,y=sdsearchtime,col="blue",pch=4)
legend(10,23000, c("DynRH2LevRocks","DynRH2Lev"), lty=c(1,5), col=c("red","blue"), pch=c(1,4))
specify_decimal <- function(x, k) trimws(format(round(x, k), nsmall=k))
#####################################################################################
#table real setup dynrh2lev and dynrh2levrocks
rrocksetup = read.table("dynrh2levrocksrealsetuptest", header=T, sep = ";")
rdsetup = read.table("dynrh2levrealsetuptest", header=T, sep = ";")
rrec <- rrocksetup[["records"]]
rkey<-rrocksetup[["keywords"]]
rrocksetuptime <-specify_decimal(rrocksetup[["setuptime"]]/1000,2)
dbsize<-specify_decimal(rrocksetup[["dbsize"]]/1048576,2)
arraysize<-specify_decimal(rrocksetup[["arraysize"]]/1048576,2)
rdsetuptime <-specify_decimal(rdsetup[["setuptime"]]/1000,2)
ruser<-rrocksetup[["user"]]
data <- data.frame(ruser,rkey,rrec,rdsetuptime,rrocksetuptime,dbsize,arraysize)
data
###################################################################
# plot real setup dynrh2lev and dynrh2levrocks
rrocksetup = read.table("dynrh2levrocksrealsetuptest", header=T, sep = ";")
rdsetup = read.table("dynrh2levrealsetuptest", header=T, sep = ";")
rrec <- rrocksetup[["records"]]
rkey<-rrocksetup[["keywords"]]
rrocksetuptime <-rrocksetup[["setuptime"]]/1000
dbsize<-rrocksetup[["dbsize"]]
arraysize<-rrocksetup[["arraysize"]]
rdsetuptime <-rdsetup[["setuptime"]]/1000
ruser<-rrocksetup[["user"]]
data <- data.frame(ruser,rkey,rrec,rdsetuptime,rrocksetuptime,dbsize,arraysize)
rd <-as.vector(data$rdsetuptime)
rd[9:13]<-0
matplot(x= rrec, cbind(rrocksetuptime,rd),type="l",col=c("red","blue"),lty=c(1,5), cex.lab=1.5,cex.axis=1.3, cex.main=1.5,xlab="Number of Records", ylab="Running Time of Setup [ms]", main="Setup Time of Realistic Test Set")
points(rrec,rdsetuptime,col="blue",pch=4)
points(rrec,rrocksetuptime,col="red",pch=1)
legend(300000,130, c("DynRH2LevRocks","DynRH2Lev"), lty=c(1,5), col=c("red","blue"), pch=c(1,4),cex=1.3)
###################################################################
# plot real storage size dynrh2lev and dynrh2levrocks
rrocksetup = read.table("dynrh2levrocksrealsetuptest", header=T, sep = ";")
rdsetup = read.table("dynrh2levrealsetuptest", header=T, sep = ";")
rdirsize = read.table("realdirsize", header=F, sep = "\t")
rrec <- rrocksetup[["records"]]
rkey<-rrocksetup[["keywords"]]
rsize<-rdirsize["V1"]
rsize<-rsize[1:13,"V1"]
dbsize<-rrocksetup[["dbsize"]]/1024
arraysize<-rrocksetup[["arraysize"]]/1024
mmsize<-rrocksetup[["mmspace"]]/1024
ruser<-rrocksetup[["user"]]
data <- data.frame(ruser,rkey,rrec,rdsetuptime,rrocksetuptime,dbsize,arraysize)
rd <-as.vector(data$rdsetuptime)
rd[9:13]<-0
matplot(x= rrec, cbind(rsize,mmsize,dbsize,arraysize),type="l",col=c("green","black","red3","red"),lty=c(1,5,6,4), cex.lab=1.5,cex.axis=1.3, cex.main=1.5,xlab="Number of Records", ylab="Storage Space [KB]", main="Storage Space Usage of DynRH2LevRocks with Realistic Test Set")
points(rrec,mmsize,col="black",pch=4)
points(rrec,rsize,col="green",pch=5)
points(rrec,dbsize,col="red3",pch=0)
points(rrec,arraysize,col="red",pch=1)
legend(300000,180000, c("DynRH2LevRocks Dictionary","DynRH2Lev Dictionary + Array","Data Size", "Inverted Index Size"), lty=c(6,4,1,5), col=c("red3","red","green","black"), pch=c(9,1,5,4),cex=1.3)
#####################################################################################
# plot real search dynrh2lev and dynrh2levrocks
library(data.table)
specify_decimal <- function(x, k) trimws(format(round(x, k), nsmall=k))
users<-c('a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j','k', 'l', 'm')
users<-paste(users,"_user",sep="")
docs<-c(35,132, 248,378,437,563, 642,721,994,1116,2249,3272, 4796, 8009)
#read in from file and clean
rrocksearch <-fread("dynrh2levrocksrealsearchtest")
rrocksearch<-rrocksearch[,V7:=NULL]
set(rrocksearch,1, "searchtime" ,NA_integer_)
set(rrocksearch,1, "result" ,NA_integer_)
rdsearch <- fread("dynrh2levrealsearchtest")
rdsearch <-rdsearch[,V7:=NULL]
set(rdsearch,1, "searchtime" ,NA_integer_)
set(rdsearch,1, "result" ,NA_integer_)
user<-"a_user"
i=1
output <- matrix(ncol=16, nrow=13)
#loop apply to subset of user
for(user in users){
rrextract <- rrocksearch[user,on="user"]
if(i<9){
rdextract <- rdsearch[user,on="user"]
#join the tables
total <- rrextract[rdextract,on=c("keyword","user","records","result","keywords")]
}
else{
total<-rrextract[]
}
#calculate separator
keywords<-mean(total$keywords)
records<-mean(total$records)
doc<-docs[i]
sep<-doc/3
#select rows for aggregation
numtotal <- total[,length(user)]
#number of keyword queries group1
num1 <-total[result<sep,length(user)]
#mean over results
mean1<-specify_decimal(total[result<sep, mean(result)],2)
#calculate mean over search times for dynrh2lev and dynrh2levrocks
search1 <- specify_decimal(total[result<sep, mean(searchtime)],2)
#number of keyword queries group2
num2 <- total[result>sep&result<2*sep,length(user)]
#mean over results group2
mean2<-specify_decimal(total[result>sep&result<2*sep, mean(result)],2)
#calculate mean over search times for dynrh2lev and dynrh2levrocks
search2 <- specify_decimal(total[result>sep&result<2*sep, mean(searchtime)],2)
#number of keyword queries group3
num3 <-total[result>2*sep,length(user)]
#mean over results group3
mean3<-specify_decimal(total[result>2*sep, mean(result)],2)
search3 <- specify_decimal(total[result>2*sep, mean(searchtime)],2)
if(i<9){
i.search1<- specify_decimal(total[result<sep, mean(i.searchtime)],2)
i.search2 <- specify_decimal(total[result>sep&result<2*sep, mean(i.searchtime)],2)
i.search3 <- specify_decimal(total[result>2*sep, mean(i.searchtime)],2)
}
else{
i.search1<-NA
i.search2<-NA
i.search3 <-NA
}
vector<-c(user,keywords,records,numtotal,num1,mean1,search1,i.search1,num2,mean2,search2,i.search2,num3,mean3,search3,i.search3)
output[i,]<-vector
i<-i+1
}
headings<-c("user","keywords","records","numtotal","num1","mean1","ds1","rs1","num2","mean2","ds2","rs2","num3","mean3","ds3","rs3")
output[is.na(output)] <- 0
#output <- data.frame(output)
colnames(output) <- headings
rocks= read.table("testdynrh2rockst", header=F, sep = ":")
# Some useful docker instructions and commands
## run mysql image
docker run --name searchitect-mysql -e MYSQL_ROOT_PASSWORD=password -e MYSQL_DATABASE=searchitect -e MYSQL_USER=se_user -e MYSQL_PASSWORD=se_pass -d mysql:5.6
-p -p 127.0.0.1:3306:3306
## permanently needed docker commands
#### build docker image -t sets tag for image
docker build . -t searchitect
#### run docker container exposes port 8080 on host
docker run -p 127.0.0.1:8080:8080 searchitect
#### stop all running containers
docker stop $(docker ps -a -q)
#### delete all docker containers
docker rm $(docker ps -a -q)
#### delete all images
docker rmi $(docker images -q)
## docker-compose
#### all in the .yml file referenced containers
1. first move at same path-level like the docker-compose.yml
2. build all resources/images referenced in docker-compose.yml
docker-compose build
3. start all services at once
docker-compose up
#### show all commands
docker-compose --help
### https://docs.docker.com/compose/reference/overview/
Usage:
docker-compose [-f <arg>...] [options] [COMMAND] [ARGS...]
docker-compose -h|--help
Options:
-f, --file FILE Specify an alternate Compose file (default: docker-compose.yml)
-p, --project-name NAME Specify an alternate project name (default: directory name)
--verbose Show more output
-v, --version Print version and exit
-H, --host HOST Daemon socket to connect to
--tls Use TLS; implied by --tlsverify
--tlscacert CA_PATH Trust certs signed only by this CA
--tlscert CLIENT_CERT_PATH Path to TLS certificate file
--tlskey TLS_KEY_PATH Path to TLS key file
--tlsverify Use TLS and verify the remote
--skip-hostname-check Don't check the daemon's hostname against the name specified
in the client certificate (for example if your docker host
is an IP address)
--project-directory PATH Specify an alternate working directory
(default: the path of the Compose file)
Commands:
build Build or rebuild services
bundle Generate a Docker bundle from the Compose file
config Validate and view the Compose file
create Create services
down Stop and remove containers, networks, images, and volumes
events Receive real time events from containers
exec Execute a command in a running container
help Get help on a command
images List images
kill Kill containers
logs View output from containers
pause Pause services
port Print the public port for a port binding
ps List containers
pull Pull service images
push Push service images
restart Restart services
rm Remove stopped containers
run Run a one-off command
scale Set number of containers for a service
start Start services
stop Stop services
top Display the running processes
unpause Unpause services
up Create and start containers
version Show the Docker-Compose version information
#### docker-compose.yml example
webapp:
image: examples/web
ports:
- "8000:8000"
volumes:
- "/data"
## install docker
#### add key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
#### add repository
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable
#### update apt repos
sudo apt-get update
#### install docker
sudo apt-get install docker
sudo apt-get install docker-compose
#### check if docker is running
sudo systemctl status docker
#### add your user to dockergroup
sudo usermod -aG docker ${USER}
#### apply new group membership or logout
su - ${USER}
#### verify your users group
id -nG
## docker commands
#### all possible docker commands
docker
Commands:
attach Attach to a running container
build Build an image from a Dockerfile
commit Create a new image from a container's changes
cp Copy files/folders between a container and the local filesystem
create Create a new container
diff Inspect changes on a container's filesystem
events Get real time events from the server
exec Run a command in a running container
export Export a container's filesystem as a tar archive
history Show the history of an image
images List images
import Import the contents from a tarball to create a filesystem image
info Display system-wide information
inspect Return low-level information on a container, image or task
kill Kill one or more running containers
load Load an image from a tar archive or STDIN
login Log in to a Docker registry.
logout Log out from a Docker registry.
logs Fetch the logs of a container
network Manage Docker networks
node Manage Docker Swarm nodes
pause Pause all processes within one or more containers
port List port mappings or a specific mapping for the container
ps List containers
pull Pull an image or a repository from a registry
push Push an image or a repository to a registry
rename Rename a container
restart Restart a container
rm Remove one or more containers
rmi Remove one or more images
run Run a command in a new container
save Save one or more images to a tar archive (streamed to STDOUT by default)
search Search the Docker Hub for images
service Manage Docker services
start Start one or more stopped containers
stats Display a live stream of container(s) resource usage statistics
stop Stop one or more running containers
swarm Manage Docker Swarm
tag Tag an image into a repository
top Display the running processes of a container
unpause Unpause all processes within one or more containers
update Update configuration of one or more containers
version Show the Docker version information
volume Manage Docker volumes
wait Block until a container stops, then print its exit code
#### check docker installation
docker info
#### hello world check
docker run hello-world
#### show docker containers -a active -l latest
docker
## create a docker file
* FROM - original images
* ADD -> can be used without build context but only once
* COPY -> better use copy COPY . /tmp/ than add
* RUN -> ‘/bin/sh’ executed other need to be RUN ["/bin/bash", "-c", "echo hello"] executed at build time
* CMD -> command executed at runtime will be overruled by entrypoint
* ENV -> set environment variables
* EXPOSE -> listening ports at runtime, need to be run with -p : to be exposed to the host
* LABEL -> define key value pairs
* ENTRYPOINT -> executed at runtime override by using --entrypoint,
* STOPSIGNAL -> sets the system call signal that will be sent to the container to exit
* USER -> user at compiletime
* VOLUME - include outer volume
* WORKDIR -> sets the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD instructions that follow it in the Dockerfile
* ONBUILD -> adds to the image a trigger instruction to be executed at a later time
* ARG -> defines a variable that users can pass at build-time to the builder
* HEALTHCHECK
version: "2"
services:
# important note: servicename MUST match backend name b_id of request urls
template:
build: ./searchitect-backend-template
ports:
- "8383:8383"
sophos:
build: ./searchitect-backend-sophos
ports:
- "8484:8484"
dynrh2lev:
build: ./searchitect-backend-dynrh2lev
ports:
- "8282:8282"
dynrh2levrocks:
build: ./searchitect-backend-dynrh2levrocks
ports:
- "8585:8585"
searchitect-gate:
build: ./searchitect-gate
ports:
- "8433:8433"
<project>
<modelVersion>4.0.0</modelVersion>
<groupId>com.example</groupId>
<artifactId>searchitect</artifactId>
<version>1</version>
<packaging>pom</packaging>
<modules>
<module>searchitect-common</module>
<module>searchitect-common-dynrh2lev</module>
<module>searchitect-common-sophos</module>
<module>searchitect-gate</module>
<module>searchitect-backend-dynrh2lev</module>
<module>searchitect-backend-dynrh2levrocks</module>
<module>searchitect-backend-sophos</module>
<module>searchitect-backend-template</module>
<module>searchitect-client-dynrh2lev-plugin</module>
<module>searchitect-client-dynrh2levrocks-plugin</module>
<module>searchitect-client-sophos-plugin</module>
<module>searchitect-client</module>
<module>searchitect-testset</module>
<module>searchitect-test</module>
</modules>
</project>
FROM openjdk:10
VOLUME /tmp
COPY ./target/searchitect-backend-dynrh2lev-0.1.0.jar /tmp/searchitect-backend-dynrh2lev-0.1.0.jar
CMD ["java", "-jar","/tmp/searchitect-backend-dynrh2lev-0.1.0.jar"]
# Searchitect-Backend-DynRH2Lev
### Functionality
* Serverpart of a Searchable Encryption scheme dynrh2lev
* Uses dynrh2lev implementation from the Clusion library
* Stores index repositories
### Packages
* searchitect
* Application.java - starts the Spring Boot application
* searchitect.controller
* CheckController.java - just for testing purpose
* BackendController.java - Restful Interface of backend implementation
* searchitect.model
* Dictionary.java
* wraps the Multimap <String,byte[]> index dictionary
* IndexImpl.java
* Entity which gets persisted - persistent fields are a Dictionary instance, an array and the update HashMap
* contains the repositoryName
* uses the dynrh2lev implementation of the Clusion library for the searchable encryption methods setup, update, search
* searchitect.services
* IndexRepository.java - like DAO extends JPARepository
### Build instructions
from inside the directory searchitect-backend-dynrh2lev
mvn clean install
### Deploy for modul test: