Build often too slow

My builds were successful, but sometimes it's very slow, so they fail. My runner is hosted on an EC2 instance from Amazon with t2.micro (1 Go Ram), I added a 10 Go swap, and I changed the storage directory of the images from /var/lib/docker to /var/lib/testdir. Why is it so slow and how can I improve that?

Logs:

Dockerfile: https://drive.google.com/open?id=0B1asRaa7Hfw8VXprLXVzSWdYR0E

gitlab-ci.yml: https://drive.google.com/open?id=0B1asRaa7Hfw8V0ZMcWdYRDdGeVU

logs: https://drive.google.com/open?id=0B1asRaa7Hfw8Sm9OSElWbE96Rnc

ZD: https://gitlab.zendesk.com/agent/tickets/75131

cc// @tmaczukin @grzesiek