How Capitalico Came Out From My Own Experience

Why can I not choose where I want to live?  I want to be where I can do what I want to do but not what I have to do for work.

Back in 2010, after working in Tokyo, I went back to my hometown due to family issues and I had to earn my living while being at home. I was not a designer or an engineer who could do freelancing online.  So I decided to trade currencies given my financial industry experience. I also thought that day trading would be a great way to make a living regardless of where I was.  Through this process, I had to learn the hard way that trading is not about sense or luck. I learned that making profits from day trading is a very disciplined process. I spent two years going through this, where I had to examine whether certain trading ideas would make profits at least in the backtest and demo-trade, then actually execute the live trades diligently while many human emotions hinder doing so.

Through my trading experience, I figured out that there are many things that we as humans are not good at but that machines may be good at.  For example, machines do not need to fight human emotions such as fear to keep executing trades strictly based on the pre-set rules.  Machines are much better at keeping an eye on multiple charts 24 hours a day; detecting chart patterns when trades should be executed. I strongly felt since then, that there are things that humans should delegate to machines, because we were not born to be good at some things.

Developing Capitalico has become personal for me, because it is solving the pain that I actually had to go through.  And more importantly, Capitalico can potentially contribute hugely to the ideal lifestyle that I had envisioned when I started day trading, where earning money is extremely flexible. Money is just one of many constraints in life. Through Capitalico, I believe that we can loosen up this constraint to motivate people to go for what they are passionate about. With this thought process, we hope to contribute to “Be a Human Being” by sharing Capitalico to the world.

7 Essential Tools That Support How We Work at Alpaca (ver. Oct 2015)

Working on a small idea that may disrupt the world is the biggest fun part of being a startup.  We enjoy how world is being changed by those small ideas that are provided by other small startups.  All of the Alpaca team members have years of working experience in famous big companies and always talk how we need to change the way we work.  I believe this is the case in most young startup companies but also guess that the details are always different.  So today I just wanted to show how we work by showing what we use today to achieve our goal in the shortest path.



There is no doubt that Slack is being adopted rapidly.  There are so many reasons and goodness, but especially for teams like us, who are distributed geographically, the team communication is key and I cannot imagine how we can run our team coming from Japan, Silicon Valley and South Asia.   Most of our toolset is integrated with Slack and we check all those updates from them, not to mention whatever team members are saying.

We used to use Google Hangout and Skype for our internal meeting.  Google Hangout was good since it can integrate with Slack, but it just didn’t work well with the party of three and above, and voice broke a lot.  Skype is still a common tool when we talk to potential business partners, investors and users, but there also often happen problems like connection is killed, screen sharing stops working, voice gets unclear, etc. especially when the number of people becomes bigger than four or five, and it is not integrated with Slack.  Then we searched alternatives, and found just works for us.  Again, connecting from different location, sometimes from bad connections using mobile, we have daily status update meeting following the common practice of Agile/Scrum methodology.  It should finish in a short time in order not to bother team members’ work and it is pretty important for us to just start and finish the meeting without technical issues.


Pivotal Tracker

Coming from Pivotal myself, the agile methodology just makes sense to workplace like us, who create a brand-new value by software.  And Pivotal Tracker is the best tool to follow it.  Single stream line of prioritized stories makes it easy to see what should be done next, and combined with the daily status meeting we ensure that everyone is working on the right things and the team is moving forward to the goal.  We actually migrated from Trello to Tracker, because Trello was not clear on this point.  We put any items to do in the icebox, not just user stories, and have everyone not just developers in the status meeting, because we are delivering the value, not just writing code.  Of course updates on Tracker are notified in a Slack channel.



We used to have an internal private GitLab server manage private repositories due to historical inevitable requirements.  Moving to GitHub’s private repository recently, we are pretty comfortable with pull request workflow, and simply we don’t need to manage on-premise software deployment like GitLab any more.  Pushes to the repositories are notified in a Slack channel.  We are managing issues in Pivotal Tracker, and not using GitHub issues.  We put many things in GitHub private repositories not just software program code.  We also love to work in the open source ecosystem and we have a couple of open source repositories so others can reuse them.



Time is the most valuable resource in a startup and moving fast is critically important for us.  We never have dedicated QA people like big companies do, so making sure our software works as soon as new change is made is what CircleCI helps us to do.  Even though our software stack is pretty complicated, thanks to CircleCI being able to run Docker, our turnaround time for one test execution is radically minimal.


Intercom, MixPanel, and New Relic

To me these are a must-have set of tools for any kind of SaaS product.  Talking directly to your users real time through the Intercom’s browser chat window keeps users engaged.  You make hypothesis of user  needs and analyze how they behave (or even how they run away) using MixPanel.  Keep sending system metrics to New Relic and watch any error to happen in New Relic.  All notifications are sent to Slack and everyone can react on those events immediately.  With these tools, your SaaS (or even website) is going to be a real shop.



I was a bit surprised that this neat tool is not so famous when I found it.  This guy is just a “tail” command as a service.  You send all log output to its network interface and look at it through the browser.  You don’t need to do tail -f in your console anymore.  Sounds overkill?  When it comes to a distributed system, it’s fairly hard to find the problem if something happens, but often concrete evidence is left in log.  From my past experience in a distributed database development, this is always the case.  With logspout (, all of the console output from any docker containers is sent to this service and we all look at our browser.


Not only these tools are fun to use, but it is also important to us that these tools help us have less belongings.  We are a distributed small team currently, but also we love and support how we work, and these tools help us work this way with less belongings.  We believe people should have different way of working, which is totally backed by our company vision, “be a human being”.


Well, these are not all of the list (obviously we heavily use Google apps for instance) but should give a good idea of how we work at Alpaca.  And we all like new stuff and are always eager to try something new, so the list may change tomorrow (in fact, some of them are just started in the last couple of weeks).  Something may be obvious to you and some may be new.  We look forward to hearing your suggestions.


Our deep-learning trading platform, Capitalico, is also developed with these tools.  We are running closed beta in a limited time and we value your feedback.  Sign up for the beta test today, and don’t miss this chance to get involved in the new trading era with Capitalico!

Run Multiple Docker Daemons with Net Container

Docker is an awesome software that could disrupt how we run software in the new cloud age.  At Alpaca, we are running dozens of servers for production cloud services as well as development environment from virtualized servers in AWS and monthly paid bare metal rental to in-house machines in our office room.  We are running everything inside Docker, so we don’t care much about underlying host OS and provisioning.  We clone the private GitHub repository and type ‘make’  that pulls pre-built Docker images from remote that all dependencies are installed in and run the services in a few minutes.  When it comes to scientific computing in Python and GPU, despite of big community effort making it easier than ever, it is still bumpy and hard to maintain.  Docker images just make it nothing but do it once and only once.  Beautiful.

As a tiny startup company, we are trying to minimize the operation cost literally every day, but running GPU in cloud inherently costs you a lot.  Our application is not CPU or memory intensive, but very GPU intensive, and running a number of small GPU servers with unnecessary CPU and memory for every developer in the team would not be affordable for us.  So we wanted to share one powerful GPU server among handful developers, which would reduce the server cost dramatically compared to running a small GPU instance for each developer.  The question was how do you share Docker environment in one machine, without interrupting others.  Well, it may sound easy given the vision of Docker, which is to say everything is capsuled and isolated, but the reality is not there yet.  We expose a fixed set of port numbers as well as some of the internal processes may start listening on unpredictable ports.  We use predictable docker container names when starting them, which could also conflict with others.  We could avoid such conflicts by offsetting port numbers and suffixing container names, but even worse was the Docker cache.  While someone tags a new Docker image for new feature, others accidentally run that image which doesn’t work with the old version of application.

It was not entirely clear to us how we could achieve it.  I looked at “Docker in Docker” or DnD (sounds crazy…) for example, but it didn’t help our situation much.  The basic idea of DnD is to allow a container process to access the host’s Docker daemon, which wouldn’t isolate anything.  Our conclusion was that we have no choice but running multiple docker daemons in one host to isolate the Docker environment for each developer.  We found it is possible to run multiple daemons, but there were a couple of issues to be resolved.  One is the network bridge issue.  You may not have heard about the docker0 bridge, but this is the one trick that Docker does to let containers communicate with external network and also provides inter-container communication.  The bridge is created behind the scene by the daemon when firstly it runs.  We need to have multiple of it, but for some reason Docker daemon doesn’t like to create one for each, so you have to create one if you want other bridges than docker0.  Did we solve the network issue by this?  No.  We were using –net=host and our application assumes each process can talk to each other without hassle of container linkage (personally I think this container linkage approach is one of the bad designs of Docker), and breaking the assumption was just not possible.  I digged into the document and found there is another net mode which is –net=container.  What it does is to let containers share the same network namespace, which means you can allow set of containers assume as if it ran on –net=host and communicate each other that way, but still put all those assumption into one network container which doesn’t contaminate host’s network space.


So in the diagram above, the server container is free to listen on any port number as far as it works within the set of containers (collectively “one application”).  In other words, the server in the daemon A can listen on the same port as the server in the B daemon does.  It turns out this then makes a daemon space effectively a virtualized host, where multiple processes run and communicate with each other much like they do in a bare metal host. You don’t have to worry about container linkage but still gain the benefit of Docker’s pre-built dependency mechanism.  Bravo.

Here is the snippet that helps you understand how exactly we do what I just described.


function start_dockers {


  for u in $USERS

      echo "Creating the bridge and starting the daemon for ${u}"
      # create a bridge
      brctl addbr ${BRIDGE_NAME}
      SUBNET=$(expr 52 + ${OFFSET})
      ip addr add 172.18.${SUBNET}.1/24 dev ${BRIDGE_NAME}
      ip link set dev ${BRIDGE_NAME} up

      # IP Masquerade, if not set yet
      if ! iptables -t nat -C POSTROUTING -j MASQUERADE -s 172.18.${SUBNET}.0/24 -d; then
        iptables -t nat -A POSTROUTING -j MASQUERADE -s 172.18.${SUBNET}.0/24 -d

      # create a docker working directory
      mkdir -p ${DOCKER_ROOT}
      chown $u:$u ${DOCKER_ROOT}

      # launch a docker daemon
      docker daemon -D \
        -g ${DOCKER_ROOT}/g \
        --exec-root=${DOCKER_ROOT}/e \
        -b ${BRIDGE_NAME} \
        --dns= \
        --iptables=false \
        -H unix://${DOCKER_ROOT}/docker.sock \
        -p ${DOCKER_ROOT}/ > ${DOCKER_ROOT}/docker-${u}.log 2>&1 &

    OFFSET=$(expr ${OFFSET} + 1)



First, create a bridge for the user, with a subnet starting from, then set IP masquerade which is necessary if you run the daemon with –iptables=false.  Then create user’s own Docker working directories, both for exec-root and graph (container caches and filesystem images).  Also, set the own unix domain socket and pid file location.  Set DOCKER_HOST to the unix domain socket specified in the -H option.


Then make sure you run a net container before starting other containers.  The net container can be anything, but it just needs to be there.

FROM ubuntu:14.04
CMD ["/bin/sh", "-c", "yes | while read line; do sleep 1; done"]
$ docker run -t -d --name net net

Let’s run Redis in the network of this net container.

$ docker run --net=container:net -d --name redis redis redis-server --appendonly yes


The last piece of this work is, don’t forget update your bash_profile to set DOCKER_HOST to be pointer to the unix domain socket path.  Then as you log in the host, you see your own docker process spaces, as if you were exclusively using the host machine with a single Docker environment.


We’ve just launched our deep learning-based trading platform, Capitalico, which is heavily using Docker, and welcoming beta test users.  If you are interested in it, please sign up for the waiting list so we can contact you!


Deep-Learning Startup Alpaca Raises $1M to Bring A.I. into Personal Investing

Deep-Learning Trading Platform Lets People Build Algorithms With Few Clicks On Historical Chart

Oct. 19, 2015 (San Mateo, CA) — Alpaca, a San Mateo based Deep-Learning startup enabling AI technology to automate professional human tasks, announced today the closing of $1,000,000 seed funding from venture capitals and angel investors to focus its business in FinTech. The company also announced the private beta launch of its Deep-Learning trading platform, Capitalico, that lets people build trading algorithms with few clicks and visually from historical chart.

Alpaca fully leverages its Deep-Learning technology into personal investing sector with Capitalico, where even beginner traders can build their own trading algorithms intuitively without any programming. Existing algorithm building services always have involved programming which is not familiar for most discretionary traders and beginner traders.

“We saw an opportunity to support traders and to-be traders who understand and process the market data visually from charts and technical indicators, and for them to backtest and automate their trading strategies,” said Yoshi Yokokawa, Alpaca’s co-founder and CEO. “I am surprised myself how easy and simple the experience is to develop complex trading algorithms on Capitalico by simply selecting a portion on historical chart without worrying anything about programming at all.”

Capitalico is now accepting signups to the waiting list for the early access to the service. Capitalico is scheduled to be available public as beta beginning of next year.

About Alpaca

Alpaca is a San Mateo based Deep-Learning startup enabling AI technology to automate professional human tasks. 2015 June, Alpaca made its debut with the easiest deep-learning SaaS for computer-vision “Labellio,” which was featured in NVIDIA’s technical blog “PARALLEL FORALL” for its advanced backend technology used in the application to enhance the existing deep-learning capability to be faster and parallel. For more information, visit .




#  #  #

Publishing Alpaca’s Presentation Slides for NVIDIA’s GTC Japan 2015

Yuki, our Chief Engineering Officer, gave a presentation at NVIDIA’s GTC Japan 2015 on “Experiments to apply Deep Learning to Forex time series data” on September 18th, 2015.  We would like to thank you for those who came to the event and spent time with us and our presentation!  For that, we are publishing the presentation slides that we used at the event on the SlideShare.

GTC Japan 2015 – Experiments to apply Deep Learning to Forex time series data from Yuki Hayashi

We continue to contribute for the deep-learning technology to get utilised in many ways.  Please follow us on twitter and Facebook!

Yuki presents at NVIDIA’s “GTC Japan 2015” Conference

On September 18th, 2015, at NVIDIA’s GPU Technology Conference “GTC Japan 2015” (URL:, our Chief Engineering Officer, Yuki gives technical presentation on “Analysing time series data of the currency market by using deep-learning.”

Yuki plans to introduce cutting edge technical tips on LSTM-based neural network structure as well as how we are maximising GPU capability on cloud.  Please check out the conference’s website for more detail.  We are looking forward to seeing you there!

Hitoshi and Yuki contribute a technical post to NVIDIA’s PARALLEL FORALL about Labellio

We are excited to let you know that Hitoshi (CTO) and Yuki (Chief Engineering Officer) have contributed the first post to NVIDIA’s technical blog “PARALLEL FORALL” with writing about the technical background of Labellio, the easiest deep-learning platform for computer vision ( ).

Open source pure python Selective Search and advanced object recognition with Labellio

There are various use cases for object recognition, categorized as follows.

  • Object Classification to recognize an object assuming one in an image
  • Localization to locate bounding box where individual objects are captured, assuming multiple objects are in an image
  • Segmentation to assign labels to pixels
  • There is an approach to learn description of visual semantics called Image Captioning, combining feature extracted from CNN into RNN

In the past, we explained labellio_cli, which does this Object Classification simplest amongst these.  To add Localization to available tasks with Labellio, we are happy to open source pure python Selective Search in GitHub, which you can also use in other tools such as Caffe.

What is Selective Search?

You could do Exhaustive Search that stupidly scans windows by sliding them to achieve Localization.  However, there would be too many regions to process and you would hit limitation on the shape or size you can process.

A solution to that is an algorithm called Selective Search that groups similar regions at pixel level and selects candidates from them.  It is a well known approach called R-CNN that uses Selective Search and CNN.  A similar technique is used in GoogleNet pipeline.  Unfortunately, there has been not easy-to-use open implemetation of it.  Caffe has example code that does this Selective Search, but it requires MATLAB, which itself is hard to install and setup, and you may give up like us.  In order to solve this situation today, we are open sourcing Selective Search module in pure python module.

This module does not require or assume Labellio, and it is in pure python, you can use it for any other Deep Learning libraries other than Caffe, such as Chainer, and we believe this will be used as a general solution for object recognition.  A bit off topic, but there is another proposed technique called OverFeat, which embodies regression for localization in its neural network.

Selective Search usage: Use Labellio to train cake classification model and extract object regions by Selective Search

As an example of how powerful Selective Search is, the following is to recognize different cakes in an image with multiple of them in a plate.

First of all, let’s build a cake classifier.  We collect training data of mont blancs, cheese cakes and shortcakes from Bing search.  We are not sure if the image search returns expected results, so go through the manual labelling mode.  As expected, the search results included some other objects that are not actually mont blanc cakes.



Have a coffee to wait until the training completes.



And download the model once it completes.


That’s it for creating a model.  The accuracy is as good as 96% and enough to use in this demo.

Detect candidate regions by Selective Search

Next is the main topic of Localization using Selective Search.  We are using the following image that has two cakes in it.


Selective Search subdivides the image with an exsting algorithm and continuously combines them into bigger regions based on the similarity.  In the original paper, Felzenswalb is used for the initial subdivision.  It will look like the following if you color the extracted small regions.


It combines these subdivided regions into bigger one by looking at similarity of color histgrams and textures, and finally builds one region.  This is done in the Alapaca’s Selective Search module by calling an API.  It is relatively simple implementation, but the result is good.  We also welcome pull requests if you can improve something such as speed.

You can install it via pip if it’s installed.

$ pip install selectivesearch

There is only one single API, selective_search, and you pass loaded image and parameters.  You can see an example here.

The following is to show the bounding boxes as the results of Selective Search.


Finally, you classify each region using CNN.  We pass sub-images into Labellio CLI and get the following result.  We set a threshold of the result probability to filter noise outside of interesting ones.


As you can see, you can easily build localized object recognition using Selective Search and CNN.  We hope this helps you use more of Labellio with our Selective Search implementation.

How to build image recognition API server with Labellio

Once you created your image recognition model on Labellio, I’m sure you want to try it in your app.  In this entry, I would like to share how to build your image classification API server using your model in your app.


Expected use case

This example assumes that you want to add new feature in your web application or smartphone app by using the recognition model you built on Labellio.  Since your app already exists, it makes a lot of sense to implement it as a service in microservice architecture, from maintainability and scalability perspective.

Let’s say we want to implement a very basic Web API server.  It will serve “http://[your domain]/classify as an endpoint that accepts images in HTTP POST method, and responds with the most probable label with score in JSON format.

The code here is very minimalistic for the explanation purpose.  In the real production development, you should consider many other things like image format check, error handling etc, which we don’t cover here.


Runtime environment

It is best to use GPU instance in AWS to implement this kind of server very quickly.  We showed how to set it up in the previous post you can find below.  Image classification is relatively heavy workload, but at the same time, it is easier to increase the number of machines to scale it out if you use AWS.

We use the same Alpaca’s AMI ami-9fe512db in the N. California region as in the previous post.

We use Python 2.7, since it is the easiest way to implement as Caffe interface and Labellio library are implemented in it.


Sample code

Here you can find the sample code.

We save image files in the working directory specified by UPLOAD_FOLDER.  Caffe model file downloaded from Labellio is stored in the MODEL_FOLDER directory.  The supported image file format is JPEG and PNG.  We only check file extension from the file name.  The program receives images sent through “/classify” in the POST method and temporarily store it in UPLOAD_FOLDER, then get the label (answer) using labellio_cli, and returns the result in JSON format.



1. Login to the GPU instance created in AWS using SSH.

2. Install Flask.

$ sudo pip install flask

3. Run the following command. This enables labellio_cli to be used.

$ source /opt/caffe/caffe.bashrc

4. Untar your Caffe model and rename it as `model` directory.  Make a directory `tmp` to put images in.

$ tar xzf [model filename]
$ mv [untarred model directory] model
$ mkdir tmp

5. Save the example code above as ``, and run the following command.

$ python



It will run Flask server listening on TCP 5000.  When testing, save the image file to classify in your local and point your HTTP client (such as curl) to the port 5000 on the server.

$ curl -F "image=@test.png"  http://[your domain]:5000/classify

Make sure your Security Group opens up the port 5000 to let the incoming network connection go through to the server.

You will get the classification result, score and class label name in JSON format.

{"0": {"score": [0.7100653052330017, 0.07795180380344391, 0.21198289096355438], "label": "alpaca"}, "label_name": {"sheep": 2, "gorilla": 1, "alpaca": 0}}

This results reads `alpaca` is the most probable class label amongst sheep, gorilla and alpaca.

Now you are ready!  Prepare your client app to perform the same HTTP POST and get the classification result.  In the real use case, you can also think about tweaking such as ignore classification result with lower score than threshold, depending on your application requirements.

How to use Caffe model generated by Labellio on your machine/AWS

Labellio makes it super easy to build your own image recognition model in a few minutes.  The last thing you want to do is to use your model in your application.  In this entry, I explain how to use your model using Docker and AWS, via Labellio’s python interface that’s publicly available.

The simplest example to use the Caffe model exported by Labellio is to build a Web API server that returns classification result from input image data.  We recommend using GPU instance in AWS to accelerate the computation.

As a starting point, I describe how to setup the server environment to run such an API server, and will show how to write API server in the following post.  If you follow the procedure I show here, it’s not difficult to build your own API server at all.


Setup GPU instance on your machine or AWS

Here is how you configure your machine with GPU attached.  You may need to use sudo to run docker depending on your system configuration.  Alternatively, you can use our AMI ami-9fe512db that we make publicly available, our AWS Northern California Region.

1. We assume you have a GPU machine with the following installed in.

2. Download the docker image.

$ sudo docker pull alpacadb/labellio-caffe-image

3. Confirm the image is pulled.  You can see alpacadb/labellio-caffe-image is available after that.

$ sudo docker images
alpacadb/labellio-caffe-image   latest 4bba1de016cb 9 days ago 3.661 GB

4. It is necessary to install and load the GPU driver on the host machine to use CUDA on docker container.  Until you run the deviceQuery command below, the driver has not yet been loaded.

$ lsmod | grep nvidia
nvidia           	8370760  0
drm               	303102  4 ttm,drm_kms_helper,cirrus,nvidia

5. Load the driver.  In order to load it, you need to some command that uses CUDA.  Here we use the deviceQuery command.  From the CUDA Toolkit, build the command and run it.

$ cd NVIDIA_CUDA-7.0_Samples/1_Utilities/deviceQuery
$ make
$ ./deviceQuery

6. You can now see the driver is loaded.

$ lsmod | grep nvidia
nvidia_uvm         	67139  0
nvidia           	8370760  1 nvidia_uvm
drm               	303102  1 nvidia

7. Kick off the alpacadb/labellio-caffe-image container using the docker command.  Remember to add –privileged option.

$ sudo docker run -ti --privileged -v /dev/null:/dev/raw1394 alpacadb/labellio-caffe-image:latest /bin/bash

Note: -v /dev/null:/dev/raw1394 is not mandatory, but you may see “libdc1394 error: Failed to initialize libdc1394” error.  This error is not problematic, but with the “-v” option we can suppress it.

8. Now you can go to the next step.


Use your image recognition model from the command line tool

We assume you are on the AWS system logged in via SSH.  In the docker environment, you may need to add the `docker exec` command.

1. Build your Caffe model to recognize images in Labellio.  In this example, “gorilla”, “alpaca”, and “sheep” labels are used.

2. Download the Caffe model files from Labellio.  If you have your model finished training, you can download it from `Download Model` button.  The model filename should look like 6db79d9b4e6552ef15586ab3ddd08c15fe260892.tar.gz.  Untar the file and rename it to `model`.

$ tar xzf [model filename]
$ mv [untarred directory] model

3. Have images to be classified.  In this example, we use images that are OK to use commercially.  Download them into your working directory named `img`.

4. Make sure the following file is sourced before using labellio_classify.

$ source /opt/caffe/caffe.bashrc

5. Run this command as it is installed in the system already.

$ labellio_classify model img

6. You will see the output as follows.

img/alpaca.jpg    alpaca    [ 0.66606939  0.07539421  0.25853643]
img/sheep.jpg    sheep    [  7.81783648e-03   3.10965086e-04   9.91871238e-01]
img/gorilla.jpg    gorilla    [ 0.00264297  0.99490935  0.00244769]

It shows filename, classification result and classification score in the order.  You can know the index in the array from the labels.json file in the model directory.  In this example, alpaca is the first, gorilla is the second and sheep is the third number.

$ cat model/labels.json && echo
{"sheep": 2, "gorilla": 1, "alpaca": 0}