Getting Started With Manta

Manta is a powerful new type of object store from the same folks that invented Node.js. The cool part is that Manta isn't just an object store - it's an object store with built-in compute. That means that you can run code and other familiar commands against the objects directly on the server.

I'll walk you through setting up Manta, and demonstrate a simple example of where you could use the compute job functionality. It's assumed you have already setup a Joyent account and have node.js installed on your machine.

Setup Guide

1. Install Manta

npm install -g manta

2. Set your Environment Variables

You can find these on your dashboard page. Just copy and paste them into your .bashrc then source with . ~/.bashrc

export MANTA_URL=https://us-east.manta.joyent.com
export MANTA_USER=your_user_name
export MANTA_KEY_ID=fd:f8:08:22:11:b2:ef:db:87:09:c0:2b:4b:fd:99:f7

3. Upload a File

At this point you should be ready to upload a file. Manta includes command line utilities for listing, creating, and destroying files and directories. Let's use this sample file.

mput /$MANTA_USER/public/beer.jpg -f beer.jpg
wget https://us-east.manta.joyent.com/$MANTA_USER/public/beer.jpg

You're done!

Working with Compute Jobs

That was easy!

Manta is pretty easy to use, but that's really only the beginning. It's got tons of powerful functionality built into it. That functionality comes in the form of map reduce jobs which you can easily run with a single shell command.

Let's start with a simple example.

Create a Thumbnail

mfind /$MANTA_USER/public | \
mjob create -w -m 'convert $MANTA_INPUT_FILE -resize 100x jpeg:- | mpipe ${MANTA_INPUT_OBJECT%.*}-100x.jpg'

This will output something like:

81e40938-d33c-42da-ab61-33527360e366
added 1 inputs to 81e40938-d33c-42da-ab61-33527360e366

Then you can retrieve your thumbnail:

wget https://us-east.manta.joyent.com/$MANTA_USER/public/beer-100x.jpg

Check out the result

Let's Think at Scale

That's a simple one-liner that will create thumbnails for all of your images, which is pretty amazing considering the alternative would be to list all your images, download them, convert them, then put them back. If you have hundreds of millions of images in your object store, then that old process can take days.

With Manta it just takes a few seconds.

Let's Clean it up a Little

That little example shows how powerful Manta can be. But we're going to make this a tiny bit fancier. What if you need 4 different thumbnails? Writing that all out on the command line might get a little ugly. At the end of the day though, you're just running code that would run on any server, like a shell script or a node app, etc.

Luckily we can submit these scripts as arguments to our map reduce jobs in Manta. These files are called assets.

Here's an example shell script we'll use as our asset:

#!/usr/bin/env bash

sizes=( 128 256 512 )

for i in "${sizes[@]}" ; do
    convert $MANTA_INPUT_FILE -resize ${i}x jpeg:- | \
      mpipe "${MANTA_INPUT_OBJECT%.*}-${i}x.jpg"
done

Let's upload it as an asset:

mput -f thumb.sh /$MANTA_USER/stor/thumb.sh

And create our job again:

mfind /$MANTA_USER/public | \
mjob create -w -s /$MANTA_USER/stor/thumb.sh \
-m 'bash /assets/$MANTA_USER/stor/thumb.sh'

Now we have 3 thumbnails:

mls /$MANTA_USER/public
beer-128x.jpg
beer-256x.jpg
beer-512x.jpg
beer.jpg

Next Steps

This is a simple example that hopefully shows you what Manta can do. The documentation is pretty extensive and they're on irc.freenode.net under #manta if you have any questions.