After last week’s Internet Of Things hack session, I became fascinated with all the fun IoT projects and technologies there are to play with. Everything you do, requires data to go into the cloud, so I figured I’d start there. In the hack series I learned how to push data into AWS Kinesis which is amazing, however it has its disadvantages. On the flip side there is the new AWS API Gateway which I also haven’t spent much time on, so that’s a perfect thing to learn. After some tinkering, I found that while AWS API Gateway -> Lambda -> DynamoDB might sound complicated, it’s actually quite easy.
Before we get started, why API over Kinesis? First, Kinesis currently costs $0.015 per hour, per shard. You need a minimum of 1 shard, so that’s a minimum of about $11 a month. If you’re sucking in data fairly constantly, this is well worth it. However if you’re tinkering around with a test IoT device, why pay money that you don’t have to? On the flip side API Gateway is charged per million API calls. Secondarily, I like working with APIs of my own design, rather than Amazon designed APIs. It means I can build lightweight interfaces on more platforms.
We’re going to start at the “bottom” and work our way upwards, so first DynamoDB.
|
Step #1 — Create a new table. Primary Key = Hash & Range Hash Name = device (Type string) Range Name = timedate (Type number) |
|
|
Step #2 — Skip the global indexes. We don’t need them for our small test. |
|
|
Step #3 — Provision Throughput. I went with 5 read, 2 write. This costs ~$1.46 a month. |
|
| Step #4 — Turn on alarms |
|
| Step #5 — Create! |
|
With that DynamoDB is done and will create itself shortly. Since it’s a NoSQL database, we don’t need to worry about the data going into it at all. So the next step is creating our Lambda function which will take the API call data and push it into DynamoDB.
|
Step #1 — Create a new Lambda function. Skip the “blueprint” Name = OurBlogLambda Runtime = Node.JS Get the code from this gist. |
|
|
Step #2 — The rest of the Lambda answers. Handler = index.handler (that’s the default) Role = Select “Basic with DynamoDB”, this will spawn a new window to walk you through and create a new IAM role. When you’re done, you should have a role named something like “lambda_dynamo”. Leave the advanced settings alone. |
|
| Step #3 — Review and create Lambda function. |
|
|
Step #4 — Test. Click the test button and paste in the following JSON: {
"device": "test",
"lat": "-1.1",
"lon": "2.2"
}
|
|
|
Step #5 — If everything went well, you’ll have a “SUCCESS” If you want, you can also go check DynamoDB and you should see a record in your database now. |
|
With that, you’ve completed the Lambda setup process. As you should have been able to figure out, our data of choice to be submitted is latitude & longitude. You can very easily change this Lambda function to be any sort of data type and/or names. You can also add/subtract the number of items being submitted to Dynamo, in this function, at any time. So with everything working, now we move on to the API Gateway.
| Step #1 — Create a new API. |
|
|
We’re going to name this “submitData” since it’s taking data in (to Lambda). Later you could create more resources for getting data out. Update 2016-02-29 Per the comments it looks like this step is no longer required. |
|
|
Step #3 — Create a method. We’re submitting data, so POST makes the most sense. Make sure you were still selecting your new “/submitdata” resource. |
|
|
Step #4 — Setup the POST to go to Lambda. Click “POST” under “/submitdata” to the left. We’re using a Lambda function and if you start to type the name, it should auto-fill for you. |
|
|
Step #5 — Save & Accept permission change. When you go to save your changes, AWS will confirm you want to give API Gateway permission to invoke Lambda. |
|
|
Step #6 — Confirm everything looks correct. Your “/submitdata — POST — Method Execution” should look almost identical to my screenshot. |
|
|
Step #7 — Test it! Click the test button and put the following code into the “Request body”: {
"device": "apitest",
"lat": "-10.01",
"lon": "20.02"
}
You should see a “SUCCESS” in the response body. You can also look at DynamoDB to confirm your row is in the database. |
|
|
Step #8 — Deploy API. Click “Deploy API”. Deployment Stage = New Stage Stage name = testing |
|
|
Step #9 — Get the URL. If everything saved previously you should now see ” Invoke URL: https://RANDOMVALUEHERE.execute-api.us-west-2.amazonaws.com/testing “. You’re almost done! |
|
|
Step #10 — Curl Test On your command line try: curl -H "Content-Type: application/json" -X POST -d "{\"device\":\"curl\",\"lat\":\"12.34\",\"lon\":\"45.67\"}" https://CHANGEME.execute-api.us-west-2.amazonaws.com/testing/submitdata(make sure that the URL matches that which you got in the previous step). You should see “SUCCESS”, and be able to see your data in DynamoDB. |
|
Congratulations, you’re all done! You can now accept data POST’ed to a REST API, fed into Lambda, and stored in DynamoDB. One caveat for you IoT people reading this tutorial though, AWS API Gateway is HTTPS ONLY. Unfortunately it doesn’t not operate on regular unencrypted HTTP. So if you’re trying to submit data directly from an Arduino, you’re out of luck. You’ll need to use a headend with more CPU power to submit to API Gateway.
Now take this skeleton and get fancy with it. Are you accepting data in one format (ex: Degrees centigrade) and need it in another (ex: Fahrenheit)? Rather than doing that in your device, Lambda can do the conversion before storage. In fact, Lambda can do a ton more (e.g. store data in S3, Launch EC2 instances, fire off messages in SNS), it can do basically anything the AWS SDK for NodeJS can do.