- Fantastic finance management app with support for multiple currencies, ideal for people on the move
Monday 29 January 2018
Toshl App
Toshl Finance
Digital Nomads - Everything
Contents
Travel Planning
Retreats
Flights
Accommodation
Getting Around
Communities & Social
Remote Jobs
Gear
Finance
Security & Backup
Communication
Languages
Food & Drink
Team Collaboration
Miscellaneous
Destination Guides
Travel Planning
Lonely Planet - The most recognised travel guides for over fourty years, start here for inspiration.
Nomad List - Search for your next base using digital nomad friendly filters such as internet quality, living costs, and places to work.
Atlas Obscura - The definitive guide to the world's wondrous and curious places.
Wikivoyage - Fantastic free travel guides that anyone can edit.
Triposo - Smart offline travel guides for your phone and tablet - iOS and Android.
Time Out - Worldwide guide to art and entertainment, food and drink, film, travel and more.
The Basetrip - Essential information for your next destination and how it compares to your home country.
DESTIGOGO - Planning a short trip away from work on a budget? Use this simple tool to discover destinations that fit your timeframe and budget.
VisaHQ - Easily find visa requirements for most countries around the world.
Numbeo - The world’s largest database of user contributed data about cities and countries. Find information such as cost of living, housing indicators, health care, traffic, crime and pollution for your next destination.
On The Grid - Neighborhood guides written by local creatives. Great for design-oriented travelers.
Retreats
Destination: Dev - Program that brings together 10 like-minded individuals for eight intense weeks of software development education and cultural immersion.
Hacker Paradise - Hacker Paradise have been organising three month long coworking retreats for developers, designers and entrepreneurs since 2014.
LiveWorkFit - A series of affordable one month retreats with a focus on mind and body health to maximize your business and personal potential.
Nomad House - Spend 10 days exploring a new location, working hard (anywhere) and connecting to a community built on shared values.
Unsettled - 30-day coworking retreats to take your work, life, and adventures beyond traditional borders.
WiFi Tribe - Coworking retreat which allows you to hop on or off at any point, stay a month, or a year to explore a whole continent with like-minded individuals, one country at a time.
Wifly Nomads - A brand new retreat launching in September 2017 designed to help fledgling nomads build startups, find remote work opportunities and learn nomadic lifestyle fundamentals.
Nomad Cruise - Join over 150 other remote workers, entrepreneurs and digital nomads on a 15-day coworking cruise across the atlantic.
Flights
Kiwi.com - Advanced flexible search tool for finding cheap flights.
Kayak - View pricing trends for journeys and set up price alerts.
AirWander - New kid on the block that helps you discover stopover destinations that'll save you money on your journey.
FlyOnward - Rent an onward flight ticket for $9.99 to satisfy immigration requirements in countries such as Malaysia and New Zealand.
App in the Air - Manage your upcoming flights and view your flight history - iOS, Android, and Windows Phone.
Flightfox - Online travel agent that guarantees to save you money on your trip.
SeatGuru - Aircraft seat maps to find the best seats for your flight.
SleepingInAirports - Detailed airport guides to make the most of your time in limbo.
Priority Pass - Gain access to over 1000 airline lounges worldwide with membership from US$99.
LoungeBuddy - Book airport lounge access from US$25 in over 200 lounges worldwide.
Hopper - Price prediction app that helps you choose the cheapest time to book your flight - iOS and Android.
GTFO - Get The Flight Out - In a hurry? GTFO lists the cheapest flights out of your home airport within the next 24 hours iOS.
Accommodation
Airbnb - Short term rooms and private apartments at affordable rates.
Kayak - Compare hotel prices across leading booking websites including Expedia, Booking.com, Agoda, and Hotels.com.
Hostelworld - Budget accommodation for nomads that prefer a more social environment or need to keep costs down.
Couchsurfing - The ultimate budget accommodation, meet new people and join in local experiences in exchange for a free bed or couch.
Colive.co - Find and organise coliving arrangements with other nomads.
Coliving - Find coliving spaces around the world to live with like-minded individuals.
Getting Around
Moovit - The worlds largest transit app supporting over 1,200 cities in over 70 countries - iOS and Android.
Citymapper - Arguably the best transit app available but limited to 40 cities. Use it when you can and fall back to Moovit if necessary - iOS and Android.
Rome2rio - Global trip planner which gets you from A to B using flights, ferries, trains and buses. Great for when you're moving between cities - iOS and Android.
Uber - The big daddy of ride sharing apps, it goes without saying this should be on your phone. Available in over 500 cities worldwide - iOS, Android, and Windows Phone.
Lyft - The largest competitor to Uber in the United States for people who want an alternative - iOS and Android.
Grab - Uber may be your go to taxi app for most of the world but, in SE Asia, Grab is king - iOS and Android.
Wikivoyage - When Rome2rio or Moovit fail to provide directions you may just find the information you need on Wikivoyage. Destination guides here typically have 'get in', 'get around' and 'get out' sections, useful in some of the more rural areas.
Maps.me - Offline maps and POIs to get you around on foot when you don't have an internet connection - iOS, Android, and Kindle Fire.
Waze - The world's largest community-based traffic and navigation app. Perfect for those brave enough to drive abroad - iOS and Android.
The Man in Seat Sixy-One - The ultimate guide to train travel around the world.
Communities & Social
Nomad List Chat - Connect with other nomads to get advice or organise meetups.
Digital Nomad Forum - Need advice from other nomads but don't have time to sit around waiting for answers in the chat room? Start a topic on the forum...
Couchsurfing - Couchsurfing isn't just for free sofas. Find groups and social events in cities around the world.
Meetup - Find and organise meetups in cities across the world.
Facebook Groups - Find groups for buying/selling, hiking, common interests, local advice and more.
Nomad List Radar - Keep track of your nomadic friends whereabouts and meet up for 🍻 when your paths cross.
Withlocals - Find paid tours and experiences with verified locals in over 20 countries.
Party with a Local - Find locals and travelers in your area that are down to party - iOS and Android.
Remote Jobs
Remote: Office Not Required - Great book by 37signals, which shows both employers and employees how they can work together, remotely, from any desk, in any space, in any place, anytime, anywhere.
We Work Remotely - Find your next remote gig with the companion website to 'Remote: Office Not Required'.
Remote Ok - Large directory of remote jobs from across the web.
Freelancer - The world's largest freelancing and crowdsourcing marketplace by number of users and projects, connecting over 22 million employers and freelancers.
Toptal - Join an exclusive network of the top freelance software developers, designers, and finance experts in the world. Top companies rely on Toptal freelancers for their most important projects.
Workspaces
Workfrom - The most comprehensive directory of public and private workspaces across the globe.
Café Wifi - Find cafes nearby that are suitable for working from and details about them such as wifi speeds, power outlet availability and seating comfort - iOS.
Copass - Global network of over 500 coworking spaces accessed via a single monthly membership.
Hoffice - Find home offices to work from for free and meet new people in the process.
Work Hard Anywhere - A collection of 7800+ work-optimized spaces in the palm of your hand. Mobile. Easy. Constantly updated by your fellow entrepreneurs, freelancers, and creatives - iOS and Android.
Gear
Roost Laptop Stand - The original portable back-saving laptop stand.
Corsair Flash Voyager Mini - Compact and stylish USB key which packs up to 128GB of memory.
Tortuga Outbreaker Backpack - Pack smart with this second generation Tortuga carry on backpack.
Minaal Carry-on 2.0 - A (more stylish?) alternative to the Tortuga Outbreaker with similar features. Check them both out and see which you prefer.
Anker Portable Chargers - USB battery packs are an essential bit of kit these days and you can't go wrong with something from Anker.
Omnicharge - For a portable charger that packs a bit more punch, check out the Omnicharge with its universal power outlet that will let you charge devices such as laptops in addition to your phone and tablet.
Logitech MX Anywhere 2 - Logitech have a track record of releasing high quality peripherals and when it comes to portable mice they don't disappoint with the MX Anywhere 2.
UE Boom 2 - If you don't mind the extra weight then portable speakers are a great way to turn any room into a home. The UE Boom series consistently comes out top in reviews thanks to great sound, rugged protection and good looks.
JBL Clip 2 - If you mind carrying extra weight, the JBL Clip 2 is a good option with only 184g. Quality is not as good as a stereo speaker but it still does the job pretty well.
LaCie Rugged Mobile Storage - Keep your backups safe with a drive from the LaCie rugged series, trusted by professionals across the globe.
Bose QC20 Noise Cancelling Earphones - Public transport can be a noisy affair but if you have the cash to spare then the QC20s are the best around when it comes to noise cancelling headphones in a small package.
Google Chromecast - Chromecast is a media streaming device that plugs into the HDMI port on your TV. Simply use a compatible mobile, tablet or PC device and the TV in your accommodation to watch your favourite TV shows, films, music, sport, games and more on the big screen
Kindle Paperwhite - Highly portable, easy on the eyes, readable in sunlight, instant access to the worlds largest bookstore and capable of holding over 1,000 books – the Kindle Paperwhite provides a great combination of features and affordability.
Mountie - Clip your smartphone or tablet onto your laptop for a portable dual monitor set up. Works well with Duet Display.
Gerber Shard - Want to take a multi-tool with you but can't get traditional ones past airport security without checked baggage? The Gerber Shard is a great solution to this predicament, packing six useful functions into an airline friendly form-factor.
Finance
Revolut - Low fee travel money card with beautiful companion apps for iOS and Android.
xCurrency - Beautiful and easy to use currency converter - iOS and Android.
TransferWise - Easy to use, low fee, money transfer service supported in over 50 countries.
Toshl Finance - Fantastic finance management app with support for multiple currencies, ideal for people on the move - iOS, Android, and Windows Phone.
The Elements Of Investing - A timeless, easy-to-read guide on life-long investment principles that can help any investor succeed.
Quidco - The UKs largest cashback website, check here before booking flights and accommodation to see if you can get cashback on your purchases.
Neomy - Set-it-and-forget-it exchange rate tracking, straight to your inbox.
Moneytis - Compare foreign money transfer services to make sure you get the best deal.
Monzo - Smartphone-friendly UK bank with zero fees abroad.
Security & Backup
Avast Antivirus - The worlds number one free antivirus solution.
1Password - The gold standard of password management apps, you should be using a password manager anyway but it's especially important on the move.
Cloak - Having a secure VPN service is important when using public wifi networks. Cloak is a great choice in the Apple ecosystem with beautiful apps for Mac and iOS.
Dropbox - The best all-rounder cross platform cloud storage service.
Google Drive - The cheapest cloud storage service currently available.
BackBlaze - Low cost and easy to use cloud backup for Mac and PC.
SpiderOak One - Dropbox, Google Drive and Backblaze all take your data security very seriously but if you're especially concerned about privacy then check out the Snowden endorsed SpiderOak.
Authy - The best choice for cross platform two-factor authentication.
Ghostery - Fantastic extension for blocking trackers in all major web browsers.
Purify - The most efficient app for blocking ads & trackers on iOS.
Find My iPhone - If you're in the Apple ecosystem then be sure to enable Find My iPhone on all your devices.
Communication
Facebook Messenger - The best way to keep in touch with friends and family – chat, video, voice calls and group chat for free.
WhatsApp - Keep in touch with people you don't have on Facebook, it goes without saying that WhatsApp should be in your kit. Available for most platforms.
Skype - The defacto option for many when it comes to business video calling.
join.me - The easiest way to organise group conferences with support for scheduling, video, whiteboarding and screen sharing.
Slack - Slack has quickly become the go-to team communication tool for many businesses. Get your company on board the train if they're not already.
WiFi Map - Get access to over 100 million free wifi hotspots worldwide - iOS and Android.
Prepaid Data Sim Card Wiki - Find out information on prepaid sim cards in over 150 countries.
ChatSim - Unlimited chat using your favourite apps in over 150 countries for a single yearly fee.
Project Fi - One single low cost plan for all of your phone calls, SMS and mobile data worldwide - U.S Citizens on Android only.
Languages
Google Translate - The big daddy of translation apps with support for over 100 languages and 50 offline languages as well as photo translation - iOS and Android.
Point It: Traveller's Language Kit - Passport sized traveler's picture dictionary containing photographs of 1,200 items grouped into categories such as food and drink, accommodation, transportation, automobiles, entertainment, nightlife and health.
Food Allergy Translate - Mobile apps and physical cards to notify restaurants of your food allergies - iOS, Android, and Windows Phone.
Fluent In 3 Months - If you've always struggled with learning foreign languages then be sure to check out this great book by Benny Lewis.
Duolingo - The original and arguably most enjoyable language learning app with support for over 20 languages - iOS, Android and Windows Phone.
Memrise - Not quite as fun as Duolingo but still a solid contender on the language learning scene with support for over 200 languages - iOS and Android.
Food & Drink
Foursquare - Find the best places to eat, drink, shop, or visit in any city in the world - iOS, Android, and Windows Phone.
HappyCow - Find great vegetarian and vegan restaurants near you, Foursquare for animal lovers 🐮 - iOS and Android.
Don't Eat Alone - Make informal meal plans with strangers & friends – connect with people over a meal.
Team Collaboration
Standup Bot - Run your standup meetings remotely via Slack and have your meeting notes automatically organised for you.
I Done This - More than 160,000 people use I Done This’s easy daily check-ins and powerful progress reports to run more effective and productive teams.
Timezone.io - Keep track of 'when' and 'where' your remote team members are. Easily organise meetings at a time suitable for everyone involved.
Dropbox Paper - Collaborate on documents and ideas with your remote colleagues using this flexible workspace.
Trello - Trello’s boards, lists, and cards enable you to organize and prioritize your projects in a fun, flexible and rewarding way.
Basecamp - Basecamp organizes your projects, internal communications, and client work in one place so you have a central source of truth.
Miscellaneous
Noizio - Ambient sound mixer to help you focus in busy places. Parisian Cafe during a thunderstorm? Check! - iOS and Android.
Duet Display - Use your iPhone or iPad as an extra display for your Mac or PC - iOS, macOS and Windows.
One Way Ticket - A documentary looking at the rise of the digital nomad around the world. Through interviews with advocates, academics and authorities, this film examines issues surrounding the supposed "dream life".
Sunday 28 January 2018
All In All
Important and awesome:
- artificial intelligence
- internet of things
- big data
- crypto/blockchain tech
Internet Of Things - Everything
Hardware
Arduino - Arduino is an open-source electronics platform based on easy-to-use hardware and software. It's intended for anyone making interactive projects.
BeagleBoard - The BeagleBoard is a low-power open-source hardware single-board computer produced by Texas Instruments in association with Digi-Key and Newark element14.
Dragonboard - The DragonBoard 410c, a product of Arrow Electronics, is the development board based on the mid-tier Qualcomm® Snapdragon™ 410E processor. It features advanced processing power, Wi-Fi, Bluetooth connectivity, and GPS, all packed into a board the size of a credit card.
HummingBoard - HummingBoard is a family of three Linux- and Android-ready, open source SBCs based on 1GHz Freescale i.MX6 SoCs, with a Pi-like 26-pin I/O connector.
Intel Galileo - The Intel® Galileo Gen 2 board is the first in a family of Arduino*-certified development and prototyping boards based on Intel® architecture and specifically designed for makers, students, educators, and DIY electronics enthusiasts.
Microduino - Microduino and mCookie bring powerful, small, stackable electronic hardware to makers, designers, engineers, students and curious tinkerers of all ages. Build open-source projects or create innovative new ones.
Node MCU (ESP 8266) - NodeMCU is an open source IoT platform. It uses the Lua scripting language. It is based on the eLua project, and built on the ESP8266 SDK 0.9.5.
OLinuXino - OLinuXino is an Open Source Software and Open Source Hardware low cost (EUR 30) Linux Industrial grade single board computer with GPIOs capable of operating from -25°C to +85°C.
Odroid - The ODROID means Open + Droid. It is a development platform for the hardware as well as the software.
Particle - A suite of hardware and software tools to help you prototype, scale, and manage your Internet of Things products.
Pinoccio - Pinoccio is a solution to add mesh networking capability and WiFi-Internet access to all yout IoT devices, and it is Arduino compatible.
Raspberry Pi - The Raspberry Pi is a low cost, credit-card sized computer that plugs into a computer monitor or TV, and uses a standard keyboard and mouse. It’s capable of doing everything you’d expect a desktop computer to do, from browsing the internet and playing high-definition video, to making spreadsheets, word-processing, and playing games.
Tessel - Tessel is a completely open source and community-driven IoT and robotics development platform. It encompases development boards, hardware module add-ons, and the software that runs on them.
UDOO - UDOO is a single-board computer with an integrated Arduino 2 compatible microcontroller, designed for computer science education, the world of Makers and the Internet of Things.
Software
Operating systems
Apache Mynewt - Apache Mynewt is a real-time, modular operating system for connected IoT devices that need to operate for long periods of time under power, memory, and storage constraints. The first connectivity stack offered is BLE 4.2.
ARM mbed - The ARM® mbed™ IoT Device Platform provides the operating system, cloud services, tools and developer ecosystem to make the creation and deployment of commercial, standards-based IoT solutions possible at scale.
Contiki - Contiki is an open source operating system for the Internet of Things. Contiki connects tiny low-cost, low-power microcontrollers to the Internet.
FreeRTOS - FreeRTOS is a popular real-time operating system kernel for embedded devices, that has been ported to 35 microcontrollers.
Google Brillo - Brillo extends the Android platform to all your connected devices, so they are easy to set up and work seamlessly with each other and your smartphone.
OpenWrt - OpenWrt is an operating system (in particular, an embedded operating system) based on the Linux kernel, primarily used on embedded devices to route network traffic. The main components are the Linux kernel, util-linux, uClibc or musl, and BusyBox. All components have been optimized for size, to be small enough for fitting into the limited storage and memory available in home routers.
Snappy Ubuntu - Snappy Ubuntu Core is a new rendition of Ubuntu with transactional updates. It provides a minimal server image with the same libraries as today’s Ubuntu, but applications are provided through a simpler mechanism.
NodeOS - NodeOS is an operating system entirely written in Javascript, and managed by npm on top of the Linux kernel.
Raspbian - Raspbian is a free operating system based on Debian optimized for the Raspberry Pi hardware.
RIOT - The friendly Operating System for the Internet of Things.
Tiny OS - TinyOS is an open source, BSD-licensed operating system designed for low-power wireless devices, such as those used in sensor networks, ubiquitous computing, personal area networks, smart buildings, and smart meters.
Windows 10 IoT Core - Windows 10 IoT is a family of Windows 10 editions targeted towards a wide range of intelligent devices, from small industrial gateways to larger more complex devices like point of sales terminals and ATMs.
Programming languages
This sections regroups every awesome programming language, whether it is compiled, interpreted or a DSL, related to embedded development.
C - A general-purpose, imperative computer programming language, supporting structured programming, lexical variable scope and recursion, while a static type system prevents many unintended operations.
C++ - A general-purpose programming language. It has imperative, object-oriented and generic programming features, while also providing facilities for low-level memory manipulation.
Groovy - Groovy is a powerful, optionally typed and dynamic language, with static-typing and static compilation capabilities, for the Java platform aimed at multiplying developers’ productivity thanks to a concise, familiar and easy to learn syntax. It is used by the SmartThings development environment to create smart applications.
Lua - Lua is a powerful, fast, lightweight, embeddable scripting language. Lua is dynamically typed, runs by interpreting bytecode for a register-based virtual machine, and has automatic memory management with incremental garbage collection, making it ideal for configuration, scripting, and rapid prototyping.
eLua - eLua stands for Embedded Lua and the project offers the full implementation of the Lua Programming Language to the embedded world, extending it with specific features for efficient and portable software embedded development.
ELFE - ELFE is a very simple and small programming language. While it is a general-purpose programming language, it is specifically tuned to facilitate the configuration and control of swarms of small devices such as sensors or actuators.
Frameworks
AllJoyn - AllJoyn is an open source software framework that makes it easy for devices and apps to discover and communicate with each other.
Apple HomeKit - HomeKit is a framework for communicating with and controlling connected accessories in a user’s home.
Blynk - Blynk is a platform for creating iOS and Android apps for connected things. You can easily build graphic interfaces for all your projects by simply dragging and dropping widgets (right on the smartphone). Supports Ethernet, WiFi, Bluetooth, GSM/GPRS, USB/Serial connections with wide range of prototyping platforms from Arduino, Raspberry, ARM mbed, Particle, RedBear, etc.
Countly IoT Analytics - Countly is a general purpose analytics platform for mobile and IoT devices, available as open source.
Eclipse Smarthome - The Eclipse SmartHome framework is designed to run on embedded devices, such as a Raspberry Pi, a BeagleBone Black or an Intel Edison. It requires a Java 7 compliant JVM and an OSGi (4.2+) framework, such as Eclipse Equinox.
Freedomotic - Freedomotic is an open source, flexible, secure Internet of Things (IoT) development framework, useful to build and manage modern smart spaces. It is targeted to private individuals (home automation) as well as business users (smart retail environments, ambient aware marketing, monitoring and analytics, etc). Written in Java, it can interact with well known standard building automation protocols as well as with "do it yourself" solutions.
Iotivity - IoTivity is an open source software framework enabling seamless device-to-device connectivity to address the emerging needs of the Internet of Things.
Kura - Kura aims at offering a Java/OSGi-based container for M2M applications running in service gateways. Kura provides or, when available, aggregates open source implementations for the most common services needed by M2M applications.
Lelylan - Lelylan is an IoT cloud platform based on a lightweight microservices architecture. The Lelylan platform is both hardware-agnostic and platform-agnostic. This means that you can connect any hardware, from the ESP8266 to the most professional embedded hardware solution and everything in between - and it can run on any public cloud, your own private datacenter, or even in a hybrid environment, whether virtualized or bare metal.
Mihini - The main goal of Mihini is to deliver an embedded runtime running on top of Linux, that exposes high-level API for building M2M applications. Mihini aims at enabling easy and portable development, by facilitating access to the I/Os of an M2M system, providing a communication layer, etc.
OpenHAB - The openHAB runtime is a set of OSGi bundles deployed on an OSGi framework (Equinox). It is therefore a pure Java solution and needs a JVM to run. Being based on OSGi, it provides a highly modular architecture, which even allows adding and removing functionality during runtime without stopping the service.
Gobot - Gobot is a framework for robotics, physical computing, and the Internet of Things, written in the Go programming language.
Home Assistant - Home Assistant is a home automation platform running on Python 3. The goal of Home Assistant is to be able to track and control all devices at home and offer a platform for automating control.
Lightweight MQTT Machine Network - LWMQN is an open source project that follows part of OMA LWM2M v1.0 specification and uses the IP-base Smart Object model to meet the minimum requirements of machine network management. It provides both server-side and machine-side libraries to make full-stack IoT development possible with JavaScript and Node.js. See also: IPSO Alliance Technical Archive.
Thingsboard IoT Gateway - Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with Thingsboard IoT Platform using OPC-UA and MQTT protocols.
Pimatic - Pimatic is a home automation framework that runs on node.js. It provides a common extensible platform for home control and automation tasks.
IOTA - Open-source distributed ledger protocol for IoT. Uses a directed acyclic graph (DAG) instead of a blockchain.
Middlewares
IFTTT - IFTTT is a web-based service that allows users to create chains of simple conditional statements, called "recipes", which are triggered based on changes to other web services such as Gmail, Facebook, Instagram, and Pinterest. IFTTT is an abbreviation of "If This Then That" (pronounced like "gift" without the "g").
Huginn - Huginn is a system for building agents that perform automated tasks for you online.
Kaa - An open-source middleware platform for rapid creation of IoT solutions.
Losant - Losant is an easy-to-use and powerful developer platform designed to help you quickly and securely build complex connected solutions. Losant uses open communication standards like REST and MQTT to provide connectivity from one to millions of devices. Losant provides powerful data collection, aggregation, and visualization features to help understand and quantify vast amounts of sensor data. Losant's drag-and-drop workflow editor allows you to trigger actions, notifications, and machine-to-machine communication without programming.
DreamFactory - DreamFactory is a free open source REST API Platform for mobile, web and IoT Applications.
I1820 - I1820 is a free open source platform which provides discovery, data collection and configuration services based on MQTT. I1820 implements a REST API for controlling the things and it stores all collected data in a Time-Series database named InfluxDB.
IOStash - IOStash is a high performance IoT platform that is free for DIY developers and non profit applications. It has multiple connectivity options and enables easy development of M2M or M2A applications. IOStash offers Nodejs and Android libraries for easy application creation.
Thingsboard - An open-source IoT platform. Device management, data collection, processing and visualization for your IoT solution.
Libraries and Tools
Cylon.js - Cylon.js is a JavaScript framework for robotics, physical computing, and the Internet of Things. It makes it incredibly easy to command robots and devices.
Luvit - Luvit implements the same APIs as Node.js, but in Lua ! While this framework is not directly involved with IoT development, it is still a great way to rapidly build powerful, yet memory efficient, embedded web applications.
Johnny-Five - Johnny-Five is the original JavaScript Robotics programming framework. Released by Bocoup in 2012, Johnny-Five is maintained by a community of passionate software developers and hardware engineers.
Pi4J - Pi4j is intended to provide a friendly object-oriented I/O API and implementation libraries for Java Programmers to access the full I/O capabilities of the Raspberry Pi platform.
WiringPi - WiringPi is a GPIO access library written in C for the BCM2835 used in the Raspberry Pi.
Node-RED - A visual tool for wiring the Internet of Things.
SmartObject - A Smart Object Class that helps you with creating IPSO Smart Objects in your JavaScript applications. See also: IPSO Alliance Technical Archive.
Miscellaneous
Amazon Dash - Amazon Dash Button is a Wi-Fi connected device that reorders your favorite item with the press of a button.
Freeboard - A real-time interactive dashboard and visualization creator implementing an intuitive drag & drop interface.
Nebula - A docker orchestrator designed to manage IoT devices
Protocols and Networks
Physical layer
- 802.15.4 (IEEE)
IEEE 802.15.4 is a standard which specifies the physical layer and media access control for low-rate wireless personal area networks (LR-WPANs). It is maintained by the IEEE 802.15 working group, which has defined it in 2003. It is the basis for the ZigBee, ISA100.11a, WirelessHART, and MiWi specifications, each of which further extends the standard by developing the upper layers which are not defined in IEEE 802.15.4. Alternatively, it can be used with 6LoWPAN and standard Internet protocols to build a wireless embedded Internet. - Wikipedia
IEEE standard 802.15.4 intends to offer the fundamental lower network layers of a type of wireless personal area network (WPAN) which focuses on low-cost, low-speed ubiquitous communication between devices. It can be contrasted with other approaches, such as Wi-Fi, which offer more bandwidth and require more power. The emphasis is on very low cost communication of nearby devices with little to no underlying infrastructure, intending to exploit this to lower power consumption even more.
- Bluetooth (Bluetooth Special Interest Group)
Bluetooth is a wireless technology standard for exchanging data over short distances (using short-wavelength UHF radio waves in the ISM band from 2.4 to 2.485 GHz) from fixed and mobile devices, and building personal area networks (PANs). Invented by telecom vendor Ericsson in 1994, it was originally conceived as a wireless alternative to RS-232 data cables. It can connect several devices, overcoming problems of synchronization. - Wikipedia
Bluetooth is managed by the Bluetooth Special Interest Group (SIG), which has more than 25,000 member companies in the areas of telecommunication, computing, networking, and consumer electronics.
- Bluetooth Low Energy (Bluetooth Special Interest Group)
Bluetooth low energy (Bluetooth LE, BLE, marketed as Bluetooth Smart) is a wireless personal area network technology designed and marketed by the Bluetooth Special Interest Group aimed at novel applications in the healthcare, fitness, beacons, security, and home entertainment industries. - Wikipedia
Compared to Classic Bluetooth, Bluetooth Smart is intended to provide considerably reduced power consumption and cost while maintaining a similar communication range. The Bluetooth SIG predicts that by 2018 more than 90 percent of Bluetooth-enabled smartphones will support Bluetooth Smart.
EC-GSM-IoT (EC-GSM-IoT Group)
Extended coverage GSM IoT (EC-GSM-IoT) is a standard-based Low Power Wide Area technology. It is based on eGPRS and designed as a high capacity, long range, low energy and low complexity cellular system for IoT communications.
The EC-GSM-IOT network trials have begun, with the first commercial launches planned for 2017. Supported by all major mobile equipment, chip set and module manufacturers, EC-GSM-IoT networks will co-exist with 2G, 3G, and 4G mobile networks. It will also benefit from all the security and privacy mobile network features, such as support for user identity confidentiality, entity authentication, confidentiality, data integrity, and mobile equipment identification.
- LoRaWAN (LoRa Alliance)
A LoRaWAN wide area network allows low bit rate communication from and to connected objects, thus participating to Internet of Things, machine-to-machine M2M, and smart city. - Wikipedia
This technology is standardized by the LoRa Alliance. It was initially developed by Cycleo, which was acquired by Semtech in 2012. LoRaWAN is an acronym for Long Range Wide-area network.
NB-IoT (3GPP)
NarrowBand IoT (NB-IoT) is a Low Power Wide Area Network (LPWAN) radio technology standard that has been developed to enable a wide range of devices and services to be connected using cellular telecommunications bands. - Wikipedia
NB-IoT is a narrowband radio technology designed for the Internet of Things (IoT), and is one of a range of Mobile IoT (MIoT) technologies standardized by the 3rd Generation Partnership Project (3GPP).
- Sigfox (Sigfox)
Sigfox is a French firm that builds wireless networks to connect low-energy objects such as electricity meters, smart watches, and washing machines, which need to be continuously on and emitting small amounts of data. Its infrastructure is intended to be a contribution to what is known as the Internet of Things (IoT). - Wikipedia
SIGFOX describes itself as "the first and only company providing global cellular connectivity for the Internet of Things." Its infrastructure is "completely independent of existing networks, such as telecommunications networks." SIGFOX seeks to provide the means for the "deployment of billions of objects and thousands of new uses" with the long-term goal of "having petabytes of data produced by everyday objects".
- Wi-Fi (Wi-Fi Alliance)
Wi-Fi (or WiFi) is a local area wireless computer networking technology that allows electronic devices to network, mainly using the 2.4 gigahertz (12 cm) UHF and 5 gigahertz (6 cm) SHF ISM radio bands. - Wikipedia
The Wi-Fi Alliance defines Wi-Fi as any "wireless local area network" (WLAN) product based on the Institute of Electrical and Electronics Engineers' (IEEE) 802.11 standards.[1] However, the term "Wi-Fi" is used in general English as a synonym for "WLAN" since most modern WLANs are based on these standards. "Wi-Fi" is a trademark of the Wi-Fi Alliance. The "Wi-Fi Certified" trademark can only be used by Wi-Fi products that successfully complete Wi-Fi Alliance interoperability certification testing.
Network / Transport layer
- 6LowPan (IETF)
6LoWPAN is an acronym of IPv6 over Low power Wireless Personal Area Networks. 6LoWPAN is the name of a concluded working group in the Internet area of the IETF. - Wikipedia
The 6LoWPAN concept originated from the idea that "the Internet Protocol could and should be applied even to the smallest devices,"and that low-power devices with limited processing capabilities should be able to participate in the Internet of Things. The 6LoWPAN group has defined encapsulation and header compression mechanisms that allow IPv6 packets to be sent and received over IEEE 802.15.4 based networks. IPv4 and IPv6 are the work horses for data delivery for local-area networks, metropolitan area networks, and wide-area networks such as the Internet. Likewise, IEEE 802.15.4 devices provide sensing communication-ability in the wireless domain. The inherent natures of the two networks though, are different.
- Thread (Thread Group)
Thread is an IPv6 based protocol for "smart" household devices to communicate on a network.
In July 2014 Google Inc's Nest Labs announced a working group with the companies Samsung, ARM Holdings, Freescale, Silicon Labs, Big Ass Fans and the lock company Yale in an attempt to have Thread become the industry standard by providing Thread certification for products. Other protocols currently in use include ZigBee and Bluetooth Smart. Thread uses 6LoWPAN, which in turn uses the IEEE 802.15.4 wireless protocol with mesh communication, as does ZigBee and other systems. Thread however is IP-addressable, with cloud access and AES encryption. It supports over 250 devices on a network.
- ZigBee (ZigBee Alliance)
ZigBee is a IEEE 802.15.4-based specification for a suite of high-level communication protocols used to create personal area networks with small, low-power digital radios. - Wikipedia
The technology defined by the ZigBee specification is intended to be simpler and less expensive than other wireless personal area networks (WPANs), such as Bluetooth or Wi-Fi. Applications include wireless light switches, electrical meters with in-home-displays, traffic management systems, and other consumer and industrial equipment that requires short-range low-rate wireless data transfer.
- Z-Wave (Z-Wave Alliance)
Z-Wave is a wireless communications specification designed to allow devices in the home (lighting, access controls, entertainment systems and household appliances, for example) to communicate with one another for the purposes of home automation. - Wikipedia
Z-Wave technology minimizes power consumption so that it is suitable for battery-operated devices. Z-Wave is designed to provide, reliable, low-latency transmission of small data packets at data rates up to 100kbit/s, unlike Wi-Fi and other IEEE 802.11-based wireless LAN systems that are designed primarily for high data rates. Z-Wave operates in the sub-gigahertz frequency range, around 900 MHz.
Application layer
CoAP (IETF)
Constrained Application Protocol (CoAP) is a software protocol intended to be used in very simple electronics devices that allows them to communicate interactively over the Internet. - Wikipedia
CoAP is particularly targeted for small low power sensors, switches, valves and similar components that need to be controlled or supervised remotely, through standard Internet networks. CoAP is an application layer protocol that is intended for use in resource-constrained internet devices, such as WSN nodes.
DTLS (IETF)
The Datagram Transport Layer Security (DTLS) communications protocol provides communications security for datagram protocols. - Wikipedia
DTLS allows datagram-based applications to communicate in a way that is designed[by whom?] to prevent eavesdropping, tampering, or message forgery. The DTLS protocol is based on the stream-oriented Transport Layer Security (TLS) protocol and is intended to provide similar security guarantees.
- Eddystone (Google)
Eddystone is a beacon technology profile released by Google in July 2015. The open source, cross-platform software gives users location and proximity data via Bluetooth low-energy beacon format. - Wikipedia
Though similar to the iBeacon released by Apple in 2013, Eddystone works on both Android and iOS, whereas iBeacon is limited to iOS platforms. A practical application of both softwares is that business owners can target potential customers based on the location of their smartphones in real time.
- HTTP (IETF)
The Hypertext Transfer Protocol (HTTP) is an application protocol for distributed, collaborative, hypermedia information systems. HTTP is the foundation of data communication for the World Wide Web. - Wikipedia
The standards development of HTTP was coordinated by the Internet Engineering Task Force (IETF) and the World Wide Web Consortium (W3C), culminating in the publication of a series of Requests for Comments (RFCs). The first definition of HTTP/1.1, the version of HTTP in common use, occurred in RFC 2068 in 1997, although this was obsoleted by RFC 2616 in 1999.
- iBeacon (Apple)
iBeacon is a protocol standardized by Apple and introduced at the Apple Worldwide Developers Conference in 2013. - Wikipedia
iBeacon uses Bluetooth low energy proximity sensing to transmit a universally unique identifier picked up by a compatible app or operating system. The identifier can be used to determine the device's physical location, track customers, or trigger a location-based action on the device such as a check-in on social media or a push notification.
- MQTT (IBM)
MQTT (formerly MQ Telemetry Transport) is a publish-subscribe based "light weight" messaging protocol for use on top of the TCP/IP protocol. It is designed for connections with remote locations where a "small code footprint" is required or the network bandwidth is limited. - Wikipedia
The publish-subscribe messaging pattern requires a message broker. The broker is responsible for distributing messages to interested clients based on the topic of a message. Andy Stanford-Clark and Arlen Nipper of Cirrus Link Solutions authored the first version of the protocol in 1999.
- STOMP
Simple (or Streaming) Text Oriented Message Protocol (STOMP), formerly known as TTMP, is a simple text-based protocol, designed for working with message-oriented middleware (MOM). - Wikipedia
STOMP provides an interoperable wire format that allows STOMP clients to talk with any message broker supporting the protocol. It is thus language-agnostic, meaning a broker developed for one programming language or platform can receive communications from client software developed in another language.
- Websocket
WebSocket is a protocol providing full-duplex communication channels over a single TCP connection. - Wikipedia
WebSocket is designed to be implemented in web browsers and web servers, but it can be used by any client or server application. The WebSocket Protocol is an independent TCP-based protocol. The WebSocket protocol makes more interaction between a browser and a website possible, facilitating live content and the creation of real-time games. This is made possible by providing a standardized way for the server to send content to the browser without being solicited by the client, and allowing for messages to be passed back and forth while keeping the connection open.
- XMPP (IETF)
Extensible Messaging and Presence Protocol (XMPP) is a communications protocol for message-oriented middleware based on XML (Extensible Markup Language). - Wikipedia
It enables the near-real-time exchange of structured yet extensible data between any two or more network entities. Designed to be extensible, the protocol has also been used for publish-subscribe systems, signalling for VoIP, video, file transfer, gaming, Internet of Things (IoT) applications such as the smart grid, and social networking services.
Technologies
This sections regroups a curated list of awesome technologies that are closely related to the IoT world.
- NFC
Near field communication (NFC) is the set of protocols that enable electronic devices to establish radio communication with each other by touching the devices together, or bringing them into proximity to a distance of typically 10cm or less. - Wikipedia
- OPCUA
OPC-UA is a not only a protocol for industrial automation but also a technology that allows semantic description and object modelling of industrial environment. Wikipedia
Standards and Alliances
Standards
ETSI M2M - The ETSI Technical Committee is developing standards for Machine to Machine Communications.
OneM2M - The purpose and goal of oneM2M is to develop technical specifications which address the need for a common M2M Service Layer that can be readily embedded within various hardware and software, and relied upon to connect the myriad of devices in the field with M2M application servers worldwide.
OPCUA - OPC Unified Architecture (OPC UA) is an industrial M2M communication protocol for interoperability developed by the OPC Foundation.
OCF - OCF, The Open Connectivity Foundation, develop standards and certification for devices involved in the Internet of Things (IoT) based around Constrained Application Protocol (CoAP).
Alliances
AIOTI - The Internet of Things Innovation (AIOTI) aims to strengthen links and build new relationships between the different IoT players (industries, SMEs, startups) and sectors.
AllSeen Alliance - The AllSeen Alliance is a nonprofit consortium dedicated to enabling and driving the widespread adoption of products, systems and services that support the Internet of Everything with an open, universal development framework supported by a vibrant ecosystem and thriving technical community.
Bluetooth Special Interest Group - The Bluetooth Special Interest Group (SIG) is the body that oversees the development of Bluetooth standards and the licensing of the Bluetooth technologies and trademarks to manufacturers.
IPSO Alliance - The IPSO Alliance provides a foundation for industry growth by fostering awareness, providing education, promoting the industry, generating research, and creating a better understanding of IP and its role in the Internet of Things.
LoRa Alliance - The LoRa Alliance is an open, non-profit association of members that believes the internet of things era is now. It was initiated by industry leaders with a mission to standardize Low Power Wide Area Networks (LPWAN) being deployed around the world to enable Internet of Things (IoT), machine-to-machine (M2M), and smart city, and industrial applications.
OPC Foundation - The mission of the OPC Foundation is to manage a global organization in which users, vendors and consortia collaborate to create data transfer standards for multi-vendor, multi-platform, secure and reliable interoperability in industrial automation. To support this mission, the OPC Foundation creates and maintains specifications, ensures compliance with OPC specifications via certification testing and collaborates with industry-leading standards organizations.
Thread Group - The Thread Group, composed of members from Nest, Samsung, ARM, Freescale, Silicon Labs, Big Ass Fans and Yale, drives the development of the Thread network protocol.
Wi-Fi Alliance - Wi-Fi Alliance® is a worldwide network of companies composed of several companies forming a global non-profit association with the goal of driving the best user experience with a new wireless networking technology – regardless of brand.
Zigbee Alliance - The ZigBee Alliance is an open, non-profit association of approximately 450 members driving development of innovative, reliable and easy-to-use ZigBee standards.
Z-Wave Alliance - Established in 2005, the Z-Wave Alliance is comprised of industry leaders throughout the globe that are dedicated to the development and extension of Z-Wave as the key enabling technology for 'smart' home and business applications.
Resources
Books
Building the Web of Things: with examples in Node.js and Raspberry Pi (2016) by Dominique Guinard and Vlad Trifa [5.0]
A hands-on guide that will teach how to design and implement scalable, flexible, and open IoT solutions using web technologies. This book focuses on providing the right balance of theory, code samples, and practical examples to enable you to successfully connect all sorts of devices to the web and to expose their services and data over REST APIs. The book covers a number of web technologies for your IoT toolbox: GPIO, Raspberry Pi, Embedded Systems, REST and HTTP, WS, MQTT, CoAP, JSON-LD, Social Networks, Node-RED, IFTTT, etc.
Abusing the Internet of Things: Blackouts, Freakouts, and Stakeouts (2015) by Nitesh Dhanjani [5.0]
future with billions of connected "things" includes monumental security concerns. This practical book explores how malicious attackers can abuse popular IoT-based devices, including wireless LED lightbulbs, electronic door locks, baby monitors, smart TVs, and connected cars.
Using the Web to Build the IoT (2016) *by Dominique Guinard
and Vlad Trifa
Using the Web to Build the IoT is a free book built as a collection of six hand-picked chapters that introduce the key technologies and concepts for building the application layer of the IoT. Dom Guinard and Vlad Trifa, selected these specific topics to give you an overview of the Web of Things architecture, along with techniques for data ingestion, searching, security, and visualization.
Building Wireless Sensor Networks: with ZigBee, XBee, Arduino, and Processing (2011) by Robert Faludi [4.5]
Get ready to create distributed sensor systems and intelligent interactive devices using the ZigBee wireless networking protocol and Series 2 XBee radios. By the time you're halfway through this fast-paced, hands-on guide, you'll have built a series of useful projects, including a complete ZigBee wireless network that delivers remotely sensed data.
Designing the Internet of Things (2013) by Adrian McEwen and Hakim Cassimally [4.0]
Whether it's called physical computing, ubiquitous computing, or the Internet of Things, it's a hot topic in technology: how to channel your inner Steve Jobs and successfully combine hardware, embedded software, web services, electronics, and cool design to create cutting-edge devices that are fun, interactive, and practical. If you'd like to create the next must-have product, this unique book is the perfect place to start.
Getting Started with Bluetooth Low Energy: Tools and Techniques for Low-Power Networking (2014) by Kevin Townsend, Carles Cufí, Akiba and Robert Davidson [4.5]
This book provides a solid, high-level overview of how devices use Ble to communicate with each other. You'll learn useful low-cost tools for developing and testing Ble-enabled mobile apps and embedded firmware and get examples using various development platforms including iOs and Android for app developers and embedded platforms for product designers and hardware engineers.
Smart Things: Ubiquitous Computing User Experience Design (2010) by Mike Kuniavsky [4.5]
Smart Things presents a problem-solving approach to addressing designers' needs and concentrates on process, rather than technological detail, to keep from being quickly outdated. It pays close attention to the capabilities and limitations of the medium in question and discusses the tradeoffs and challenges of design in a commercial environment.
JavaScript on Things: Hardware for Web Developers (2018 - est.) by Lyza Danger Gardner [early access book]
JavaScript on Things is your first step into the exciting and downright entertaining world of programming for small electronics. If you know enough JavaScript to hack a website together, you'll be making things bleep, blink and spin faster than you can say "nodebot". This fully-illustrated, hands-on book shows you how to get going with platforms like Arduino, Tessel, and Raspberry Pi.
Articles
A Simple Explanation Of 'The Internet Of Things' (Forbes) - This article attemps to give an answer to what exactly is the “Internet of things” and what impact it is going to have on us.
IoT security. Is there an app for that ? - The Internet of Things World conference investigates IoT application development, security, and business models.
The IoT Testing Atlas - A testing methodology for managing the permutations of parameters while testing an IoT based product.
Papers
A Reference Architecture for the Internet of Things - This white paper introduces a Reference Architecture for the Internet of Things (IoT): this includes the devices as well as the server-side and cloud architecture required to interact with and manage the devices.
Developing solutions for the Internet of Things - Intel's vision in enabling secure and seamless solutions for the Internet of Things (IoT).
Evaluation of indoor positioning based on Bluetooth Smart technology - Master of Science Thesis in the Programme Computer Systems and Networks.
IoT: A Vision, Architectural Elements, and Future Directions - This paper presents a cloud centric vision for worldwide implementation of Internet of Things. The key enabling technologies and application domains that are likely to drive IoT research in the near future are discussed.
Realizing the Potential of the Internet of Things - A white paper from the Telecommunications Industry Association (TIA) written in the form of a set of recommendations to policy maker on leveraging and realizing the potential of the Internet of Things market.
The Internet of Things: Evolution or Revolution ? - This white paper compares the current Internet of Things market rise to other industrial revolutions, the challenges it introduces, as well as its consequences on our daily lives.
Arduino - Arduino is an open-source electronics platform based on easy-to-use hardware and software. It's intended for anyone making interactive projects.
BeagleBoard - The BeagleBoard is a low-power open-source hardware single-board computer produced by Texas Instruments in association with Digi-Key and Newark element14.
Dragonboard - The DragonBoard 410c, a product of Arrow Electronics, is the development board based on the mid-tier Qualcomm® Snapdragon™ 410E processor. It features advanced processing power, Wi-Fi, Bluetooth connectivity, and GPS, all packed into a board the size of a credit card.
HummingBoard - HummingBoard is a family of three Linux- and Android-ready, open source SBCs based on 1GHz Freescale i.MX6 SoCs, with a Pi-like 26-pin I/O connector.
Intel Galileo - The Intel® Galileo Gen 2 board is the first in a family of Arduino*-certified development and prototyping boards based on Intel® architecture and specifically designed for makers, students, educators, and DIY electronics enthusiasts.
Microduino - Microduino and mCookie bring powerful, small, stackable electronic hardware to makers, designers, engineers, students and curious tinkerers of all ages. Build open-source projects or create innovative new ones.
Node MCU (ESP 8266) - NodeMCU is an open source IoT platform. It uses the Lua scripting language. It is based on the eLua project, and built on the ESP8266 SDK 0.9.5.
OLinuXino - OLinuXino is an Open Source Software and Open Source Hardware low cost (EUR 30) Linux Industrial grade single board computer with GPIOs capable of operating from -25°C to +85°C.
Odroid - The ODROID means Open + Droid. It is a development platform for the hardware as well as the software.
Particle - A suite of hardware and software tools to help you prototype, scale, and manage your Internet of Things products.
Pinoccio - Pinoccio is a solution to add mesh networking capability and WiFi-Internet access to all yout IoT devices, and it is Arduino compatible.
Raspberry Pi - The Raspberry Pi is a low cost, credit-card sized computer that plugs into a computer monitor or TV, and uses a standard keyboard and mouse. It’s capable of doing everything you’d expect a desktop computer to do, from browsing the internet and playing high-definition video, to making spreadsheets, word-processing, and playing games.
Tessel - Tessel is a completely open source and community-driven IoT and robotics development platform. It encompases development boards, hardware module add-ons, and the software that runs on them.
UDOO - UDOO is a single-board computer with an integrated Arduino 2 compatible microcontroller, designed for computer science education, the world of Makers and the Internet of Things.
Software
Operating systems
Apache Mynewt - Apache Mynewt is a real-time, modular operating system for connected IoT devices that need to operate for long periods of time under power, memory, and storage constraints. The first connectivity stack offered is BLE 4.2.
ARM mbed - The ARM® mbed™ IoT Device Platform provides the operating system, cloud services, tools and developer ecosystem to make the creation and deployment of commercial, standards-based IoT solutions possible at scale.
Contiki - Contiki is an open source operating system for the Internet of Things. Contiki connects tiny low-cost, low-power microcontrollers to the Internet.
FreeRTOS - FreeRTOS is a popular real-time operating system kernel for embedded devices, that has been ported to 35 microcontrollers.
Google Brillo - Brillo extends the Android platform to all your connected devices, so they are easy to set up and work seamlessly with each other and your smartphone.
OpenWrt - OpenWrt is an operating system (in particular, an embedded operating system) based on the Linux kernel, primarily used on embedded devices to route network traffic. The main components are the Linux kernel, util-linux, uClibc or musl, and BusyBox. All components have been optimized for size, to be small enough for fitting into the limited storage and memory available in home routers.
Snappy Ubuntu - Snappy Ubuntu Core is a new rendition of Ubuntu with transactional updates. It provides a minimal server image with the same libraries as today’s Ubuntu, but applications are provided through a simpler mechanism.
NodeOS - NodeOS is an operating system entirely written in Javascript, and managed by npm on top of the Linux kernel.
Raspbian - Raspbian is a free operating system based on Debian optimized for the Raspberry Pi hardware.
RIOT - The friendly Operating System for the Internet of Things.
Tiny OS - TinyOS is an open source, BSD-licensed operating system designed for low-power wireless devices, such as those used in sensor networks, ubiquitous computing, personal area networks, smart buildings, and smart meters.
Windows 10 IoT Core - Windows 10 IoT is a family of Windows 10 editions targeted towards a wide range of intelligent devices, from small industrial gateways to larger more complex devices like point of sales terminals and ATMs.
Programming languages
This sections regroups every awesome programming language, whether it is compiled, interpreted or a DSL, related to embedded development.
C - A general-purpose, imperative computer programming language, supporting structured programming, lexical variable scope and recursion, while a static type system prevents many unintended operations.
C++ - A general-purpose programming language. It has imperative, object-oriented and generic programming features, while also providing facilities for low-level memory manipulation.
Groovy - Groovy is a powerful, optionally typed and dynamic language, with static-typing and static compilation capabilities, for the Java platform aimed at multiplying developers’ productivity thanks to a concise, familiar and easy to learn syntax. It is used by the SmartThings development environment to create smart applications.
Lua - Lua is a powerful, fast, lightweight, embeddable scripting language. Lua is dynamically typed, runs by interpreting bytecode for a register-based virtual machine, and has automatic memory management with incremental garbage collection, making it ideal for configuration, scripting, and rapid prototyping.
eLua - eLua stands for Embedded Lua and the project offers the full implementation of the Lua Programming Language to the embedded world, extending it with specific features for efficient and portable software embedded development.
ELFE - ELFE is a very simple and small programming language. While it is a general-purpose programming language, it is specifically tuned to facilitate the configuration and control of swarms of small devices such as sensors or actuators.
Frameworks
AllJoyn - AllJoyn is an open source software framework that makes it easy for devices and apps to discover and communicate with each other.
Apple HomeKit - HomeKit is a framework for communicating with and controlling connected accessories in a user’s home.
Blynk - Blynk is a platform for creating iOS and Android apps for connected things. You can easily build graphic interfaces for all your projects by simply dragging and dropping widgets (right on the smartphone). Supports Ethernet, WiFi, Bluetooth, GSM/GPRS, USB/Serial connections with wide range of prototyping platforms from Arduino, Raspberry, ARM mbed, Particle, RedBear, etc.
Countly IoT Analytics - Countly is a general purpose analytics platform for mobile and IoT devices, available as open source.
Eclipse Smarthome - The Eclipse SmartHome framework is designed to run on embedded devices, such as a Raspberry Pi, a BeagleBone Black or an Intel Edison. It requires a Java 7 compliant JVM and an OSGi (4.2+) framework, such as Eclipse Equinox.
Freedomotic - Freedomotic is an open source, flexible, secure Internet of Things (IoT) development framework, useful to build and manage modern smart spaces. It is targeted to private individuals (home automation) as well as business users (smart retail environments, ambient aware marketing, monitoring and analytics, etc). Written in Java, it can interact with well known standard building automation protocols as well as with "do it yourself" solutions.
Iotivity - IoTivity is an open source software framework enabling seamless device-to-device connectivity to address the emerging needs of the Internet of Things.
Kura - Kura aims at offering a Java/OSGi-based container for M2M applications running in service gateways. Kura provides or, when available, aggregates open source implementations for the most common services needed by M2M applications.
Lelylan - Lelylan is an IoT cloud platform based on a lightweight microservices architecture. The Lelylan platform is both hardware-agnostic and platform-agnostic. This means that you can connect any hardware, from the ESP8266 to the most professional embedded hardware solution and everything in between - and it can run on any public cloud, your own private datacenter, or even in a hybrid environment, whether virtualized or bare metal.
Mihini - The main goal of Mihini is to deliver an embedded runtime running on top of Linux, that exposes high-level API for building M2M applications. Mihini aims at enabling easy and portable development, by facilitating access to the I/Os of an M2M system, providing a communication layer, etc.
OpenHAB - The openHAB runtime is a set of OSGi bundles deployed on an OSGi framework (Equinox). It is therefore a pure Java solution and needs a JVM to run. Being based on OSGi, it provides a highly modular architecture, which even allows adding and removing functionality during runtime without stopping the service.
Gobot - Gobot is a framework for robotics, physical computing, and the Internet of Things, written in the Go programming language.
Home Assistant - Home Assistant is a home automation platform running on Python 3. The goal of Home Assistant is to be able to track and control all devices at home and offer a platform for automating control.
Lightweight MQTT Machine Network - LWMQN is an open source project that follows part of OMA LWM2M v1.0 specification and uses the IP-base Smart Object model to meet the minimum requirements of machine network management. It provides both server-side and machine-side libraries to make full-stack IoT development possible with JavaScript and Node.js. See also: IPSO Alliance Technical Archive.
Thingsboard IoT Gateway - Open-source IoT Gateway - integrates devices connected to legacy and third-party systems with Thingsboard IoT Platform using OPC-UA and MQTT protocols.
Pimatic - Pimatic is a home automation framework that runs on node.js. It provides a common extensible platform for home control and automation tasks.
IOTA - Open-source distributed ledger protocol for IoT. Uses a directed acyclic graph (DAG) instead of a blockchain.
Middlewares
IFTTT - IFTTT is a web-based service that allows users to create chains of simple conditional statements, called "recipes", which are triggered based on changes to other web services such as Gmail, Facebook, Instagram, and Pinterest. IFTTT is an abbreviation of "If This Then That" (pronounced like "gift" without the "g").
Huginn - Huginn is a system for building agents that perform automated tasks for you online.
Kaa - An open-source middleware platform for rapid creation of IoT solutions.
Losant - Losant is an easy-to-use and powerful developer platform designed to help you quickly and securely build complex connected solutions. Losant uses open communication standards like REST and MQTT to provide connectivity from one to millions of devices. Losant provides powerful data collection, aggregation, and visualization features to help understand and quantify vast amounts of sensor data. Losant's drag-and-drop workflow editor allows you to trigger actions, notifications, and machine-to-machine communication without programming.
DreamFactory - DreamFactory is a free open source REST API Platform for mobile, web and IoT Applications.
I1820 - I1820 is a free open source platform which provides discovery, data collection and configuration services based on MQTT. I1820 implements a REST API for controlling the things and it stores all collected data in a Time-Series database named InfluxDB.
IOStash - IOStash is a high performance IoT platform that is free for DIY developers and non profit applications. It has multiple connectivity options and enables easy development of M2M or M2A applications. IOStash offers Nodejs and Android libraries for easy application creation.
Thingsboard - An open-source IoT platform. Device management, data collection, processing and visualization for your IoT solution.
Libraries and Tools
Cylon.js - Cylon.js is a JavaScript framework for robotics, physical computing, and the Internet of Things. It makes it incredibly easy to command robots and devices.
Luvit - Luvit implements the same APIs as Node.js, but in Lua ! While this framework is not directly involved with IoT development, it is still a great way to rapidly build powerful, yet memory efficient, embedded web applications.
Johnny-Five - Johnny-Five is the original JavaScript Robotics programming framework. Released by Bocoup in 2012, Johnny-Five is maintained by a community of passionate software developers and hardware engineers.
Pi4J - Pi4j is intended to provide a friendly object-oriented I/O API and implementation libraries for Java Programmers to access the full I/O capabilities of the Raspberry Pi platform.
WiringPi - WiringPi is a GPIO access library written in C for the BCM2835 used in the Raspberry Pi.
Node-RED - A visual tool for wiring the Internet of Things.
SmartObject - A Smart Object Class that helps you with creating IPSO Smart Objects in your JavaScript applications. See also: IPSO Alliance Technical Archive.
Miscellaneous
Amazon Dash - Amazon Dash Button is a Wi-Fi connected device that reorders your favorite item with the press of a button.
Freeboard - A real-time interactive dashboard and visualization creator implementing an intuitive drag & drop interface.
Nebula - A docker orchestrator designed to manage IoT devices
Protocols and Networks
Physical layer
- 802.15.4 (IEEE)
IEEE 802.15.4 is a standard which specifies the physical layer and media access control for low-rate wireless personal area networks (LR-WPANs). It is maintained by the IEEE 802.15 working group, which has defined it in 2003. It is the basis for the ZigBee, ISA100.11a, WirelessHART, and MiWi specifications, each of which further extends the standard by developing the upper layers which are not defined in IEEE 802.15.4. Alternatively, it can be used with 6LoWPAN and standard Internet protocols to build a wireless embedded Internet. - Wikipedia
IEEE standard 802.15.4 intends to offer the fundamental lower network layers of a type of wireless personal area network (WPAN) which focuses on low-cost, low-speed ubiquitous communication between devices. It can be contrasted with other approaches, such as Wi-Fi, which offer more bandwidth and require more power. The emphasis is on very low cost communication of nearby devices with little to no underlying infrastructure, intending to exploit this to lower power consumption even more.
- Bluetooth (Bluetooth Special Interest Group)
Bluetooth is a wireless technology standard for exchanging data over short distances (using short-wavelength UHF radio waves in the ISM band from 2.4 to 2.485 GHz) from fixed and mobile devices, and building personal area networks (PANs). Invented by telecom vendor Ericsson in 1994, it was originally conceived as a wireless alternative to RS-232 data cables. It can connect several devices, overcoming problems of synchronization. - Wikipedia
Bluetooth is managed by the Bluetooth Special Interest Group (SIG), which has more than 25,000 member companies in the areas of telecommunication, computing, networking, and consumer electronics.
- Bluetooth Low Energy (Bluetooth Special Interest Group)
Bluetooth low energy (Bluetooth LE, BLE, marketed as Bluetooth Smart) is a wireless personal area network technology designed and marketed by the Bluetooth Special Interest Group aimed at novel applications in the healthcare, fitness, beacons, security, and home entertainment industries. - Wikipedia
Compared to Classic Bluetooth, Bluetooth Smart is intended to provide considerably reduced power consumption and cost while maintaining a similar communication range. The Bluetooth SIG predicts that by 2018 more than 90 percent of Bluetooth-enabled smartphones will support Bluetooth Smart.
EC-GSM-IoT (EC-GSM-IoT Group)
Extended coverage GSM IoT (EC-GSM-IoT) is a standard-based Low Power Wide Area technology. It is based on eGPRS and designed as a high capacity, long range, low energy and low complexity cellular system for IoT communications.
The EC-GSM-IOT network trials have begun, with the first commercial launches planned for 2017. Supported by all major mobile equipment, chip set and module manufacturers, EC-GSM-IoT networks will co-exist with 2G, 3G, and 4G mobile networks. It will also benefit from all the security and privacy mobile network features, such as support for user identity confidentiality, entity authentication, confidentiality, data integrity, and mobile equipment identification.
- LoRaWAN (LoRa Alliance)
A LoRaWAN wide area network allows low bit rate communication from and to connected objects, thus participating to Internet of Things, machine-to-machine M2M, and smart city. - Wikipedia
This technology is standardized by the LoRa Alliance. It was initially developed by Cycleo, which was acquired by Semtech in 2012. LoRaWAN is an acronym for Long Range Wide-area network.
NB-IoT (3GPP)
NarrowBand IoT (NB-IoT) is a Low Power Wide Area Network (LPWAN) radio technology standard that has been developed to enable a wide range of devices and services to be connected using cellular telecommunications bands. - Wikipedia
NB-IoT is a narrowband radio technology designed for the Internet of Things (IoT), and is one of a range of Mobile IoT (MIoT) technologies standardized by the 3rd Generation Partnership Project (3GPP).
- Sigfox (Sigfox)
Sigfox is a French firm that builds wireless networks to connect low-energy objects such as electricity meters, smart watches, and washing machines, which need to be continuously on and emitting small amounts of data. Its infrastructure is intended to be a contribution to what is known as the Internet of Things (IoT). - Wikipedia
SIGFOX describes itself as "the first and only company providing global cellular connectivity for the Internet of Things." Its infrastructure is "completely independent of existing networks, such as telecommunications networks." SIGFOX seeks to provide the means for the "deployment of billions of objects and thousands of new uses" with the long-term goal of "having petabytes of data produced by everyday objects".
- Wi-Fi (Wi-Fi Alliance)
Wi-Fi (or WiFi) is a local area wireless computer networking technology that allows electronic devices to network, mainly using the 2.4 gigahertz (12 cm) UHF and 5 gigahertz (6 cm) SHF ISM radio bands. - Wikipedia
The Wi-Fi Alliance defines Wi-Fi as any "wireless local area network" (WLAN) product based on the Institute of Electrical and Electronics Engineers' (IEEE) 802.11 standards.[1] However, the term "Wi-Fi" is used in general English as a synonym for "WLAN" since most modern WLANs are based on these standards. "Wi-Fi" is a trademark of the Wi-Fi Alliance. The "Wi-Fi Certified" trademark can only be used by Wi-Fi products that successfully complete Wi-Fi Alliance interoperability certification testing.
Network / Transport layer
- 6LowPan (IETF)
6LoWPAN is an acronym of IPv6 over Low power Wireless Personal Area Networks. 6LoWPAN is the name of a concluded working group in the Internet area of the IETF. - Wikipedia
The 6LoWPAN concept originated from the idea that "the Internet Protocol could and should be applied even to the smallest devices,"and that low-power devices with limited processing capabilities should be able to participate in the Internet of Things. The 6LoWPAN group has defined encapsulation and header compression mechanisms that allow IPv6 packets to be sent and received over IEEE 802.15.4 based networks. IPv4 and IPv6 are the work horses for data delivery for local-area networks, metropolitan area networks, and wide-area networks such as the Internet. Likewise, IEEE 802.15.4 devices provide sensing communication-ability in the wireless domain. The inherent natures of the two networks though, are different.
- Thread (Thread Group)
Thread is an IPv6 based protocol for "smart" household devices to communicate on a network.
In July 2014 Google Inc's Nest Labs announced a working group with the companies Samsung, ARM Holdings, Freescale, Silicon Labs, Big Ass Fans and the lock company Yale in an attempt to have Thread become the industry standard by providing Thread certification for products. Other protocols currently in use include ZigBee and Bluetooth Smart. Thread uses 6LoWPAN, which in turn uses the IEEE 802.15.4 wireless protocol with mesh communication, as does ZigBee and other systems. Thread however is IP-addressable, with cloud access and AES encryption. It supports over 250 devices on a network.
- ZigBee (ZigBee Alliance)
ZigBee is a IEEE 802.15.4-based specification for a suite of high-level communication protocols used to create personal area networks with small, low-power digital radios. - Wikipedia
The technology defined by the ZigBee specification is intended to be simpler and less expensive than other wireless personal area networks (WPANs), such as Bluetooth or Wi-Fi. Applications include wireless light switches, electrical meters with in-home-displays, traffic management systems, and other consumer and industrial equipment that requires short-range low-rate wireless data transfer.
- Z-Wave (Z-Wave Alliance)
Z-Wave is a wireless communications specification designed to allow devices in the home (lighting, access controls, entertainment systems and household appliances, for example) to communicate with one another for the purposes of home automation. - Wikipedia
Z-Wave technology minimizes power consumption so that it is suitable for battery-operated devices. Z-Wave is designed to provide, reliable, low-latency transmission of small data packets at data rates up to 100kbit/s, unlike Wi-Fi and other IEEE 802.11-based wireless LAN systems that are designed primarily for high data rates. Z-Wave operates in the sub-gigahertz frequency range, around 900 MHz.
Application layer
CoAP (IETF)
Constrained Application Protocol (CoAP) is a software protocol intended to be used in very simple electronics devices that allows them to communicate interactively over the Internet. - Wikipedia
CoAP is particularly targeted for small low power sensors, switches, valves and similar components that need to be controlled or supervised remotely, through standard Internet networks. CoAP is an application layer protocol that is intended for use in resource-constrained internet devices, such as WSN nodes.
DTLS (IETF)
The Datagram Transport Layer Security (DTLS) communications protocol provides communications security for datagram protocols. - Wikipedia
DTLS allows datagram-based applications to communicate in a way that is designed[by whom?] to prevent eavesdropping, tampering, or message forgery. The DTLS protocol is based on the stream-oriented Transport Layer Security (TLS) protocol and is intended to provide similar security guarantees.
- Eddystone (Google)
Eddystone is a beacon technology profile released by Google in July 2015. The open source, cross-platform software gives users location and proximity data via Bluetooth low-energy beacon format. - Wikipedia
Though similar to the iBeacon released by Apple in 2013, Eddystone works on both Android and iOS, whereas iBeacon is limited to iOS platforms. A practical application of both softwares is that business owners can target potential customers based on the location of their smartphones in real time.
- HTTP (IETF)
The Hypertext Transfer Protocol (HTTP) is an application protocol for distributed, collaborative, hypermedia information systems. HTTP is the foundation of data communication for the World Wide Web. - Wikipedia
The standards development of HTTP was coordinated by the Internet Engineering Task Force (IETF) and the World Wide Web Consortium (W3C), culminating in the publication of a series of Requests for Comments (RFCs). The first definition of HTTP/1.1, the version of HTTP in common use, occurred in RFC 2068 in 1997, although this was obsoleted by RFC 2616 in 1999.
- iBeacon (Apple)
iBeacon is a protocol standardized by Apple and introduced at the Apple Worldwide Developers Conference in 2013. - Wikipedia
iBeacon uses Bluetooth low energy proximity sensing to transmit a universally unique identifier picked up by a compatible app or operating system. The identifier can be used to determine the device's physical location, track customers, or trigger a location-based action on the device such as a check-in on social media or a push notification.
- MQTT (IBM)
MQTT (formerly MQ Telemetry Transport) is a publish-subscribe based "light weight" messaging protocol for use on top of the TCP/IP protocol. It is designed for connections with remote locations where a "small code footprint" is required or the network bandwidth is limited. - Wikipedia
The publish-subscribe messaging pattern requires a message broker. The broker is responsible for distributing messages to interested clients based on the topic of a message. Andy Stanford-Clark and Arlen Nipper of Cirrus Link Solutions authored the first version of the protocol in 1999.
- STOMP
Simple (or Streaming) Text Oriented Message Protocol (STOMP), formerly known as TTMP, is a simple text-based protocol, designed for working with message-oriented middleware (MOM). - Wikipedia
STOMP provides an interoperable wire format that allows STOMP clients to talk with any message broker supporting the protocol. It is thus language-agnostic, meaning a broker developed for one programming language or platform can receive communications from client software developed in another language.
- Websocket
WebSocket is a protocol providing full-duplex communication channels over a single TCP connection. - Wikipedia
WebSocket is designed to be implemented in web browsers and web servers, but it can be used by any client or server application. The WebSocket Protocol is an independent TCP-based protocol. The WebSocket protocol makes more interaction between a browser and a website possible, facilitating live content and the creation of real-time games. This is made possible by providing a standardized way for the server to send content to the browser without being solicited by the client, and allowing for messages to be passed back and forth while keeping the connection open.
- XMPP (IETF)
Extensible Messaging and Presence Protocol (XMPP) is a communications protocol for message-oriented middleware based on XML (Extensible Markup Language). - Wikipedia
It enables the near-real-time exchange of structured yet extensible data between any two or more network entities. Designed to be extensible, the protocol has also been used for publish-subscribe systems, signalling for VoIP, video, file transfer, gaming, Internet of Things (IoT) applications such as the smart grid, and social networking services.
Technologies
This sections regroups a curated list of awesome technologies that are closely related to the IoT world.
- NFC
Near field communication (NFC) is the set of protocols that enable electronic devices to establish radio communication with each other by touching the devices together, or bringing them into proximity to a distance of typically 10cm or less. - Wikipedia
- OPCUA
OPC-UA is a not only a protocol for industrial automation but also a technology that allows semantic description and object modelling of industrial environment. Wikipedia
Standards and Alliances
Standards
ETSI M2M - The ETSI Technical Committee is developing standards for Machine to Machine Communications.
OneM2M - The purpose and goal of oneM2M is to develop technical specifications which address the need for a common M2M Service Layer that can be readily embedded within various hardware and software, and relied upon to connect the myriad of devices in the field with M2M application servers worldwide.
OPCUA - OPC Unified Architecture (OPC UA) is an industrial M2M communication protocol for interoperability developed by the OPC Foundation.
OCF - OCF, The Open Connectivity Foundation, develop standards and certification for devices involved in the Internet of Things (IoT) based around Constrained Application Protocol (CoAP).
Alliances
AIOTI - The Internet of Things Innovation (AIOTI) aims to strengthen links and build new relationships between the different IoT players (industries, SMEs, startups) and sectors.
AllSeen Alliance - The AllSeen Alliance is a nonprofit consortium dedicated to enabling and driving the widespread adoption of products, systems and services that support the Internet of Everything with an open, universal development framework supported by a vibrant ecosystem and thriving technical community.
Bluetooth Special Interest Group - The Bluetooth Special Interest Group (SIG) is the body that oversees the development of Bluetooth standards and the licensing of the Bluetooth technologies and trademarks to manufacturers.
IPSO Alliance - The IPSO Alliance provides a foundation for industry growth by fostering awareness, providing education, promoting the industry, generating research, and creating a better understanding of IP and its role in the Internet of Things.
LoRa Alliance - The LoRa Alliance is an open, non-profit association of members that believes the internet of things era is now. It was initiated by industry leaders with a mission to standardize Low Power Wide Area Networks (LPWAN) being deployed around the world to enable Internet of Things (IoT), machine-to-machine (M2M), and smart city, and industrial applications.
OPC Foundation - The mission of the OPC Foundation is to manage a global organization in which users, vendors and consortia collaborate to create data transfer standards for multi-vendor, multi-platform, secure and reliable interoperability in industrial automation. To support this mission, the OPC Foundation creates and maintains specifications, ensures compliance with OPC specifications via certification testing and collaborates with industry-leading standards organizations.
Thread Group - The Thread Group, composed of members from Nest, Samsung, ARM, Freescale, Silicon Labs, Big Ass Fans and Yale, drives the development of the Thread network protocol.
Wi-Fi Alliance - Wi-Fi Alliance® is a worldwide network of companies composed of several companies forming a global non-profit association with the goal of driving the best user experience with a new wireless networking technology – regardless of brand.
Zigbee Alliance - The ZigBee Alliance is an open, non-profit association of approximately 450 members driving development of innovative, reliable and easy-to-use ZigBee standards.
Z-Wave Alliance - Established in 2005, the Z-Wave Alliance is comprised of industry leaders throughout the globe that are dedicated to the development and extension of Z-Wave as the key enabling technology for 'smart' home and business applications.
Resources
Books
Building the Web of Things: with examples in Node.js and Raspberry Pi (2016) by Dominique Guinard and Vlad Trifa [5.0]
A hands-on guide that will teach how to design and implement scalable, flexible, and open IoT solutions using web technologies. This book focuses on providing the right balance of theory, code samples, and practical examples to enable you to successfully connect all sorts of devices to the web and to expose their services and data over REST APIs. The book covers a number of web technologies for your IoT toolbox: GPIO, Raspberry Pi, Embedded Systems, REST and HTTP, WS, MQTT, CoAP, JSON-LD, Social Networks, Node-RED, IFTTT, etc.
Abusing the Internet of Things: Blackouts, Freakouts, and Stakeouts (2015) by Nitesh Dhanjani [5.0]
future with billions of connected "things" includes monumental security concerns. This practical book explores how malicious attackers can abuse popular IoT-based devices, including wireless LED lightbulbs, electronic door locks, baby monitors, smart TVs, and connected cars.
Using the Web to Build the IoT (2016) *by Dominique Guinard
and Vlad Trifa
Using the Web to Build the IoT is a free book built as a collection of six hand-picked chapters that introduce the key technologies and concepts for building the application layer of the IoT. Dom Guinard and Vlad Trifa, selected these specific topics to give you an overview of the Web of Things architecture, along with techniques for data ingestion, searching, security, and visualization.
Building Wireless Sensor Networks: with ZigBee, XBee, Arduino, and Processing (2011) by Robert Faludi [4.5]
Get ready to create distributed sensor systems and intelligent interactive devices using the ZigBee wireless networking protocol and Series 2 XBee radios. By the time you're halfway through this fast-paced, hands-on guide, you'll have built a series of useful projects, including a complete ZigBee wireless network that delivers remotely sensed data.
Designing the Internet of Things (2013) by Adrian McEwen and Hakim Cassimally [4.0]
Whether it's called physical computing, ubiquitous computing, or the Internet of Things, it's a hot topic in technology: how to channel your inner Steve Jobs and successfully combine hardware, embedded software, web services, electronics, and cool design to create cutting-edge devices that are fun, interactive, and practical. If you'd like to create the next must-have product, this unique book is the perfect place to start.
Getting Started with Bluetooth Low Energy: Tools and Techniques for Low-Power Networking (2014) by Kevin Townsend, Carles Cufí, Akiba and Robert Davidson [4.5]
This book provides a solid, high-level overview of how devices use Ble to communicate with each other. You'll learn useful low-cost tools for developing and testing Ble-enabled mobile apps and embedded firmware and get examples using various development platforms including iOs and Android for app developers and embedded platforms for product designers and hardware engineers.
Smart Things: Ubiquitous Computing User Experience Design (2010) by Mike Kuniavsky [4.5]
Smart Things presents a problem-solving approach to addressing designers' needs and concentrates on process, rather than technological detail, to keep from being quickly outdated. It pays close attention to the capabilities and limitations of the medium in question and discusses the tradeoffs and challenges of design in a commercial environment.
JavaScript on Things: Hardware for Web Developers (2018 - est.) by Lyza Danger Gardner [early access book]
JavaScript on Things is your first step into the exciting and downright entertaining world of programming for small electronics. If you know enough JavaScript to hack a website together, you'll be making things bleep, blink and spin faster than you can say "nodebot". This fully-illustrated, hands-on book shows you how to get going with platforms like Arduino, Tessel, and Raspberry Pi.
Articles
A Simple Explanation Of 'The Internet Of Things' (Forbes) - This article attemps to give an answer to what exactly is the “Internet of things” and what impact it is going to have on us.
IoT security. Is there an app for that ? - The Internet of Things World conference investigates IoT application development, security, and business models.
The IoT Testing Atlas - A testing methodology for managing the permutations of parameters while testing an IoT based product.
Papers
A Reference Architecture for the Internet of Things - This white paper introduces a Reference Architecture for the Internet of Things (IoT): this includes the devices as well as the server-side and cloud architecture required to interact with and manage the devices.
Developing solutions for the Internet of Things - Intel's vision in enabling secure and seamless solutions for the Internet of Things (IoT).
Evaluation of indoor positioning based on Bluetooth Smart technology - Master of Science Thesis in the Programme Computer Systems and Networks.
IoT: A Vision, Architectural Elements, and Future Directions - This paper presents a cloud centric vision for worldwide implementation of Internet of Things. The key enabling technologies and application domains that are likely to drive IoT research in the near future are discussed.
Realizing the Potential of the Internet of Things - A white paper from the Telecommunications Industry Association (TIA) written in the form of a set of recommendations to policy maker on leveraging and realizing the potential of the Internet of Things market.
The Internet of Things: Evolution or Revolution ? - This white paper compares the current Internet of Things market rise to other industrial revolutions, the challenges it introduces, as well as its consequences on our daily lives.
Big Data - Everything
RDBMS
MySQL The world's most popular open source database.
PostgreSQL The world's most advanced open source database.
Oracle Database - object-relational database management system.
Teradata - high-performance MPP data warehouse platform.
Frameworks
IBM Streams - platform for distributed processing and real-time analytics. Integrates with many of the popular technologies in the Big Data ecosystem (Kafka, HDFS, Spark, etc.)
Apache Hadoop - framework for distributed processing. Integrates MapReduce (parallel processing), YARN (job scheduling) and HDFS (distributed file system).
Tigon - High Throughput Real-time Stream Processing Framework.
Pachyderm - Pachyderm is a data storage platform built on Docker and Kubernetes to provide reproducible data processing and analysis.
Distributed Programming
AddThis Hydra - distributed data processing and storage system originally developed at AddThis.
AMPLab SIMR - run Spark on Hadoop MapReduce v1.
Apache APEX - a unified, enterprise platform for big data stream and batch processing.
Apache Beam - an unified model and set of language-specific SDKs for defining and executing data processing workflows.
Apache Crunch - a simple Java API for tasks like joining and data aggregation that are tedious to implement on plain MapReduce.
Apache DataFu - collection of user-defined functions for Hadoop and Pig developed by LinkedIn.
Apache Flink - high-performance runtime, and automatic program optimization.
Apache Gearpump - real-time big data streaming engine based on Akka.
Apache Gora - framework for in-memory data model and persistence.
Apache Hama - BSP (Bulk Synchronous Parallel) computing framework.
Apache MapReduce - programming model for processing large data sets with a parallel, distributed algorithm on a cluster.
Apache Pig - high level language to express data analysis programs for Hadoop.
Apache REEF - retainable evaluator execution framework to simplify and unify the lower layers of big data systems.
Apache S4 - framework for stream processing, implementation of S4.
Apache Spark - framework for in-memory cluster computing.
Apache Spark Streaming - framework for stream processing, part of Spark.
Apache Storm - framework for stream processing by Twitter also on YARN.
Apache Samza - stream processing framework, based on Kafka and YARN.
Apache Tez - application framework for executing a complex DAG (directed acyclic graph) of tasks, built on YARN.
Apache Twill - abstraction over YARN that reduces the complexity of developing distributed applications.
Cascalog - data processing and querying library.
Cheetah - High Performance, Custom Data Warehouse on Top of MapReduce.
Concurrent Cascading - framework for data management/analytics on Hadoop.
Damballa Parkour - MapReduce library for Clojure.
Datasalt Pangool - alternative MapReduce paradigm.
DataTorrent StrAM - real-time engine is designed to enable distributed, asynchronous, real time in-memory big-data computations in as unblocked a way as possible, with minimal overhead and impact on performance.
Facebook Corona - Hadoop enhancement which removes single point of failure.
Facebook Peregrine - Map Reduce framework.
Facebook Scuba - distributed in-memory datastore.
Google Dataflow - create data pipelines to help themæingest, transform and analyze data.
Google MapReduce - map reduce framework.
Google MillWheel - fault tolerant stream processing framework.
IBM Streams - platform for distributed processing and real-time analytics. Provides toolkits for advanced analytics like geospatial, time series, etc. out of the box.
JAQL - declarative programming language for working with structured, semi-structured and unstructured data.
Kite - is a set of libraries, tools, examples, and documentation focused on making it easier to build systems on top of the Hadoop ecosystem.
Metamarkets Druid - framework for real-time analysis of large datasets.
Netflix PigPen - map-reduce for Clojure which compiles to Apache Pig.
Nokia Disco - MapReduce framework developed by Nokia.
Onyx - Distributed computation for the cloud.
Pinterest Pinlater - asynchronous job execution system.
Pydoop - Python MapReduce and HDFS API for Hadoop.
Rackerlabs Blueflood - multi-tenant distributed metric processing system
Skale - High performance distributed data processing in NodeJS.
Stratosphere - general purpose cluster computing framework.
Streamdrill - useful for counting activities of event streams over different time windows and finding the most active one.
streamsx.topology - Libraries to enable building IBM Streams application in Java, Python or Scala.
Tuktu - Easy-to-use platform for batch and streaming computation, built using Scala, Akka and Play!
Twitter Heron - Heron is a realtime, distributed, fault-tolerant stream processing engine from Twitter replacing Storm.
Twitter Scalding - Scala library for Map Reduce jobs, built on Cascading.
Twitter Summingbird - Streaming MapReduce with Scalding and Storm, by Twitter.
Twitter TSAR - TimeSeries AggregatoR by Twitter.
Wallaroo - The ultrafast and elastic data processing engine. Big or fast data - no fuss, no Java needed.
Distributed Filesystem
Ambry - a distributed object store that supports storage of trillion of small immutable objects as well as billions of large objects.
Apache HDFS - a way to store large files across multiple machines.
Apache Kudu - Hadoop's storage layer to enable fast analytics on fast data.
BeeGFS - formerly FhGFS, parallel distributed file system.
Ceph Filesystem - software storage platform designed.
Disco DDFS - distributed filesystem.
Facebook Haystack - object storage system.
Google Colossus - distributed filesystem (GFS2).
Google GFS - distributed filesystem.
Google Megastore - scalable, highly available storage.
GridGain - GGFS, Hadoop compliant in-memory file system.
Lustre file system - high-performance distributed filesystem.
Microsoft Azure Data Lake Store - HDFS-compatible storage in Azure cloud
Quantcast File System QFS - open-source distributed file system.
Red Hat GlusterFS - scale-out network-attached storage file system.
Seaweed-FS - simple and highly scalable distributed file system.
Alluxio - reliable file sharing at memory speed across cluster frameworks.
Tahoe-LAFS - decentralized cloud storage system.
Baidu File System - distributed filesystem.
Document Data Model
Actian Versant - commercial object-oriented database management systems .
Crate Data - is an open source massively scalable data store. It requires zero administration.
Facebook Apollo - Facebook’s Paxos-like NoSQL database.
jumboDB - document oriented datastore over Hadoop.
LinkedIn Espresso - horizontally scalable document-oriented NoSQL data store.
MarkLogic - Schema-agnostic Enterprise NoSQL database technology.
Microsoft Azure DocumentDB - NoSQL cloud database service with protocol support for MongoDB
MongoDB - Document-oriented database system.
RavenDB - A transactional, open-source Document Database.
RethinkDB - document database that supports queries like table joins and group by.
Key Map Data Model
Note: There is some term confusion in the industry, and two different things are called "Columnar Databases". Some, listed here, are distributed, persistent databases built around the "key-map" data model: all data has a (possibly composite) key, with which a map of key-value pairs is associated. In some systems, multiple such value maps can be associated with a key, and these maps are referred to as "column families" (with value map keys being referred to as "columns").
Another group of technologies that can also be called "columnar databases" is distinguished by how it stores data, on disk or in memory -- rather than storing data the traditional way, where all column values for a given key are stored next to each other, "row by row", these systems store all column values next to each other. So more work is needed to get all columns for a given key, but less work is needed to get all values for a given column.
The former group is referred to as "key map data model" here. The line between these and the Key-value Data Model stores is fairly blurry.
The latter, being more about the storage format than about the data model, is listed under Columnar Databases.
You can read more about this distinction on Prof. Daniel Abadi's blog: Distinguishing two major types of Column Stores.
Apache Accumulo - distributed key/value store, built on Hadoop.
Apache Cassandra - column-oriented distributed datastore, inspired by BigTable.
Apache HBase - column-oriented distributed datastore, inspired by BigTable.
Baidu Tera - an Internet-scale database, inspired by BigTable.
Facebook HydraBase - evolution of HBase made by Facebook.
Google BigTable - column-oriented distributed datastore.
Google Cloud Datastore - is a fully managed, schemaless database for storing non-relational data over BigTable.
Hypertable - column-oriented distributed datastore, inspired by BigTable.
InfiniDB - is accessed through a MySQL interface and use massive parallel processing to parallelize queries.
Tephra - Transactions for HBase.
Twitter Manhattan - real-time, multi-tenant distributed database for Twitter scale.
ScyllaDB - column-oriented distributed datastore written in C++, totally compatible with Apache Cassandra.
Key-value Data Model
Aerospike - NoSQL flash-optimized, in-memory. Open source and "Server code in 'C' (not Java or Erlang) precisely tuned to avoid context switching and memory copies."
Amazon DynamoDB - distributed key/value store, implementation of Dynamo paper.
Badger - a fast, simple, efficient, and persistent key-value store written natively in Go.
Bolt - an embedded key-value database for Go.
BTDB - Key Value Database in .Net with Object DB Layer, RPC, dynamic IL and much more
BuntDB - a fast, embeddable, in-memory key/value database for Go with custom indexing and geospatial support.
Edis - is a protocol-compatible Server replacement for Redis.
ElephantDB - Distributed database specialized in exporting data from Hadoop.
EventStore - distributed time series database.
GridDB - suitable for sensor data stored in a timeseries.
HyperDex - a scalable, next generation key-value and document store with a wide array of features, including consistency, fault tolerance and high performance.
LinkedIn Krati - is a simple persistent data store with very low latency and high throughput.
Linkedin Voldemort - distributed key/value storage system.
Oracle NoSQL Database - distributed key-value database by Oracle Corporation.
Redis - in memory key value datastore.
Riak - a decentralized datastore.
Storehaus - library to work with asynchronous key value stores, by Twitter.
SummitDB - an in-memory, NoSQL key/value database, with disk persistance and using the Raft consensus algorithm.
Tarantool - an efficient NoSQL database and a Lua application server.
TiKV - a distributed key-value database powered by Rust and inspired by Google Spanner and HBase.
Tile38 - a geolocation data store, spatial index, and realtime geofence, supporting a variety of object types including latitude/longitude points, bounding boxes, XYZ tiles, Geohashes, and GeoJSON
TreodeDB - key-value store that's replicated and sharded and provides atomic multirow writes.
Graph Data Model
AgensGraph - a new generation multi-model graph database for the modern complex data environment.
Apache Giraph - implementation of Pregel, based on Hadoop.
Apache Spark Bagel - implementation of Pregel, part of Spark.
ArangoDB - multi model distributed database.
DGraph - A scalable, distributed, low latency, high throughput graph database aimed at providing Google production level scale and throughput, with low enough latency to be serving real time user queries, over terabytes of structured data.
EliasDB - a lightweight graph based database that does not require any third-party libraries.
Facebook TAO - TAO is the distributed data store that is widely used at facebook to store and serve the social graph.
GCHQ Gaffer - Gaffer by GCHQ is a framework that makes it easy to store large-scale graphs in which the nodes and edges have statistics.
Google Cayley - open-source graph database.
Google Pregel - graph processing framework.
GraphLab PowerGraph - a core C++ GraphLab API and a collection of high-performance machine learning and data mining toolkits built on top of the GraphLab API.
GraphX - resilient Distributed Graph System on Spark.
Gremlin - graph traversal Language.
Infovore - RDF-centric Map/Reduce framework.
Intel GraphBuilder - tools to construct large-scale graphs on top of Hadoop.
MapGraph - Massively Parallel Graph processing on GPUs.
Neo4j - graph database writting entirely in Java.
OrientDB - document and graph database.
Phoebus - framework for large scale graph processing.
Titan - distributed graph database, built over Cassandra.
Twitter FlockDB - distributed graph database.
NodeXL - A free, open-source template for Microsoft® Excel® 2007, 2010, 2013 and 2016 that makes it easy to explore network graphs.
Columnar Databases
Note please read the note on Key-Map Data Model section.
Columnar Storage - an explanation of what columnar storage is and when you might want it.
Actian Vector - column-oriented analytic database.
C-Store - column oriented DBMS.
ClickHouse - an open-source column-oriented database management system that allows generating analytical data reports in real time.
EventQL - a distributed, column-oriented database built for large-scale event collection and analytics.
MonetDB - column store database.
Parquet - columnar storage format for Hadoop.
Pivotal Greenplum - purpose-built, dedicated analytic data warehouse that offers a columnar engine as well as a traditional row-based one.
Vertica - is designed to manage large, fast-growing volumes of data and provide very fast query performance when used for data warehouses.
SQream DB - A GPU powered big data database, designed for analytics and data warehousing, with ANSI-92 compliant SQL, suitable for data sets from 10TB to 1PB.
Google BigQuery Google's cloud offering backed by their pioneering work on Dremel.
Amazon Redshift Amazon's cloud offering, also based on a columnar datastore backend.
IndexR an open-source columnar storage format for fast & realtime analytic with big data.
NewSQL Databases
Actian Ingres - commercially supported, open-source SQL relational database management system.
Amazon RedShift - data warehouse service, based on PostgreSQL.
BayesDB - statistic oriented SQL database.
Bedrock - a simple, modular, networked and distributed transaction layer built atop SQLite.
CitusDB - scales out PostgreSQL through sharding and replication.
Cockroach - Scalable, Geo-Replicated, Transactional Datastore.
Comdb2 - a clustered RDBMS built on optimistic concurrency control techniques.
Datomic - distributed database designed to enable scalable, flexible and intelligent applications.
FoundationDB - distributed database, inspired by F1.
Google F1 - distributed SQL database built on Spanner.
Google Spanner - globally distributed semi-relational database.
H-Store - is an experimental main-memory, parallel database management system that is optimized for on-line transaction processing (OLTP) applications.
Haeinsa - linearly scalable multi-row, multi-table transaction library for HBase based on Percolator.
HandlerSocket - NoSQL plugin for MySQL/MariaDB.
InfiniSQL - infinity scalable RDBMS.
MemSQL - in memory SQL database witho optimized columnar storage on flash.
NuoDB - SQL/ACID compliant distributed database.
Oracle TimesTen in-Memory Database - in-memory, relational database management system with persistence and recoverability.
Pivotal GemFire XD - Low-latency, in-memory, distributed SQL data store. Provides SQL interface to in-memory table data, persistable in HDFS.
SAP HANA - is an in-memory, column-oriented, relational database management system.
SenseiDB - distributed, realtime, semi-structured database.
Sky - database used for flexible, high performance analysis of behavioral data.
SymmetricDS - open source software for both file and database synchronization.
Map-D - GPU in-memory database, big data analysis and visualization platform
TiDB - TiDB is a distributed SQL database. Inspired by the design of Google F1.
VoltDB - claims to be fastest in-memory database
Time-Series Databases
Axibase Time Series Database - Integrated time series database on top of HBase with built-in visualization, rule-engine and SQL support.
Chronix - a time series storage built to store time series highly compressed and for fast access times.
Cube - uses MongoDB to store time series data.
Heroic - is a scalable time series database based on Cassandra and Elasticsearch.
InfluxDB - distributed time series database.
Kairosdb - similar to OpenTSDB but allows for Cassandra.
Newts - a time series database based on Apache Cassandra.
OpenTSDB - distributed time series database on top of HBase.
Prometheus - a time series database and service monitoring system.
Beringei - Facebook's in-memory time-series database.
TrailDB - an efficient tool for storing and querying series of events.
Druid Column oriented distributed data store ideal for powering interactive applications
Riak-TS Riak TS is the only enterprise-grade NoSQL time series database optimized specifically for IoT and Time Series data.
Akumuli Akumuli is a numeric time-series database. It can be used to capture, store and process time-series data in real-time. The word "akumuli" can be translated from esperanto as "accumulate".
Rhombus A time-series object store for Cassandra that handles all the complexity of building wide row indexes.
Dalmatiner DB Fast distributed metrics database
Blueflood A distributed system designed to ingest and process time series data
Timely Timely is a time series database application that provides secure access to time series data based on Accumulo and Grafana.
SiriDB Highly-scalable, robust and fast, open source time series database with cluster functionality.
SQL-like processing
Actian SQL for Hadoop - high performance interactive SQL access to all Hadoop data.
Apache Drill - framework for interactive analysis, inspired by Dremel.
Apache HCatalog - table and storage management layer for Hadoop.
Apache Hive - SQL-like data warehouse system for Hadoop.
Apache Calcite - framework that allows efficient translation of queries involving heterogeneous and federated data.
Apache Phoenix - SQL skin over HBase.
Aster Database - SQL-like analytic processing for MapReduce.
Cloudera Impala - framework for interactive analysis, Inspired by Dremel.
Concurrent Lingual - SQL-like query language for Cascading.
Datasalt Splout SQL - full SQL query engine for big datasets.
Facebook PrestoDB - distributed SQL query engine.
Google BigQuery - framework for interactive analysis, implementation of Dremel.
PipelineDB - an open-source relational database that runs SQL queries continuously on streams, incrementally storing results in tables.
Pivotal HDB - SQL-like data warehouse system for Hadoop.
RainstorDB - database for storing petabyte-scale volumes of structured and semi-structured data.
Spark Catalyst - is a Query Optimization Framework for Spark and Shark.
SparkSQL - Manipulating Structured Data Using Spark.
Splice Machine - a full-featured SQL-on-Hadoop RDBMS with ACID transactions.
Stinger - interactive query for Hive.
Tajo - distributed data warehouse system on Hadoop.
Trafodion - enterprise-class SQL-on-HBase solution targeting big data transactional or operational workloads.
Data Ingestion
Amazon Kinesis - real-time processing of streaming data at massive scale.
Apache Chukwa - data collection system.
Apache Flume - service to manage large amount of log data.
Apache Kafka - distributed publish-subscribe messaging system.
Apache Sqoop - tool to transfer data between Hadoop and a structured datastore.
Cloudera Morphlines - framework that help ETL to Solr, HBase and HDFS.
Embulk - open-source bulk data loader that helps data transfer between various databases, storages, file formats, and cloud services.
Facebook Scribe - streamed log data aggregator.
Fluentd - tool to collect events and logs.
Google Photon - geographically distributed system for joining multiple continuously flowing streams of data in real-time with high scalability and low latency.
Heka - open source stream processing software system.
HIHO - framework for connecting disparate data sources with Hadoop.
Kestrel - distributed message queue system.
LinkedIn Databus - stream of change capture events for a database.
LinkedIn Kamikaze - utility package for compressing sorted integer arrays.
LinkedIn White Elephant - log aggregator and dashboard.
Logstash - a tool for managing events and logs.
Netflix Suro - log agregattor like Storm and Samza based on Chukwa.
Pinterest Secor - is a service implementing Kafka log persistance.
Linkedin Gobblin - linkedin's universal data ingestion framework.
Skizze - sketch data store to deal with all problems around counting and sketching using probabilistic data-structures.
StreamSets Data Collector - continuous big data ingest infrastructure with a simple to use IDE.
Yahoo Pulsar - a distributed pub-sub messaging platform with a very flexible messaging model and an intuitive client API.
Alooma - data pipeline as a service enabling moving data sources such as MySQL into data warehouses.
Service Programming
Akka Toolkit - runtime for distributed, and fault tolerant event-driven applications on the JVM.
Apache Avro - data serialization system.
Apache Curator - Java libaries for Apache ZooKeeper.
Apache Karaf - OSGi runtime that runs on top of any OSGi framework.
Apache Thrift - framework to build binary protocols.
Apache Zookeeper - centralized service for process management.
Google Chubby - a lock service for loosely-coupled distributed systems.
Hydrosphere Mist - a service for exposing Apache Spark analytics jobs and machine learning models as realtime, batch or reactive web services.
Linkedin Norbert - cluster manager.
OpenMPI - message passing framework.
Serf - decentralized solution for service discovery and orchestration.
Spotify Luigi - a Python package for building complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, handling failures, command line integration, and much more.
Spring XD - distributed and extensible system for data ingestion, real time analytics, batch processing, and data export.
Twitter Elephant Bird - libraries for working with LZOP-compressed data.
Twitter Finagle - asynchronous network stack for the JVM.
Scheduling
Apache Airflow - a platform to programmatically author, schedule and monitor workflows.
Apache Aurora - is a service scheduler that runs on top of Apache Mesos.
Apache Falcon - data management framework.
Apache Oozie - workflow job scheduler.
Azure Data Factory - cloud-based pipeline orchestration for on-prem, cloud and HDInsight
Chronos - distributed and fault-tolerant scheduler.
Linkedin Azkaban - batch workflow job scheduler.
Schedoscope - Scala DSL for agile scheduling of Hadoop jobs.
Sparrow - scheduling platform.
Machine Learning
Azure ML Studio - Cloud-based AzureML, R, Python Machine Learning platform
brain - Neural networks in JavaScript.
Cloudera Oryx - real-time large-scale machine learning.
Concurrent Pattern - machine learning library for Cascading.
convnetjs - Deep Learning in Javascript. Train Convolutional Neural Networks (or ordinary ones) in your browser.
DataVec - A vectorization and data preprocessing library for deep learning in Java and Scala. Part of the Deeplearning4j ecosystem.
Deeplearning4j - Fast, open deep learning for the JVM (Java, Scala, Clojure). A neural network configuration layer powered by a C++ library. Uses Spark and Hadoop to train nets on multiple GPUs and CPUs.
Decider - Flexible and Extensible Machine Learning in Ruby.
ENCOG - machine learning framework that supports a variety of advanced algorithms, as well as support classes to normalize and process data.
etcML - text classification with machine learning.
Etsy Conjecture - scalable Machine Learning in Scalding.
GraphLab Create - A machine learning platform in Python with a broad collection of ML toolkits, data engineering, and deployment tools.
H2O - statistical, machine learning and math runtime with Hadoop. R and Python.
Keras - An intuitive neural net API inspired by Torch that runs atop Theano and Tensorflow.
Mahout - An Apache-backed machine learning library for Hadoop.
MLbase - distributed machine learning libraries for the BDAS stack.
MLPNeuralNet - Fast multilayer perceptron neural network library for iOS and Mac OS X.
MOA - MOA performs big data stream mining in real time, and large scale machine learning.
MonkeyLearn - Text mining made easy. Extract and classify data from text.
ND4J - A matrix library for the JVM. Numpy for Java.
nupic - Numenta Platform for Intelligent Computing: a brain-inspired machine intelligence platform, and biologically accurate neural network based on cortical learning algorithms.
PredictionIO - machine learning server buit on Hadoop, Mahout and Cascading.
RL4J - Reinforcement learning for Java and Scala. Includes Deep-Q learning and A3C algorithms, and integrates with Open AI's Gym. Runs in the Deeplearning4j ecosystem.
SAMOA - distributed streaming machine learning framework.
scikit-learn - scikit-learn: machine learning in Python.
Spark MLlib - a Spark implementation of some common machine learning (ML) functionality.
Sibyl - System for Large Scale Machine Learning at Google.
TensorFlow - Library from Google for machine learning using data flow graphs.
Theano - A Python-focused machine learning library supported by the University of Montreal.
Torch - A deep learning library with a Lua API, supported by NYU and Facebook.
Velox - System for serving machine learning predictions.
Vowpal Wabbit - learning system sponsored by Microsoft and Yahoo!.
WEKA - suite of machine learning software.
BidMach - CPU and GPU-accelerated Machine Learning Library.
Benchmarking
Apache Hadoop Benchmarking - micro-benchmarks for testing Hadoop performances.
Berkeley SWIM Benchmark - real-world big data workload benchmark.
Intel HiBench - a Hadoop benchmark suite.
PUMA Benchmarking - benchmark suite for MapReduce applications.
Yahoo Gridmix3 - Hadoop cluster benchmarking from Yahoo engineer team.
Deeplearning4j Benchmarks
Security
Apache Ranger - Central security admin & fine-grained authorization for Hadoop
Apache Eagle - real time monitoring solution
Apache Knox Gateway - single point of secure access for Hadoop clusters.
Apache Sentry - security module for data stored in Hadoop.
BDA - The vulnerability detector for Hadoop and Spark
System Deployment
Apache Ambari - operational framework for Hadoop mangement.
Apache Bigtop - system deployment framework for the Hadoop ecosystem.
Apache Helix - cluster management framework.
Apache Mesos - cluster manager.
Apache Slider - is a YARN application to deploy existing distributed applications on YARN.
Apache Whirr - set of libraries for running cloud services.
Apache YARN - Cluster manager.
Brooklyn - library that simplifies application deployment and management.
Buildoop - Similar to Apache BigTop based on Groovy language.
Cloudera HUE - web application for interacting with Hadoop.
Facebook Prism - multi datacenters replication system.
Google Borg - job scheduling and monitoring system.
Google Omega - job scheduling and monitoring system.
Hortonworks HOYA - application that can deploy HBase cluster on YARN.
Marathon - Mesos framework for long-running services.
Applications
411 - an web application for alert management resulting from scheduled searches into Elasticsearch.
Adobe spindle - Next-generation web analytics processing with Scala, Spark, and Parquet.
Apache Kiji - framework to collect and analyze data in real-time, based on HBase.
Apache Metron - a platform that integrates a variety of open source big data technologies in order to offer a centralized tool for security monitoring and analysis.
Apache Nutch - open source web crawler.
Apache OODT - capturing, processing and sharing of data for NASA's scientific archives.
Apache Tika - content analysis toolkit.
Argus - Time series monitoring and alerting platform.
AthenaX - a streaming analytics platform that enables users to run production-quality, large scale streaming analytics using Structured Query Language (SQL).
Atlas - a backend for managing dimensional time series data.
Countly - open source mobile and web analytics platform, based on Node.js & MongoDB.
Domino - Run, scale, share, and deploy models — without any infrastructure.
Eclipse BIRT - Eclipse-based reporting system.
ElastAert - ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in ElasticSearch.
Eventhub - open source event analytics platform.
Hermes - asynchronous message broker built on top of Kafka.
HIPI Library - API for performing image processing tasks on Hadoop's MapReduce.
Hunk - Splunk analytics for Hadoop.
Imhotep - Large scale analytics platform by indeed.
MADlib - data-processing library of an RDBMS to analyze data.
Kapacitor - an open source framework for processing, monitoring, and alerting on time series data.
Kylin - open source Distributed Analytics Engine from eBay.
PivotalR - R on Pivotal HD / HAWQ and PostgreSQL.
Rakam - open-source real-time custom analytics platform powered by Postgresql, Kinesis and PrestoDB.
Qubole - auto-scaling Hadoop cluster, built-in data connectors.
Sense - Cloud Platform for Data Science and Big Data Analytics.
SnappyData - a distributed in-memory data store for real-time operational analytics, delivering stream analytics, OLTP (online transaction processing) and OLAP (online analytical processing) built on Spark in a single integrated cluster.
Snowplow - enterprise-strength web and event analytics, powered by Hadoop, Kinesis, Redshift and Postgres.
SparkR - R frontend for Spark.
Splunk - analyzer for machine-generated data.
Sumo Logic - cloud based analyzer for machine-generated data.
Talend - unified open source environment for YARN, Hadoop, HBASE, Hive, HCatalog & Pig.
Warp - query by example tool for big data (OS X app)
Search engine and framework
Apache Lucene - Search engine library.
Apache Solr - Search platform for Apache Lucene.
Elassandra - is a fork of Elasticsearch modified to run on top of Apache Cassandra in a scalable and resilient peer-to-peer architecture.
ElasticSearch - Search and analytics engine based on Apache Lucene.
Enigma.io – Freemium robust web application for exploring, filtering, analyzing, searching and exporting massive datasets scraped from across the Web.
Facebook Unicorn - social graph search platform.
Google Caffeine - continuous indexing system.
Google Percolator - continuous indexing system.
TeraGoogle - large search index.
HBase Coprocessor - implementation of Percolator, part of HBase.
Lily HBase Indexer - quickly and easily search for any content stored in HBase.
LinkedIn Bobo - is a Faceted Search implementation written purely in Java, an extension to Apache Lucene.
LinkedIn Cleo - is a flexible software library for enabling rapid development of partial, out-of-order and real-time typeahead search.
LinkedIn Galene - search architecture at LinkedIn.
LinkedIn Zoie - is a realtime search/indexing system written in Java.
MG4J - MG4J (Managing Gigabytes for Java) is a full-text search engine for large document collections written in Java. It is highly customisable, high-performance and provides state-of-the-art features and new research algorithms.
Sphinx Search Server - fulltext search engine.
MySQL forks and evolutions
Amazon RDS - MySQL databases in Amazon's cloud.
Drizzle - evolution of MySQL 6.0.
Google Cloud SQL - MySQL databases in Google's cloud.
MariaDB - enhanced, drop-in replacement for MySQL.
MySQL Cluster - MySQL implementation using NDB Cluster storage engine.
Percona Server - enhanced, drop-in replacement for MySQL.
ProxySQL - High Performance Proxy for MySQL.
TokuDB - TokuDB is a storage engine for MySQL and MariaDB.
WebScaleSQL - is a collaboration among engineers from several companies that face similar challenges in running MySQL at scale.
PostgreSQL forks and evolutions
HadoopDB - hybrid of MapReduce and DBMS.
IBM Netezza - high-performance data warehouse appliances.
Postgres-XL - Scalable Open Source PostgreSQL-based Database Cluster.
RecDB - Open Source Recommendation Engine Built Entirely Inside PostgreSQL.
Stado - open source MPP database system solely targeted at data warehousing and data mart applications.
Yahoo Everest - multi-peta-byte database / MPP derived by PostgreSQL.
TimescaleDB - An open-source time-series database optimized for fast ingest and complex queries
PipelineDB - The Streaming SQL Database. An open-source relational database that runs SQL queries continuously on streams, incrementally storing results in tables
Memcached forks and evolutions
Facebook McDipper - key/value cache for flash storage.
Facebook Memcached - fork of Memcache.
Twemproxy - A fast, light-weight proxy for memcached and redis.
Twitter Fatcache - key/value cache for flash storage.
Twitter Twemcache - fork of Memcache.
Embedded Databases
Actian PSQL - ACID-compliant DBMS developed by Pervasive Software, optimized for embedding in applications.
BerkeleyDB - a software library that provides a high-performance embedded database for key/value data.
HanoiDB - Erlang LSM BTree Storage.
LevelDB - a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.
LMDB - ultra-fast, ultra-compact key-value embedded data store developed by Symas.
RocksDB - embeddable persistent key-value store for fast storage based on LevelDB.
Business Intelligence
BIME Analytics - business intelligence platform in the cloud.
Chartio - lean business intelligence platform to visualize and explore your data.
datapine - self-service business intelligence tool in the cloud.
GoodData - platform for data products and embedded analytics.
Jaspersoft - powerful business intelligence suite.
Jedox Palo - customisable Business Intelligence platform.
Jethrodata - Interactive Big Data Analytics.
Microsoft - business intelligence software and platform.
Microstrategy - software platforms for business intelligence, mobile intelligence, and network applications.
Pentaho - business intelligence platform.
Qlik - business intelligence and analytics platform.
Redash - Open source business intelligence platform, supporting multiple data sources and planned queries.
Saiku - open source analytics platform.
SpagoBI - open source business intelligence platform.
SparklineData SNAP - modern B.I platform powered by Apache Spark.
Tableau - business intelligence platform.
Zoomdata - Big Data Analytics.
Metabase - The simplest, fastest way to get business intelligence and analytics to everyone in your company
Data Visualization
Airpal - Web UI for PrestoDB.
AnyChart - fast, simple and flexible JavaScript (HTML5) charting library featuring pure JS API.
Arbor - graph visualization library using web workers and jQuery.
Banana - visualize logs and time-stamped data stored in Solr. Port of Kibana.
Bloomery - Web UI for Impala.
Bokeh - A powerful Python interactive visualization library that targets modern web browsers for presentation, with the goal of providing elegant, concise construction of novel graphics in the style of D3.js, but also delivering this capability with high-performance interactivity over very large or streaming datasets.
C3 - D3-based reusable chart library
CartoDB - open-source or freemium hosting for geospatial databases with powerful front-end editing capabilities and a robust API.
chartd - responsive, retina-compatible charts with just an img tag.
Chart.js - open source HTML5 Charts visualizations.
Chartist.js - another open source HTML5 Charts visualization.
Crossfilter - JavaScript library for exploring large multivariate datasets in the browser. Works well with dc.js and d3.js.
Cubism - JavaScript library for time series visualization.
Cytoscape - JavaScript library for visualizing complex networks.
DC.js - Dimensional charting built to work natively with crossfilter rendered using d3.js. Excellent for connecting charts/additional metadata to hover events in D3.
D3 - javaScript library for manipulating documents.
D3.compose - Compose complex, data-driven visualizations from reusable charts and components.
D3Plus - A fairly robust set of reusable charts and styles for d3.js.
Echarts - Baidus enterprise charts.
Envisionjs - dynamic HTML5 visualization.
FnordMetric - write SQL queries that return SVG charts rather than tables
Freeboard - pen source real-time dashboard builder for IOT and other web mashups.
Gephi - An award-winning open-source platform for visualizing and manipulating large graphs and network connections. It's like Photoshop, but for graphs. Available for Windows and Mac OS X.
Google Charts - simple charting API.
Grafana - graphite dashboard frontend, editor and graph composer.
Graphite - scalable Realtime Graphing.
Highcharts - simple and flexible charting API.
IPython - provides a rich architecture for interactive computing.
Kibana - visualize logs and time-stamped data
Lumify - open source big data analysis and visualization platform
Matplotlib - plotting with Python.
Metricsgraphic.js - a library built on top of D3 that is optimized for time-series data
NVD3 - chart components for d3.js.
Peity - Progressive SVG bar, line and pie charts.
Plot.ly - Easy-to-use web service that allows for rapid creation of complex charts, from heatmaps to histograms. Upload data to create and style charts with Plotly's online spreadsheet. Fork others' plots.
Plotly.js The open source javascript graphing library that powers plotly.
Recline - simple but powerful library for building data applications in pure Javascript and HTML.
Redash - open-source platform to query and visualize data.
ReCharts - A composable charting library built on React components
Shiny - a web application framework for R.
Sigma.js - JavaScript library dedicated to graph drawing.
Superset - a data exploration platform designed to be visual, intuitive and interactive, making it easy to slice, dice and visualize data and perform analytics at the speed of thought.
Vega - a visualization grammar.
Zeppelin - a notebook-style collaborative data analysis.
Zing Charts - JavaScript charting library for big data.
Internet of things and sensor data
Apache Edgent (Incubating) - a programming model and micro-kernel style runtime that can be embedded in gateways and small footprint edge devices enabling local, real-time, analytics on the edge devices.
Azure IoT Hub - Cloud-based bi-directional monitoring and messaging hub
TempoIQ - Cloud-based sensor analytics.
2lemetry - Platform for Internet of things.
Pubnub - Data stream network
ThingWorx - Rapid development and connection of intelligent systems
IFTTT - If this then that
Evrything- Making products smart
Interesting Readings
Big Data Benchmark - Benchmark of Redshift, Hive, Shark, Impala and Stiger/Tez.
NoSQL Comparison - Cassandra vs MongoDB vs CouchDB vs Redis vs Riak vs HBase vs Couchbase vs Neo4j vs Hypertable vs ElasticSearch vs Accumulo vs VoltDB vs Scalaris comparison.
Monitoring Kafka performance - Guide to monitoring Apache Kafka, including native methods for metrics collection.
Monitoring Hadoop performance - Guide to monitoring Hadoop, with an overview of Hadoop architecture, and native methods for metrics collection.
Interesting Papers
2015 - 2016
2015 - Facebook - One Trillion Edges: Graph Processing at Facebook-Scale.
2013 - 2014
2014 - Stanford - Mining of Massive Datasets.
2013 - AMPLab - Presto: Distributed Machine Learning and Graph Processing with Sparse Matrices.
2013 - AMPLab - MLbase: A Distributed Machine-learning System.
2013 - AMPLab - Shark: SQL and Rich Analytics at Scale.
2013 - AMPLab - GraphX: A Resilient Distributed Graph System on Spark.
2013 - Google - HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm.
2013 - Microsoft - Scalable Progressive Analytics on Big Data in the Cloud.
2013 - Metamarkets - Druid: A Real-time Analytical Data Store.
2013 - Google - Online, Asynchronous Schema Change in F1.
2013 - Google - F1: A Distributed SQL Database That Scales.
2013 - Google - MillWheel: Fault-Tolerant Stream Processing at Internet Scale.
2013 - Facebook - Scuba: Diving into Data at Facebook.
2013 - Facebook - Unicorn: A System for Searching the Social Graph.
2013 - Facebook - Scaling Memcache at Facebook.
2011 - 2012
2012 - Twitter - The Unified Logging Infrastructure for Data Analytics at Twitter.
2012 - AMPLab - Blink and It’s Done: Interactive Queries on Very Large Data.
2012 - AMPLab - Fast and Interactive Analytics over Hadoop Data with Spark.
2012 - AMPLab - Shark: Fast Data Analysis Using Coarse-grained Distributed Memory.
2012 - Microsoft - Paxos Replicated State Machines as the Basis of a High-Performance Data Store.
2012 - Microsoft - Paxos Made Parallel.
2012 - AMPLab - BlinkDB: Queries with Bounded Errors and Bounded Response Times on Very Large Data.
2012 - Google - Processing a trillion cells per mouse click.
2012 - Google - Spanner: Google’s Globally-Distributed Database.
2011 - AMPLab - Scarlett: Coping with Skewed Popularity Content in MapReduce Clusters.
2011 - AMPLab - Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center.
2011 - Google - Megastore: Providing Scalable, Highly Available Storage for Interactive Services.
2001 - 2010
2010 - Facebook - Finding a needle in Haystack: Facebook’s photo storage.
2010 - AMPLab - Spark: Cluster Computing with Working Sets.
2010 - Google - Pregel: A System for Large-Scale Graph Processing.
2010 - Google - Large-scale Incremental Processing Using Distributed Transactions and Notifications base of Percolator and Caffeine.
2010 - Google - Dremel: Interactive Analysis of Web-Scale Datasets.
2010 - Yahoo - S4: Distributed Stream Computing Platform.
2009 - HadoopDB: An Architectural Hybrid of MapReduce and DBMS Technologies for Analytical Workloads.
2008 - AMPLab - Chukwa: A large-scale monitoring system.
2007 - Amazon - Dynamo: Amazon’s Highly Available Key-value Store.
2006 - Google - The Chubby lock service for loosely-coupled distributed systems.
2006 - Google - Bigtable: A Distributed Storage System for Structured Data.
2004 - Google - MapReduce: Simplied Data Processing on Large Clusters.
2003 - Google - The Google File System.
Videos
Books
Streaming
Streaming Data - Streaming Data introduces the concepts and requirements of streaming and real-time data systems.
Storm Applied - Storm Applied is a practical guide to using Apache Storm for the real-world tasks associated with processing and analyzing real-time data streams.
Fundamentals of Stream Processing: Application Design, Systems, and Analytics - This comprehensive, hands-on guide combining the fundamental building blocks and emerging research in stream processing is ideal for application designers, system builders, analytic developers, as well as students and researchers in the field.
Stream Data Processing: A Quality of Service Perspective - Presents a new paradigm suitable for stream and complex event processing.
Unified Log Processing - Unified Log Processing is a practical guide to implementing a unified log of event streams (Kafka or Kinesis) in your business
Kafka Streams in Action - Kafka Streams in Action teaches you everything you need to know to implement stream processing on data flowing into your Kafka platform, allowing you to focus on getting more from your data without sacrificing time or effort.
Big Data - Big Data teaches you to build big data systems using an architecture that takes advantage of clustered hardware along with new tools designed specifically to capture and analyze web-scale data.
Spark in Action - Spark in Action teaches you the theory and skills you need to effectively handle batch and streaming data using Spark. Fully updated for Spark 2.0.
Kafka in Action - Kafka in Action is a fast-paced introduction to every aspect of working with Kafka you need to really reap its benefits.
Reactive Data Handling - Reactive Data Handling is a collection of five hand-picked chapters, selected by Manuel Bernhardt, that introduce you to building reactive applications capable of handling real-time processing with large data loads--free eBook!
Distributed systems
Distributed Systems for fun and profit – Theory of distributed systems. Include parts about time and ordering, replication and impossibility results.
Data Visualization
The beauty of data visualization
Designing Data Visualizations with Noah Iliinsky
Hans Rosling's 200 Countries, 200 Years, 4 Minutes
Ice Bucket Challenge Data Visualization
Subscribe to:
Posts (Atom)