Tuesday, June 23, 2015

Building a Weather Sensor using NodeMCU with a REST API
[PART1 - FLASHING ESP8266]

Project Introduction

A couple of weeks ago, I received my newly ordered ESP8266 SoC. In some way related with my thesis research, I got really interested in the idea of building a small and very cheap energy sensor.

Also the main reason why I choose ESP8266 over an Arduino, Raspberry-PI or other alternatives its because, they consume too much power, have too many resources for an embedded device—most Arduinos found out there have Dual or Quad core processors, — and they are very expensive for the end goal of a simple sensor. Anyway, it would be a very long post to compare between ESP8266 and alternatives in a fair and measured way, so I will save it for a future post.

So, to warm up I decided to build a simple weather sensor, just to get used about how to flash new firmware, access GPIOs, access the MCU via USB and so on. Hopefully, this post will show you how to get started on programming an ESP8266 with NodeMCU.

Fig.1 ESP8266 - ESP12E
Fig.2 ESP8266 on Amica R2 DevBoard v1.0

Required Equipment
  1. ESP8266 with DevBoard (NodeMCU LUA Amica R2 | ESP8266 Wifi Board)
  2. Breadboard
  3. Wires
  4. USB to MicroUSB cable
  5. DHT11 sensor
Some notes on the equipment, for a production setup you just need the ESP8266 (Fig.1) which costs about 2-3 US$, the DHT11 sensor, the wires and some soldering material. But since we do not want to build permanently soldered prototypes we use a breadboard and a ESP8266 mounted on a DevBoard that can be plugged on our breadboard. That's why we need the equipment on Fig.2, which its just a normal ESP8266 mounted on a DevBoard, with a convenient MicroUSB connector so that you can plug it onto your computer and start hacking!

Additionally, if you buy an NodeMCU Amica R2 | ESP8266, the seller will send you an already flashed ESP8266 with NodeMCU firmware, where you can run Lua scripts.

Where to buy? http://www.electrodragon.com (trustworthy, cheap and quality material).

Our final prototype will be something like this, accessible through your browser, returning a JSON response back to you with the sensor readings! Great isn't it!




Getting Started

The following instructions are targeted for OSX users, and they may work for Linux environments. Windows users will not need wine, just install the drivers and you are good to go.

Installing the drivers

 

If you didn't notice the DevBoard MicroUSB connector is just a simple UART Bridge (Universal Asynchronous Receiver/Transmitter), so in order for your OS to recognize it, you will need to install the drivers for that UART Bridge, which is an CP2102 VCP.

The drivers for any OS can be found here, for OSX users just download your driver from the list.

Flashing ESP8266

 

To flash your ESP8266 with a new firmware you can use 1 of 2 tools, that I recommend:

  1. Esptool, to install it jump on your terminal and git this https://github.com/themadinventor/esptool. Follow their tutorial on git to setup the tool.

  2. Espressif Flasher Tool (the makers of ESP8266), just download the flashing tool from here. Its a forum of the Espressif Systems, where they post new versions of their flashing tool. Pay attention to the file name, it should be something like:
    "FLASH_DOWNLOAD_TOOLS_v[VERSION]_[RELEASE_DATE].rar".

After choosing your tool, you will need to download the latest NodeMCU firmware, you can found it here:
https://github.com/nodemcu/nodemcu-firmware/raw/master/pre_build/latest/nodemcu_latest.bin.


Assuming that you downloaded the latest firmware to your "~/Downloads" folder, and that the file is named "nodemcu_latest.bin", you can flash the ESP8266.

Before flashing the device, you will need to do the following:

  • Connect the ESP to your computer

  • Check the port where it is connect. On windows you can find this on the Device Manager. On OSX you just type "ls /dev/" on the terminal and check if you have something like "cu.SLAB_USBtoUART" or "tty.SLAB_USBtoUART" in your /dev directory.

  • If you succeed finding the port name, consider this tag [ESP8266_PORT_NAME] as the port name you just found. I will use this tag further on this post.

Flash Mode:
You will need to press the FLASH button and then the RST button while pressing the FLASH button. This will put the device in flash mode. After doing this release both any button you are pressing. Run the flashing procedure. After completion press the RST button again, and after that unplug the device from your computer. Plug it again to use your newly flashed device.

Option 1: Using the Esptool

Put your device in Flash Mode, then run the following command on your terminal (you can write the entire command on just one line):
python esptool.py --port /dev/[ESP8266_PORT_NAME] \
    write_flash 0x0 ~/Downloads/nodemcu_latest.bin -fs 32m -fm dio -ff 40m
Complete the Flashing Mode procedure and you have flashed your device with success hopefully.

Option 2: Using the Espressif Flasher Tool
  • Install Homebrew on you OSX

  • Run the command "brew install wine" to install wine. This will enable you to run the Flasher Tool Windows executable from your OSX.
  • After installing wine you need to make you your device visible within wine.
    cd ~/.wine /dosdevices
    ln -s /dev/tty.[ESP8266_PORT_NAME] com1
    
  • The previous point will make your device visible on the COM1.
  • Now start the flasher tool running the command bellow from your terminal:
    wine flash_download_tool_v[VERSION]_[RELEASE_DATE].exe 
    
  • Select your image and adjust the settings according to this screenshot:




  • Now connect the ESP8266 device using USB and perform first the Flash Mode procedure.
  • Click START in the flasher tool and wait until it is finished. Resume the last part of the Flash Mode procedure and you have flashed your device with success.

This is my 1st post of a series of 3 posts about Building a Weather Sensor using NodeMCU with a REST API. I hope you liked it, stay tuned I'll be posting the 2nd and 3rd very soon.

Thanks for reading!

Saturday, April 25, 2015

Running SQL Server script from the command-line

Have you ever needed to run a Transact-SQL script of 1 or 2GB into your SQL Server? Well that happened to me yesterday.

Normally I work with machines with enough resources to open such files, so I'm able to create the databases on SQL Server using the Management Studio. But since last week I've been working on a MacMini with just 4GB of memory, what to do when you need to open a 2GB SQL file into Management Studio? Well go do something else, because it's going to take some time.

Today I found a very elegant way to do it, using the utility SQLCMD.

Using this tool, available in your path if you have SQL Server installed, all you need to do is:
  • Open a command prompt window.
  • In the Command Prompt window, type: sqlcmd -S myServer\instanceName -i C:\myScript.sql
  • Press ENTER.
That's all folks, have fun! P.S: SQL Server 2014 Rocks!

Sunday, February 15, 2015

Avoiding [osgi.wiring.package=sun.misc] Exception with OSGi Self-Contained Bundles

On my last post "WebSockets with Grizzly on Apache Felix (OSGi)", a problem came up when installing my simple bundle into OSGi.

The exceptions faced were:


  org.osgi.framework.BundleException: Unresolved constraint in bundle 
  automationmiddleware.plugins.remoteagent [29]: Unable to resolve 29.0: missing requirement 
  [29.0] osgi.wiring.package; (&(osgi.wiring.package=org.glassfish.grizzly)(version>=2.3.0)(
  !(version>=3.0.0)))



  org.osgi.framework.BundleException: Unresolved constraint in bundle 
  automationmiddleware.plugins.remoteagent [29]: Unable to resolve 29.0: missing requirement 
  [29.0] osgi.wiring.package; (osgi.wiring.package=sun.misc)
at org.apache.felix.framework.Felix.resolveBundleRevision(Felix.java:4002)


The first exception happens because Felix cannot find org.glassfish.grizzly package which is referenced by my bundle. This is easily solvable by dropping the grizzly-httpservice-bundle-2.3.18.jar bundle into the bundles folder of Felix, this bundle contains the package that Felix is failing to resolve.

The second exception occurs because by default the maven plugin maven-bundle-plugin that is typically used to package your bundle (JAR) has it parameter Import-Package set to "*".Which means "include everything that is referenced in your jar as a package to be resolved". You can easily also solve this by modifying your Felix configuration file adding this line:


  org.osgi.framework.system.packages.extra=sun.misc


The problem with both solutions above is that, they can be merely seen as a workaround, so please do not do this on a production project. 

Following the solutions above will expose all your Felix bundles to those 2 bundles, which by them selves, may dependent on different versions of sun.misc or org.glassfish.grizzly. Such dependency may cause "General Error" exceptions later on! Most probably when shipping to production, according to Murphy's law. To do it by the book as recommended here, follow the tips bellow:


  • Packaging Self-Contained Bundles, means that you bundles embed all their dependencies within them, to do this I modified my previously pom.xml file to this:

    ...
    <build>
     <sourceDirectory>src</sourceDirectory>
     <plugins>
      <plugin>
       <groupId>org.apache.felix</groupId>
       <artifactId>maven-bundle-plugin</artifactId>
       <extensions>true</extensions>
       <configuration>
        <instructions>
         <Bundle-SymbolicName>${project.artifactId};singleton:=true</Bundle-SymbolicName>
         <Bundle-Version>${project.version}</Bundle-Version>
         <Bundle-Activator>lumina.plugins.pandoraremote.Activator</Bundle-Activator>
         <Include-Resource>grizzly-httpservice-bundle-2.3.18.jar</Include-Resource>
         <Bundle-ClassPath>.,grizzly-httpservice-bundle-2.3.18.jar</Bundle-ClassPath>
         <Import-Package>sun.misc.*;resolution:=optional,*</Import-Package>
        </instructions>
       </configuration>
      </plugin>
      ...
     </plugins>
    </build>
    ...

    If you notice, on the line 14 and 15, I embed my dependency on grizzly-httpservice-bundle-2.3.18.jar within my bundle (JAR) and I also add that JAR to my classpath. This way when resolving my bundle, Felix will find the org.glassfish.grizzly package in that JAR and it will then successfully resolve my bundle dependencies.
      
  • About the sun.misc exception, as I was saying by default the Import-Package parameter of the maven plugin maven-bundle-plugin, is set to "*", to prevent Felix from trying to resolve sun.misc all you need to do is to override that parameter with this:

    <Import-Package>sun.misc.*;resolution:=optional,*</Import-Package>
    


    By declaring sun.misc resolution as optional, Felix will skip sun.misc resolution step, and your package will load successfully.

Doing things this way will help you keep your sanity a few more years, and your colleagues too ;) !

Note: Thanks to my colleague Pedro Domingues for calling my attention on the workaround I mentioned at the beginning of this post. Without his kind advise I would have carried on with my poor solution, which could eventually cause more harm than good!

Saturday, February 14, 2015

WebSockets with Grizzly on Apache Felix (OSGi)

Since July 2014, I have been in contact with OSGi and Java technologies. With a Microsoft background on my past, this technology is kind a new to me and I'm still settling on this new Java oriented world (jars/bundles, maven, nexus repositories, eclipse and other). Too bad that the receptivity to languages like C# and web technologies like ASP.NET are not very good with this Java Mafia folks that I'm working here at my university.

OSGi technology is a set of specifications that defines a dynamic component system for Java. These specifications reduce software complexity by providing a modular architecture for large-scale distributed systems as well as small, embedded applications.

On a recent project of mine that I'm currently working on, I use Apache Felix. Apache Felix is a community effort to implement the OSGi Framework and Service platform and other interesting OSGi-related technologies under the Apache license.

This project is somewhat related with my MSc Thesis, an envolves among other things realtime communication. The main goal of the system is to acquired realtime data from multiple remote endpoints distributed physically through multiple facilities, these endpoints are also known as remote agents.

The remote agents were designed to be thin clients, therefore and by principle, all the complex logic of this system relies on the central system. The central system is responsible for knowing what information must be gathered from building devices through this remote agents, and to send them instructions informing what they should do.

To accomplish this, some concerns arise:
  • Who starts the connection?
        
  • What type of connection is required? Is polling enough or are web sockets preferable?
       
  • Assuming the network middleware to gather data from building devices already exists and is based on an OSGi infrastructure using Apache Felix, how can we extend this existing system?
        
Since there is a need to acquire realtime data, and all the configuration are sent from the central system to the remote agents. These remote agents must be receptive to establish a connection with an external system. Therefore, the central system starts the connection negotiation. To address the need for realtime data, Web Sockets will be used. 

The objective resides in sending a batch of commands to configure the remote agents for the data that must be gathered. After the configuration procedure, the agent sends data continuously from the building devices to the main server, see Fig. 1.

Fig 1: Central System controls multiple agents to gather realtime
data from multiple buildings.

To solve this problem, I proposed to solve a simpler problem related with this, which consists in implementing an extension to this network middleware, that is able to accept web socket connections. Once established, these web sockets would send from the remote agents to the central system a predefined message ("hello world") every 5 seconds. This simpler solution will then enable us to solve a more complex problem (remember Divide and Conquer).

After researching a bit, I found Project Grizzly. Grizzly’s goal is to help developers to build scalable and robust servers using Java™ NIO API as well as offering extended framework components: Web Framework (HTTP/S), WebSocket, Comet, and more!

From my colleagues experiences, OSGi can be both heaven and hell when not approached properly, so since Grizzly comes with an OSGi HTTP Service implementation (including WebSockets), this seemed a great solution to start with!

Digging some internet tutorials, most outdate and featuring bugs, I came up with an working example which I am glade to be able to share with the community.

First this is a maven project, therefore we need to add Grizzly HTTP Service as a dependency. My pom.xml file is listed bellow:

<project
    xmlns="..."
    xmlns:xsi="..."
 xsi:schemaLoc.ation="...">
    <modelVersion>4.0.0</modelVersion>
    <artifactId>automationmiddleware.plugins.remoteagent</artifactId>
    <version>0.1.0-SNAPSHOT</version>
    <description>Remote Agent</description>
    <name>${project.artifactId}</name>
    <packaging>bundle</packaging>
    <build>
        <sourceDirectory>src</sourceDirectory>
    </build>
    <dependencies>
        <dependency>
            <groupId>org.glassfish.grizzly.osgi</groupId>
            <artifactId>grizzly-httpservice-bundle</artifactId>
            <version>2.3.18</version>
        </dependency>
    </dependencies>
</project>

Next we will need to create 2 classes. The HelloWorldApplication Class, responsible for accepting connection requests and determine if a WebSocket should be create or not. The second class is the HelloWorldSocket Class, this class represents the WebSocket and contains the programming logic inherent to the HelloWorldSocket—a program that sends "Hello World" messages every 5 seconds, not very cleaver in my personal opinion, but forms the base for a more complex logic like the one discussed above.

The HelloWorldApplication Class is listed bellow:

package automationmiddleware.plugins.remoteagent;

import org.glassfish.grizzly.http.HttpRequestPacket;
import org.glassfish.grizzly.websockets.ProtocolHandler;
import org.glassfish.grizzly.websockets.WebSocket;
import org.glassfish.grizzly.websockets.WebSocketApplication;
import org.glassfish.grizzly.websockets.WebSocketListener;

/**
 * HelloWorldApplication Class, accepts websocket connections for a given endpoint.
 * 
 * @author João Pinho
 */
public class HelloWorldApplication extends
        WebSocketApplication {
    
    @Override
    public boolean isApplicationRequest(HttpRequestPacket request) {
        return request.getRequestURI().endsWith("/helloworld");
    }

    @Override
    public WebSocket createSocket(ProtocolHandler handler,
                                  HttpRequestPacket requestPacket,
                                  WebSocketListener... listeners) {
        return new HelloWorldSocket(handler, requestPacket, listeners);
    }
}

The HelloWorldSocket Class is also listed bellow:

package automationmiddleware.plugins.remoteagent;

import org.glassfish.grizzly.http.HttpRequestPacket;
import org.glassfish.grizzly.websockets.DataFrame;
import org.glassfish.grizzly.websockets.DefaultWebSocket;
import org.glassfish.grizzly.websockets.ProtocolHandler;
import org.glassfish.grizzly.websockets.WebSocketListener;

/**
 * HelloWorldSocket Class.
 * 
 * @author João Pinho
 */
public class HelloWorldSocket extends DefaultWebSocket implements Runnable {
    private static final int TIMEOUT = 5000;

    public HelloWorldSocket(ProtocolHandler protocolHandler,
                            HttpRequestPacket request,
                            WebSocketListener[] listeners) {
        super(protocolHandler, request, listeners);
    }

    private Thread handlerThread;
    private boolean connected = false;

    @Override
    public void onConnect() {
        handlerThread = new Thread(this);
        connected = true;
        handlerThread.start();
        super.onConnect();
    }

    @SuppressWarnings("static-access")
    public void run() {
        while (connected) {
            try {
                Thread.currentThread().sleep(TIMEOUT);
            } catch (Exception e) {
                e.printStackTrace();
                return;
            }
            send("hello world!");
        }
    }

    @Override
    public void onClose(DataFrame frame) {
        connected = false;
        super.onClose(frame);
    }
}

Since our program needs to run within an OSGi bundle (aka, JAR) we need to create an Activator Class. An activator class has two methods a method Start called by Apache Felix to start your bundle and a method Stop, called by Felix when your bundle is stopped.

The Activator Class in the example looks like this:

package automationmiddleware.plugins.remoteagent;

import java.io.IOException;

import org.glassfish.grizzly.http.server.HttpServer;
import org.glassfish.grizzly.http.server.NetworkListener;
import org.glassfish.grizzly.websockets.WebSocketAddOn;
import org.glassfish.grizzly.websockets.WebSocketEngine;
import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;
import org.osgi.service.log.LogService;

import shared.osgi.services.logger.Logger;

/**
 * Remote Agent bundle starts a self-hosted web server listening on a predefined
 * port accepting for websocket connections.
 * 
 * @author João Pinho
 */
public final class Activator
        implements BundleActivator {

    private static final String HELLOWORLDAPP = "/helloworld";
    private static final String CONTEXT_PATH = "/remoteagent";
    private static final int SERVER_PORT = 9000;
    private static HttpServer server;
    private static HelloWorldApplication helloApp;

    @Override
    public void start(BundleContext context) throws Exception {
        new Logger().log(LogService.LOG_INFO, "[Remote Agent] started!");
        runServer();
    }

    @Override
    public void stop(BundleContext context) throws Exception {
        stopServer();
        new Logger().log(LogService.LOG_INFO, "[Remote Agent] stopped!");
    }

    private static void runServer() {
        server = HttpServer.createSimpleServer(CONTEXT_PATH, SERVER_PORT);
        final WebSocketAddOn addon = new WebSocketAddOn();
        new Logger().log(LogService.LOG_INFO, "[Remote Agent] server started on port " + SERVER_PORT + ".");

        for (NetworkListener listener : server.getListeners()) {
            listener.registerAddOn(addon);
        }

        helloApp = new HelloWorldApplication();
        WebSocketEngine.getEngine().register(CONTEXT_PATH, HELLOWORLDAPP, helloApp);

        try {
            server.start();
        } catch (IOException e) {
            e.printStackTrace();
        }

        new Logger().log(LogService.LOG_INFO, "[Remote Agent] server accepting connection to "
                                              + CONTEXT_PATH + HELLOWORLDAPP + ".");
    }

    private static void stopServer() {
        if (server != null) {
            WebSocketEngine.getEngine().unregister(helloApp);
            server.shutdownNow();
        }
    }
}

After this we need to deploy our code. Unfortunately I faced some problems:
  1. After dropping my bundle (JAR) into Felix "/bundles" folder, I started felix and I got a collision of HTTP ports. For some reason Apache Felix Web Console and Grizzly HTTP Server were both trying to listen on port 9193 (which is configured in the Felix config file "/config/config.properties").

    The problem was that Felix Web Console bundle was trying to start Jetty HTTP Server on the port defined in this Felix property org.osgi.service.http.port.

    I was intrigued why Felix Web Console needed Jetty Server in first place, shouldn't it run on some isolated mode? To solve this I when to Maven Central, searched for felix-webconsole latests version and I downloaded its JAR together with its dependencies and put them all in a folder called "webconsole" inside "bundles" directory of Apache Felix.

    This solved it, and I managed to get Felix Web Console working side by side with Grizzly. Don't ask me how did I remember such solution, I think it had something to do with God intervention... Just popped into my head! Further study on this can be found at this StackOverflow answer.
      
  2. When resolving my bundle, Felix trowed this exception "Unresolved constraint in bundle org.glassfish.grizzly.osgi.grizzly-httpservice-bundle [13]: Unable to resolve 13.0: missing requirement [13.0] osgi.wiring.package; (osgi.wiring.package=sun.misc))".

    The little bastard was "sun.misc", this bundle was preventing grizzly bundle from loading. To solve this you need to change your Felix config.properties with this:

    org.osgi.framework.system.packages.extra=sun.misc
After this, my bundle started successfully.

Update: After talking to my colleagues about this, they warn me to the fact that this is just an workaround. Doing this will expose all bundles to this package, which by them selves may dependent on different version of sun.misc, which can cause "General Error" exceptions later on! Most probably when shipping to production, according to Murphy's law. To do it by the book as recommended here, please read my next post!

To test it I did a very simple web page example, the code is listed bellow:

<html>
<head>
 <script type="text/javascript" src="https://code.jquery.com/jquery-2.1.3.min.js"></script>
 <script type="text/javascript">
  var wsUri = 'ws://localhost:9000/remoteagent/helloworld';
  websocket = new WebSocket(wsUri);
 
  websocket.onopen = function(event) {
   if (console)
    console.log('connection oppened.');
  };
 
  websocket.onclose = function(event) {
   if (console)
    console.log('connection closed.');
  };
 
  websocket.onerror = function(event) {
   if (console)
    console.log('connection error.');
  };
 
  websocket.onmessage = function(event) {
   $('#container').append('<div>msg:' + event.data + '</div>');
  };
 </script>
</head>
<body>
 <div id="container"></div>
</body>
</html>

And thats it! If you are trying to replicate this example and you faced some problems, please feel free to contact me. Hope you like this. Stay tuned!