Full NGINX Logging Example

In a previous post I talked about how to setup NGINX to log both the request and response headers and bodies to help with debugging. It’s a lot easier to set up a HTTP reverse proxy than sort out all the private keys when trying to capture HTTPS with wireshark.

I needed to use that again recently and it took me a little while to remember exactly how to put it all together again, so this is just a really short follow up post with a full minimal example of the nginx.conf file needed.

worker_processes 1;
load_module modules/ndk_http_module.so;
load_module modules/ngx_http_lua_module.so;
pcre_jit on;

events {
  worker_connections 1024;
}

error_log /dev/stderr;

http {
  log_format log_req_resp '$remote_addr - $remote_user [$time_local] '
  '"$request" $status $body_bytes_sent '
  '"$http_referer" "$http_user_agent" $request_time req_body:"$request_body" resp_body:"$resp_body" '
  'req_headers:"$req_header" resp_headers:"$resp_header"';

  server {
    listen 80;
    access_log /dev/stdout log_req_resp;

    root /var/www/html;

    lua_need_request_body on;

    set $resp_body "";
    body_filter_by_lua '
      local resp_body = string.sub(ngx.arg[1], 1, 1000)
      ngx.ctx.buffered = (ngx.ctx.buffered or "") .. resp_body
      if ngx.arg[2] then
        ngx.var.resp_body = ngx.ctx.buffered
      end
    ';

    set $req_header "";
    set $resp_header "";
    header_filter_by_lua '
      local h = ngx.req.get_headers()
      for k, v in pairs(h) do
        if (type(v) == "table") then
          ngx.var.req_header = ngx.var.req_header .. k.."="..table.concat(v,",").." "
        else
          ngx.var.req_header = ngx.var.req_header .. k.."="..v.." "
        end
      end
      local rh = ngx.resp.get_headers()
      for k, v in pairs(rh) do
        ngx.var.resp_header = ngx.var.resp_header .. k.."="..v.." "
      end
      ';

  }
}

It also has a small improvement to allow for duplicate HTTP headers in the request (which is in spec). It will now concatenate the values in to a comma separated list.

This is intended to be used with the following Docker container (the default nginx container doe not have the lua module installed)

FROM debian:latest

RUN apt-get update && apt-get install -y libnginx-mod-http-lua libnginx-mod-http-ndk

CMD ["nginx", "-g", "daemon off;"]

Run as follows

docker run --rm -v `pwd`/nginx.conf:/etc/nginx/nginx.conf:ro -p 80:80 custom-nginx

Multipart HTTP Post requests with Volley on Android

It’s been a little while since I’ve done any really serious Android development, but a couple of projects have brought me back to it.

Early on in one of those projects I had to make some HTTP requests, my first thought was to make use of the Apache HTTP Client classes as I had done many times before on Android. Which is why I was a little surprised when the usual ctrl-space didn’t auto complete any of the expected class names.

It turns out the classes were removed in Android 6.0 and the notice suggests using the HttpURLConnection class. A little bit more digging turned up a wrapper for this called Volley.

Volley is a wrapper round the HttpURLConnection class to provides a neat asynchronous interface that does IO in the background and then delivers results to the Main thread so UI updates can be done with out further faffing around switching threads. There is also a nice set of tutorials on the Android Developers pages.

The first few requests all worked fine, but there was one which was a little bit more tricky. The HTTP endpoint in question accepts a multipart-form payload. A bit of googling/searching on Stackoverflow turned up a number of approaches to this and best seamed to be documented in this gist

This was close to what I wanted but not quite what I needed so I have taken some of the core concepts and built my own MultipathRequest object.

...
MultipartRequest request = new MultipartRequest(url, headers, 
    new Response.Listener<NetworkResponse>() {
        @Override
        public void onResponse(NetworkResponse response) {
        ...
        }
    },
    new Response.ErrorListener() {
        @Override
        public void onErrorResponse(VolleyError error) {
        ...
        }
    });
    
request.addPart(new FormPart(fieldName,value));
request.addPart(new FilePart(fileFieldName, mimeType, fileName, data);

requestQueue.add(request);
...

I’ve stuck the code up on github here. You can include it in your Android Project by adding the following to the build.gradle in the root of the project:

allprojects {
  repositories {
    ...
    maven { url 'https://jitpack.io' }
  }
}

And then this to the dependencies section of the modules build.gradle:

dependencies {
  compile 'com.github.hardillb:MultiPartVolley:0.0.3'
}