Consuming data via REST service with Golang

A typical use case in modern systems involves fetching data over a web service API and presenting it via a client.

Service overview

The service chosen for use is the Consumer Financial Protection Bureau’s Complaints Database. It’s permissive license, free cost, and authenticated API nicely fits our use-case.

As an added bonus, their API supports SoQL queries which enables us to interact with their data more efficiently.

Specific use case

We’ll use the following specific use case to go through the typical activity described in the introduction:

Graph the number of complaints registered with the CFPB in the past 3 days, grouped by company.

Fetch data for count of recent complaints

To begin, we know we need the count of recent complaints from the CFPB’s database. We’ll isolate this interaction to a new function:

// CompanyComplaintCount represents a company and complaints about it
type CompanyComplaintCount struct {
    Company string `json:"company"`
    Count   int    `json:"count_company,string"`
}

func getRecentCompanyComplaints(ctx context.Context) ([]CompanyComplaintCount, error) {
    log.Infof(ctx, "Starting http request to get consumer data")
    dateParam := time.Now().AddDate(0, 0, -2)
    newURL := fmt.Sprintf("https://data.consumerfinance.gov/resource/jhzv-w97w.json?$select=company,count(company)&$where=date_received%%3E=%%27%v%%27&$group=company&$order=count(company)%%20desc", dateParam.Format("2006-01-02T00:00:00"))

    log.Infof(ctx, "URL: %v", newURL)

    req, err := http.NewRequest("GET", newURL, nil)
    req.Header.Set("X-App-Token", os.Getenv("SODA_API_KEY"))

    client := urlfetch.Client(ctx)
    // get all recent complaints grouped by company
    resp, err := client.Do(req)
    target := []CompanyComplaintCount{}
    if err != nil {
        log.Errorf(ctx, "Error doing http request: %v", err)
        return nil, err
    }
    log.Infof(ctx, "HTTP GET returned status %v", resp.Status)
    json.NewDecoder(resp.Body).Decode(&target)

    log.Infof(ctx, "Number of records found on server %v", len(target))
    return target, nil
}

Note, the use of urlfetch.Client(ctx) is Google App Engine-specific, and is the necessary way of executing outbound HTTP requests on GAE’s platform. They encourage this behavior, which uses their URL Fetch service behind the scenes, in lieu of the standard net/http Go package, where you’d normally issue requests requiring custom headers by creating a new http.Client and executing client.Get(...).

Reviewing the outbound data request

We need to create our SoQL query that fetches a count of companies who have complaints logged in the past 3 days, then groups them by each company.

We then take that query and add it to the query parameter to the service endpoint with the following lines:

dateParam := time.Now().AddDate(0, 0, -2)
newURL := fmt.Sprintf("https://data.consumerfinance.gov/resource/jhzv-w97w.json?$select=company,count(company)&$where=date_received%%3E=%%27%v%%27&$group=company&$order=count(company)%%20desc", dateParam.Format("2006-01-02T00:00:00"))

We configure our HTTP request to include a header containing our API key for authenticated access:

req, err := http.NewRequest("GET", newURL, nil)
req.Header.Set("X-App-Token", os.Getenv("SODA_API_KEY"))
client := urlfetch.Client(ctx)

The SODA_API_KEY is read from a system environment variable, which is a good practice which avoids the need to hardode credentials in code. In our GAE app, the value exists in our app.yaml file.

Lastly we execute the outbound request and parse the results into our CompanyComplaintCount target which is returned by the function:

resp, err := client.Do(req)
target := []CompanyComplaintCount{}
if err != nil {
    log.Errorf(ctx, "Error doing http request: %v", err)
    return nail, err
}
json.NewDecoder(resp.Body).Decode(&target)
return target, nil

Notice also that the CompanyComplaintCount struct has built-in support for specifying the mapping between the struct’s fields and the JSON object’s:

Company string `json:"company"`
Count   int    `json:"count_company,string"`

Build page to render output

I’m using Charts.js for the graph.

Since the point of our use-case it to explore Golang, it’s enough to render the Javascript configuration server-side. Charts.js also works nicely if we wanted to hook it into a single-page Javascript applciation.

We get started by specifying a canvas element inside a file named complaint-report.html:

<canvas id="myChart" width="400" height="400"></canvas>

Then configure the myChart element using JQuery:

var ctx = document.getElementById("myChart");
var myChart = new Chart(ctx, {
    type: 'bar',
    data: {
        labels: {{.Labels}},
        datasets: [{
            label: {{printf "%s" .Title }},
            data: {{.Data}},
            backgroundColor: {{.BackgroundColor}},
            borderColor: {{.BorderColor}},
            borderWidth: 1
        }]
    },
    options: {
        scales: {
            yAxes: [{
                ticks: {
                    beginAtZero:true,
                    stepSize: 1
                },
            }],
            xAxes: [{
                display: false
            }],
        },
         legend: {
            display: false,
        },
        responsive: true,
    }
});

Inspecting the chart-rendering code

The JavaScript code is a basic JQuery function which grabs the myChart elemnent and configures it using the Charts.js graph API.

We supply the data for graph on the server-side when the template renders per each request.

Here you see the Go template syntax which specifies the graph’s colors, dependent on the value of the Model struct supplied to the view template:

backgroundColor: {{.BackgroundColor}},
borderColor: {{.BorderColor}},

The full struct we’re using for this graph looks like:

type ChartData struct {
    Title           string
    Message         string
    Labels          []string
    Data            []int
    BackgroundColor []string
    BorderColor     []string
}

Rendering the view handler

Our remaining task is to register a route for the view handler and implement its logic. This will involve invoking our getRecentCompanyComplaints() function to get the data, using the data to configure our ChartData struct, and passing that struct to the complaint-report.html template.

And the logic

func consumerFeedHandler(w http.ResponseWriter, r *http.Request) {
    ctx := appengine.NewContext(r)
    results, err := getRecentCompanyComplaints(ctx)

    // iterate over array of results and compile arrays of distinct data`
    labels := make([]string, 0)
    data := make([]int, 0)
    backgroundPallete := []string{
        "rgba(255, 99, 132, 0.2)",
        "rgba(54, 162, 235, 0.2)",
        "rgba(75, 192, 192, 0.2)",
        "rgba(255, 206, 86, 0.2)",
        "rgba(153, 102, 255, 0.2)",
        "rgba(255, 159, 64, 0.2)"}
    borderPallette := []string{
        "rgba(255,99,132,1)",
        "rgba(54, 162, 235, 1)",
        "rgba(75, 192, 192, 1)",
        "rgba(255, 206, 86, 1)",
        "rgba(153, 102, 255, 1)",
        "rgba(255, 159, 64, 1)"}

    backgroundColor := make([]string, 0)
    borderColor := make([]string, 0)

    numBackgrounds := len(backgroundPallete)
    for _, record := range results {
        data = append(data, record.Count)
        labels = append(labels, record.Company)
        colorIndex := record.Count % numBackgrounds
        backgroundColor = append(backgroundColor, backgroundPallete[colorIndex])
        borderColor = append(borderColor, borderPallette[colorIndex])
    }

    chartData := ChartData{Title: "Complaints the past three days",
        Message:         "This chart depicts complaints against companies made to the Consumer Financial Protection Bureau in the past three days.  Data is fetched from their Complaints Database via their web API.",
        Labels:          labels,
        Data:            data,
        BackgroundColor: backgroundColor,
        BorderColor:     borderColor}

    theme.Render("complaint-report.html", w, &chartData)
}

Inspect code to render data to struct and view template

We get our results array by invoking our outbound HTTP request method:

results, err := getRecentCompanyComplaints(ctx)

Then for each result, supply the number of complaints and company name:

for _, record := range results {
        data = append(data, record.Count)
        labels = append(labels, record.Company)
        //...

Earlier we define arrays of appropriate fill/border colors. Then use modulus to loop through these colors so each bar in the graph cycles through colors appropriately:

colorIndex := record.Count % numBackgrounds
backgroundColor = append(backgroundColor, backgroundPallete[colorIndex])
borderColor = append(borderColor, borderPallette[colorIndex])

The last task is to construct the struct with these values and render the template:

    chartData := ChartData{Title: "Complaints the past three days",
        Message:         "This chart depicts complaints against companies made to the Consumer Financial Protection Bureau in the past three days.  Data is fetched from their Complaints Database via their web API.",
        Labels:          labels,
        Data:            data,
        BackgroundColor: backgroundColor,
        BorderColor:     borderColor}

    theme.Render("complaint-report.html", w, &chartData)

Viewing the graph

Live demo of all this code put together is running at: M0d3rn C0ad Consumer Complaints

The resultant graph looks like: Chart of company complaint counts

We futher supply an onClick event to our template so that the bars are clickable, which lets a person drill-down to see the details of each complaint per company: Detail of company complaints

Summary

We satisfyied our use case to expereince a typical task in fetching data via an outbound HTTP request, transforming that data, and rending it to a view.

We further followed a best practice to extract authentication credentials from the application code and configured our request to include custom headers.

Another approach we could’ve taken is to fetch the data using the same backend code and make it available via a RESTful service to a more modern client, such as a single-page JavaScript application or a mobile app. While that activity falls outside our use case, it’s a likely topic for a future post.

◀   Hosting a static site for free on Firebase Simple Nginx site in Docker   ▶