Skip to main content

Processing a large dataset in less than 100 lines of Node.js with async.queue

If you’re more of a skip to the code person, check out the gist here.

caolan’s async.queue to the rescue


To fix the call stack issue needed to manage API calls by pushing them into a queue where they could be processed in parallel. 

Pushing items to the queue

Image IDs are in a newline delimited JSON file. First convert this file into a JSON object using readFileSync. The object contains a list of image IDs and in queue want to send each image to the Vision API. The queue takes a task (in this case my object of image IDs) and a callback function, called when the worker is finished processing:

q.push(imageIds, function (err) {
 if (err) {
  console.log(err)
 }
});


Defining the queue

The queue takes a function and a concurrency number as parameters. Let’s start with the function: we pass it a task (our image ID from above) and a callback, which will be called when the worker completes a task. Inside the function is where do image processing.
This function should return some JSON about the image which want to write to a local JSON file. Will define that in the next step.

                        Concurrency tells Node.js the maximum number of workers to process our task in parallel. Playing with the number until  found a balance of something that wasn’t too slow, but also didn’t result in API limits or call stack errors. The number will vary depending on what you’re doing, so it’s definitely ok to fine tune it by hand until you find your “magic number.” Here’s queue:

let q = async.queue(callVision, 20);

Processing images


Last, it’s time to write the callVision() function referenced above. This part isn’t exactly async.queue specific, but it’s still important because it’s the meat of my queue task. 
Here using Google’s Cloud Vision API for image analysis, and use the Google Cloud Node.js module to call it. Once get a JSON response for each image, create a JSON string of the response to write to a newline delimited JSON file (using this format because it’s  what BigQuery expects, which is where will be storing the data eventually). Once this function completes, the data is sent back to the queue where it is written to local JSON file. You can find all of the callVision() code in the gist.

That’s it!  you’ve done something interesting with async.queue 



                                                                                        *Sara Robinson






Comments

Popular posts from this blog

Design Tools to Help You Create Your Next Project- Part 3

CoolorsCoolorsis a super fast color scheme generator. You can explore thousands of pre-existing color schemes (each one features five colors). Or, you can generate your own in a matter of minutes. Once you go to the “generate” page, hit the space bar to start with a different color scheme, and then you can adjust each color’s hue, saturation, and brightness accordingly.

Web GradientsWeb Gradientsis a collection of almost 200 background gradients, created by the itmeo team. You can use each of these content backdrops for any part of your website. You’ll find a .PNG version of each gradient, as well as easy-to-copy CSS3 crossbrowser code. Bonus: there are even curated packs for Sketch & Photoshop.

Color Hunt On Color Hunt

MySQL Connector/Net 8.0.8-dmr has been released on 11th July, 2017

MySQL Connector/Net 8.0.8 is the fifth development release that expands cross-platform
support to Linux and macOS when using Microsoft’s .NET Core framework. Now, .NET
developers can use the X DevAPI with .NET Core and Entity Framework Core (EF Core)
1.0 to create server applications that run on Windows, Linux and macOS. We are very
excited about this change and really look forward to your feedback on it! MySQL Connector/Net 8.0.8 is also the seventh development release of MySQL
Connector/Net to add support for the new X DevAPI. The X DevAPI enables application
developers to write code that combines the strengths of the relational and document
models using a modern, NoSQL-like syntax that does not assume previous experience
writing traditional SQL. To learn more about how to write applications using the X DevAPI, see
http://dev.mysql.com/doc/x-devapi-userguide/en/index.html.
For more information about how the X DevAPI is implemented in Connector/Net, see
http://dev.mysql.com/doc/dev/connector-n…

Get start with Vue.js

Getting Started The official guide assumes intermediate level knowledge of HTML, CSS, and JavaScript. If you are totally new to frontend development, it might not be the best idea to jump right into a framework as your first step - grasp the basics then come back! Prior experience with other frameworks helps, but is not required. The easiest way to try out Vue.js is using theJSFiddle Hello World example. Feel free to open it in another tab and follow along as we go through some basic examples. Or, you can simplycreate anindex.htmlfileand include Vue with: <scriptsrc="https://unpkg.com/vue"></script> TheInstallationpage provides more options of installing Vue. Note that wedo notrecommend beginners to start withvue-cli, especially if you are not yet familiar with Node.js-based build tools. Declarative Rendering At the core of Vue.js is a system that enables us to declaratively render data to the DOM using straightforward template syntax: <divid="app"> …