Product Listing Creator: Upload CSV feature

In the last post, I wrote about how I setup the application on both production and local environments. I also setup continuous integration using Heroku and GitHub, so that I can only focus on developing features now. As soon as a feature is developed, I can push it to the repository and it will be available on the production URL. This enables me to release my app at any time.

I also talked about focussing on more important features so that I can showcase my application with minimum features. This saves a lot of time. In this post, I am writing about developing the first feature in my application. It is to upload a CSV file containing the list of products. This is the most basic feature and somebody will start from here only. So, I decided to develop it first. I will be developing the complete feature including the UI, backend and database integration.

But before that, my application is still a React starter application, and I need to change that. First, remove the logo of React and replace it with my own application logo. I used Ucraft logo creator and Favicomatic to create my logo and favicon corresponding to it. Then I edited the App.js to replace the header with my custom header.

Once the header, title, logo, etc. was setup, I started developing the upload feature. First I developed the UI of the feature. For this, I created a new file called Main.js. Then I added the following code to create my component:

import React, { Component } from 'react';

class Main extends component {
  render () {
    return (<div className="main">
              <input className="uploadfile" type="file"></input>
              <button className="uploadbutton">Upload</button>
            </div>);
  }
}

export default Main;

Now use this component in App.js in render method after importing it:

import Main from './Main';

...
render () {
  ..
  <Main />
}

On running the application, the upload component UI can be seen. I need to fix the styling for it. For this, create a file called Main.css and add styles for classes in Main.js in it. Then import it in Main.js:

import './Main.css';

Once, I have the UI of upload component, I need to make it work, that is I should be able to upload a CSV file to it. For this, I need to implement the Upload click button. In the Main.js add an ‘onClick’ method for Upload button:

<button onClick={() => this.uploadFile()}...

Then implement the uploadFile method, but before that, I need to add a ref attribute to the input to use it in my code:

<input ref="fileinput"...

Now implement uploadFile method:

uploadFile () {
  var input = this.refs.fileinput;
  var file = input.files[0]

}

To upload the file, I need to implement a fetch method in service.js:

import 'whatwg-fetch';
import { API_URL } from './constants';

export const service = {
  uplaodFile: function(file, callback, errorCallback) {
    fetch(API_URL + '/upload', {
      method: 'POST',
      data: file
    }).then(result => {
         callback(result);
       })
      .catch(err => {
         errorCallback(err);
       });
  } 
}

Then use this service method in Main.js ‘uploadFile’ function:

import { service } from './service';
...

uploadFile () {
 ..
 service.uploadFile(file, result => {
  
 }, err => {

 });
}

To use this service, I need to develop a server side API called ‘/upload’. In the server application, edit server.js and add this code:

app.post('/upload', function(req, res) {

});

However, to work with the uploaded file in req body, I need to use an NPM package called ‘multer’. Install the package with this command:
npm install --save multer

Once it is installed, use it in server.js:

var multer = require('multer');
var upload = multer({dest: 'uploads/'});
var uploadFile = require('./upload');

...
app.post('/upload', upload.single('file'), function(req, res, next) {
  var file = req.file;
  upload.uploadFile(file.filename, db, function(){

  }, function(err) {

  });
});

This puts the file property on req object. To take this file and add all its contents to the database, we need to write some code for converting each line in CSV into a JSON object and then save it to the database. As the file can be large, I am using streams for this process. It will take less memory on the server and also code becomes extremely concise.

For this, let’s create a new file called upload.js and use it for this processing. In upload.js:

var fs = require('fs');
var split = require('split3');
var through = require('through3');

var upload = function(fileName, db, callback, errorCallback) {
  var fileHeaders = [];
  var stream = fs.createReadStream(__dirname + '/uploads/' + fileName)
   .pipe(split())
   .pipe(through(function(buffer, encoding, next){
      if (fileHeaders.length === 0) {
        fileHeaders = buffer.toString().split(',');
      } else {
        var fields = buffer.toString().split(',');
        var obj = fileHeaders.reduce(function  (acc, curr, index) {
          var newObj = {};
          newObj[curr] = fields[index];
          return Object.assign(acc, newObj);
        }, {});

        this.push(JSON.stringify(obj));
        next();
    }, function (done) {
      done();
    });
}

module.exports = {
  upload: upload
}

For this code to work, we need to install NPM modules split3 and through3. ‘split’ module takes a stream and converts it into a stream after splitting by desired characters. ‘through3’ module takes a stream and transforms it using a callback that we can write. In the code above, we have created a read stream from the file in uploads folder using fs.createReadStream and then pipe it to ‘split’ to split it based on a new line. Then we transformed it using ‘through’ to convert each line coming in the stream to a JSON object. The result is a stream of JSON objects that are to be added to the database.

To add these objects to the database, we subscribe to the stream using ‘on’ method and ‘data’ event. We also handle any error in the stream using ‘error’ event:

stream.on('data', function(data) {
  database.addProduct(db, data, function(result {
    callback(result);
  }, function(err){
    console.log(err);
  });
});

stream.on('error', function(error) {
  errorCallback(error);
});

Now, lets implement addProduct in database.js file:

var addProduct = function(db, data, callback, errorCallback) {
  var collection = db.collection('MainList')

  collection.insert(data, function(err, result) {
    if (err) {
      errorCallback(err);
      return;
    }

    callback(result);
  });
}

module.exports = {
  addProduct: addProduct
}

This completes the implementation of upload component of my application.