Extending Remake: The Custom Backend Tutorial 😎

David Miranda

Jun 6, 2021 • 11 min read

As you’re building your web app, you may encounter times where you need to do more with your app than what Remake does for you.

If that’s the case, your app may benefit from extending Remake’s backend code.

The backend of Remake contains the JavaScript code that runs on your server, which is different from the frontend code that runs in your users' web browsers.

The backend code gives us the ability to integrate our server with other Node libraries and web APIs, allowing us greater control over Remake’s power, and opening up new app possibilities! ✨

Important: If you want to use Remake's built-in hosting service (i.e. with the command remake deploy), you won't be able to deploy your modified backend code. So, if you to work with modified backend code, you need to host Remake yourself.

How we'll customize Remake's backend

The project we will build is a statistics panel, which will give us some stats about how our web app is being used.

Note: As with all web apps, user data privacy is important, so remember to follow any applicable user data privacy laws. Privacy law compliance is a topic outside the scope of this tutorial, but do not forget that it is also a very important topic for all web app developers to understand!

After this tutorial, you should feel more comfortable extending Remake’s backend with your own backend code.

Want to skip to the code? All the source code from this tutorial

💡 If you get stuck and a Google search doesn’t get you the relevant information, feel free to swing by Remake's community chat on Discord and ask for help. We’re very friendly!

Setup

We’ll start with the default Remake project, which we’ll generate the normal way using the remake-cli utility.

In your terminal, create a new remake project using the following command:

npx remake create backend_project

When asked to pick a starter template, choose Default Starter.

Now inside the backend_project folder, we have our new project files.

You can make sure the application was generated correctly by changing directory into the backend_project folder and running npm run dev:

cd backend_project
npm run dev

Once the server starts, navigate to http://localhost:3000/ in your web browser while it’s running.

Since there’s not much there, let’s fill our app/pages/app-index.hbs file with the default todo list starter code:

<div object>
  <ul array key="todos" sortable>
    {{#for todo in todos}}
      <li 
        object 
        key:text="@innerText"
        edit:text
      >{{default todo.text "New todo item"}}</li>
    {{/for}}
  </ul>
  <button new:todo>Add Todo</button>
</div>

Refresh your page and you should see a basic todo list app after you create an account and log in.

Components

To extend Remake, we’ll first start with a look at _remake/main.js, and the files inside _remake/lib/ as well. In order to modify Remake, we first have to know a bit about how it works!

Express Server (main.js)

Remake’s backend uses the Node Express framework for routing, Passport.js for authentication, and json for storing data on the server. In the main.js file, you’ll find 2 function calls beneath a comment called “REMAKE CORE FRAMEWORK”, called initApiNew and initApiSave. Those are the ones we’ll be adding callbacks to in this tutorial.

API Endpoints

Remake’s frontend and backend communicate in two ways: initially when pages load, and also after the page is loaded, through endpoints. API endpoints are just URLs the backend provides for the frontend to call. The files that do this are within the _remake/lib folder. Don’t be afraid to open them up and look at what they do!

Here are the endpoints of interest to us:

/new/ endpoint
- Called when a new item is created by the user.
- Contained in init-api-new.js
/save/ endpoint
- Called when something is changed by the user.
- Contained in init-api-save.js

Callbacks

The easiest way to hook into Remake’s behavior on the backend would be to use callbacks. Unfortunately, Remake doesn’t yet have backend callbacks. So let’s add them right now!

Adding Callbacks

initApiNew

In the file _remake/lib/init-api-new.js, add the following code to the function definition of initApiNew:

export function initApiNew ({app}, callback) {

Inside this function there is an app.post call. We need to call our callback at the end of this function provided to app.post, right below the final res.json call:

if(callback != null)
  callback({
    app,
    user: {
      name: currentUser.details.username,
      email: currentUser.details.email
    },
    data
  });

This callback will cause Remake to provide us with the user details (name and email) and copy of the data when a user creates a new entry. We’re also passing the app itself as the first parameter, since it’s always useful to have.

Now in your _remake/lib/main.js, we’ll provide our newly changed code with our own callback to log the data mentioned above:

initApiNew({ app }, ({app, user, data}) => {
  console.log(“initApiNew callback data: ”, data);
});

Save and test the above code using npm run dev. Using your web app, add a new item to the todo list. In the server console, you’ll notice we now see the data logged.

Callbacks allow code separation, so we can more easily maintain our application’s custom behavior across remake updates. But they sometimes aren’t enough. So let’s explore other customization options.

initApiSave

Let’s do the same thing for the save endpoint in init-api-save.js.

export function initApiSave ({app}, callback) {

And underneath res.json({success: true}));, add the following code:

if(callback != null)
  callback({
    app,
    user: {
      name: currentUser.details.username,
      email: currentUser.details.email
    },
    data: {
      newData: existingData,
      oldData
    }
  });

We’ll also need to get a copy of the old data, since it’s changed in this function. On line 36, underneath the declaration of oldData, add:

let oldData = {...existingData};

I’ve chosen to provide not just the data in the callback, but the old data as well. This won’t be used in our application, but would be useful for purposes where you need to know the data that changed.

Let’s call the callback to check that it works. Do this in main.js:

initApiSave({ app }, ({app, user, data}) => {
  console.log(“initApiNew callback data: ”, data);
});

Since we are not using file uploads, we will not be adding a callback to the upload endpoint. But you could do so if you have a need for it.

Custom Backend Code

Combining what we know from the changes above, we can replace the 3 init function calls in main.js with initBackend({app}), a function we’ll write and store in backend.js, which we’ll put inside our app folder in the project directory.

First, in Remake’s main.js file, let’s import our backend. Beneath the other imports, add this:

let backend = null;
try {
  backend = require("../app/backend"); // optional
} catch (err) {
  if (err.code != “MODULE_NOT_FOUND”) {
    throw err;
  }
}

We’re going to make this import optional since requiring the backend file would break Remake if it wasn’t there.

Then, we’ll replace the 2 initApi calls in _remake/lib/main.js with this code. We’re also going to add an optional init function for our backend and have it run prior to the initRenderedRoutes function, which will allow us to override Remake’s default routes.

// REMAKE CORE FRAMEWORK
initUserAccounts({app});
initApiNew({app}, backend.onNew);
initApiSave({app}, backend.onSave);
initApiUpload({app});
if (backend.init != null) {
  backend.init({app});
}
initRenderedRoutes({app});

Now create a new file backend.js in your app folder with the contents. Notice we are moving our callbacks into this file:

const onNew = ({app, user, data}) => {
    console.log("onSave callback data:",
    "\nUser:", user.name,
    "\nData:", data.data
  );
};

const onSave = ({app, user, data}) => {
  console.log("onSave callback data:",
    "\nUser:", user.name,
    "\nOld Data:", data.oldData,
    "\nNew Data:", data.newData
  );
};

const init = ({app}) => {
  console.log("Custom backend initialized...");
}

const run = ({app}) => {
  console.log("Custom backend running...");
};

export { init, onNew, onSave, run };

We’ve also added another function, run, which we’ll call from Remake after it has started.

Inside of Remake’s main.js at the bottom of the the app.listen callback, call our run function:

app.listen(PORT, () => {
  console.log('\n');
  showConsoleSuccess(`Visit your Remake app: http://localhost:${PORT}`);
  showConsoleSuccess(`Check this log to see the requests made by the app, as you use it.`);
  console.log('\n');

  if (process.send) {
	process.send("online");
  }
  backend.run({app});
});

Now would be a good time to test your application the same way as before, using npm run dev. You’ll notice that our logging still works, but it has now been moved to our own file.

Our Project

Now that our backend code is running, it would be nice to do something useful besides logging, and this is where your creativity can come in!

For this tutorial, we’ll create a statistics page which will show us some useful information about how our web app is being used.

We’ll measure:

The most active users of our web app
The last time someone used our app

Using Callbacks to Measure User Activity

Let’s use our backend callbacks to record some information about user activity. Let’s count the activity of each user by counting the number of calls to new and save.

The counting is rather simple. It’s just the incrementing of a value that we’ll store in json. We’re also going to use Date.now() to get a timestamp of the last activity, which we’ll also store.

The Stats JSON Database

Tip: If you’re using JSON as a database, as Remake does, an important thing to remember is to be careful when using async/await or callbacks. If you’re reading in data, changing it, and writing data, there is a real chance the database gets opened and changed in another place in your code as well, opening up the real risk of data loss.

For this reason we’re using synchronous calls when working with the JSON and being careful to make sure node is not allowed to do a context switch while in the middle of working with a file. For more information about this concept, research transactional data and asynchronous programming.

For our project, we’re going to generate a new json file inside Remake’s json database folder called stats.json. Once populated, the json file will look something like this:

{
  "userActivity":{
    "[email protected]": {
      "activity": 12,
      "lastUseTimestamp": 1610088558699
    }
  },
  "lastUseTimestamp": 1610088558699,
  "lastUser": "[email protected]"
}

As you can see, we will have an entry per-user, with an activity count and a date of the last modification on the user account. This will allow us to see the most active users and the last time each user used the app.

We’re also tracking the last use timestamp out of all users and the last active user’s email.

Populating the Database

We’re going to make our new stats file create itself if it doesn’t exist, and fill in fields that don’t exist as we need them.

Since we’re counting both additions and modifications, we’ll need to add our code to both endpoints: new and save. Since this will involve the same code, let’s create a function for incrementing the count.

The following code examples in this section will occur inside a new function inside of backend.js called incrementUserActivity, which will take one object parameter containing the name and email of the user that did the activity.

Here is the stub:

const incrementUserActivity = ({name, email}) => {
  // our new code will go here...
};

Before we start writing this function, let’s add two imports and set the location for our stats.json file we’ll use at the top of the file:

const jsonfile = require("jsonfile");
const path = require("upath");
const statsFile = path.join(__dirname, "data/database/stats.json");

Reading JSON

The code below shows how to read a JSON file synchronously. If the file does not exist, an exception will be thrown, and if that happens, we will create it. Put the following code inside our new incrementUserActivity function.

  // open or create a stats.json file if it doesn't exist
  let stats = {};
  try {
    stats = jsonfile.readFileSync(statsFile);
  } catch (err) {
    if (err instanceof Error && err.code == “ENOENT”) {
      // file not found? create it!
      jsonfile.writeFileSync(statsFile, {});
    } else {
      // unknown error, rethrow
      throw err;
    }
  }

Set Default Stats

At the first run of our application, our stats.json won’t exist. The code we wrote previously will create it, but it will still start empty. Because of this, default values are needed to be filled in if they don’t exist.

Similarly, if an entry for a specific user doesn’t yet exist, we’ll create that too and set the initial activity counter to 0:

  // create user activity entry in stats.json if it doesn't exist
  if(stats.userActivity === undefined) {
    stats.userActivity = {};
  }
 
  if(stats.userActivity[email] === undefined) {
    stats.userActivity[email] = {};
  }
 
  if(stats.userActivity[email].activity === undefined) {
    stats.userActivity[email].activity = 0;
  }

Recording Our Data

As we talked about before, we’ll increment the activity counter stored in the current user and keep a timestamp of when it happened in two places, as well as the email of the last user to use the app.

  stats.userActivity[email].activity += 1;
  let timestamp = Date.now();
  stats.userActivity[email].lastUseTimestamp = timestamp;
  stats.lastUseTimestamp = timestamp;
  stats.lastUser = email;

Writing JSON

Writing our data into the stats.json file at the end of the function is simple, as it involves only one call:

  jsonfile.writeFileSync(statsFile, stats, {spaces:2});

The spaces option is provided so that the json is formatted with 2 spaces per “tab” so we can read it easily.

Calling Our Function

Our incrementUserActivity function should be called from both endpoints: new and save.

Here’s how we call our function:

const onNew = ({app, user, data}) => {
  incrementUserActivity(user);
};

const onSave = ({app, user, data}) => {
  incrementUserActivity(user);
};

Before you test, make sure to cause some activity to happen by adding or changing some items and viewing stats.json to see the changes.

The Statistics Page

To make our app display the stats, we need to add a new /stats/ route inside our backend’s init function.

Stats Route

The most quick and dirty approach to displaying the states would be logging them to the page and to the server console. Let’s try that first:

const init = ({app}) => {
  // Create the route for our page at /stats
  app.get("/stats", (req, res) => {
    res.set('Content-Type', 'text/plain'); // plain text page
    res.write("Stats:\n\n");
    try {
      const stats = jsonfile.readFileSync(statsFile);
      console.log(stats);
      res.write(JSON.stringify(stats, null, 2));
    } catch (err) {
      res.write("{}");
    }
    res.end();
  });
};

Now would be a good time to test. Firstly, make sure there is some activity in your todo list to log. Then, navigate to https://localhost:3000/stats/.

Restricting the Page

Since this isn’t a frontend tutorial, and only one user will see this page, we won’t be implementing any fancy template rendering. Instead we’ll render a very simple stats page which we’ll restrict to a single user called admin.

When you test this code, remember to be logged in as admin, or change the code below to match your username.

Begin by wrapping the route code in the following if statement. We’ll also set the page to plain text, since that’s how we’ll render the stats.

app.get("/stats", (req, res) => {
  res.set(“Content-Type”, “text/plain”); // plain text page
  if(req.isAuthenticated() && req.user.details.username === "admin") {
    // only admin user can see this...
  }else{
    res.status(403); // not authorized
    res.end();
  }
  // ...

Read Stats

Inside the above if statement, we’ll read the json stats similarly to how we did it before, and we’ll check to see the stats are empty:

  let stats = {};
  try {
    stats = jsonfile.readFileSync(statsFile);
  } catch (err) {
    // ignore file not found, otherwise throw
    if(!(err instanceof Error) || err.code !== "ENOENT")
      throw err;
  }
  if(stats == null || stats.userActivity == null) {
    res.write(“No stats recorded!”);
    res.end();
    return;
  }

This will read from our stats file and render a message to us if there are no stats recorded.

Calculate Stats

We can compute the most active user by looping through our stats object, checking each user’s activity, and keeping the most active user found and the activity level. We’ll also add up the total activity level.

  let mostActiveUser = null;
  let totalActivity = 0;
  let mostActivity = 0;
  if(stats.userActivity != null) {
    for(var email in stats.userActivity) {
      let user = stats.userActivity[email];
      if(user.activity != null) {
        if(user.activity > mostActivity) {
          mostActivity = user.activity;
          mostActiveUser = email;
        }
        totalActivity += user.activity;
      }
    }
  }

Render Stats

Here we’ll render each of the stats that we have available:

  if(totalActivity !== null)
    res.write("Total Activity: " + totalActivity + "\n");
  if(stats.lastUseTimestamp != null && stats.lastUser != null)
    res.write("Last Activity: " +
      (new Date(stats.lastUseTimestamp)).toString() +
      " by " +
      stats.lastUser + "\n"
    );
  if(mostActiveUser !== null)
    res.write("Most Active User: " +
      mostActiveUser +
      " (" + mostActivity + ")\n"
    );
  res.end();

We show the total activity, last activity, and the most active user.

Our stats page at https://localhost:3000/stats should look something like this:

Stats

Total Activity: 18
Last Activity: Fri Jan 08 2021 01:19:50 GMT-0800 (Pacific Standard Time)
  by [email protected]
Most Active User: [email protected] (14)

Conclusion

Want to view all the code at once? All the source code from this tutorial

If the code provided in this tutorial isn’t working for you, be sure to check the repository linked at the top of the page. This is where you’ll find the full example listing. If you’re looking for the stats, make sure your username is admin.

Thanks for reading this tutorial! I hope you learned that the Remake backend can be customized and expanded to open up new server-side possibilities. In the future, Remakes backend customization should become easier, but the integration should be similar. Be sure to watch Remake’s development closely as new things are being added frequently!

Bye for now! 😎