DevOps - Building a minimalistic CI/CD with Nodejs

Guilherme Rossato, December 2023

TLDR; In this post I implement a very simple CI/CD process for web apps that executes a deployment process and restarts the production app instance that runs on a remote server automatically when changes are submit to its private repository. I walk through the steps so that you can either learn how it works or implement your own self-hosted process and adjust it to your needs. The code for the deployment process is stored on this public github repository.

A CI/CD process helps developers by automating the testing, building and deployment process for web apps. The standard industry practice is more complicated than what we will be implementing but this will give you an idea of how it works, and perhaps you can use and customize this solution for your needs.

Modern web apps built with typescript often have the following pipeline steps on their deployment process:

Install dependencies: npm install / npm ci / yarn
Build the project: npm run build / yarn build
Test the project: npm run build / yarn test
Server restart: npm run start / yarn start

Cloud providers

This deployment process is very common and most cloud providers provide the tools for you to setup your own pipeline process in their infrastructure:

Amazon: AWS Code Pipeline
Google: Cloud Deploy
Microsoft: Azure Pipelines

Most are reliable, provide observability with dashboards, easy access to streams of logs, pipeline restarting, failure notifications, integrations, are easily scalable, etc. As downsides they are a bit hard to configure (as each of them reivented the wheel by themselves) and more expensive the faster you want your deployment to run, but they are used by pretty much everyone in one way or another.

Creating our own CI/CD process will give us maximum flexibility and efficiency, but most importantly it will allow us to learn what goes on internally on these tools and what they are abstracting.

As a bonus we can also privately our apps entirely as no external service will need to receive the source of our code, only the developer that is correctly configured to access our remote server can clone and push changes.

We will suffer on observability as we will not implement a dashboard to display the status of our pipeline jobs, this is left as an exercise, log files will just be freely written on the remote server as an example, but you should implement your own ways of keeping track of when deployment processes fail, such as sending an external notification like an email.

How this is supposed to be used

A developer clones the repository from the production server to his machine, develops locally, and eventually pushes his changes back to the remote server through git.
The push triggers a git hook and the remote server creates a new folder with the updated contents of the repository for the new deployment to start.
The pipeline for the deployment process is executed and its steps are processed.
The previous instance of the app is stopped and a new one is started on the new version where the deployment process was successful.

You could change this setup to separate the repository storage from the production server, but in an effort to simplify this tutorial I will walk you through the setup as if they are the same.

Setup

You should test whether or not you have SSH access to the remote machine where the repository will be stored:

ssh [server-username]@[ip-or-host-of-server]

This way when you create your repository at a directory like /home/user/my-app.git you will be able clone the repository from the client with this command:

git clone ssh://[server-username]@[ip-or-host-of-server]:[ssh-port]/home/user/my-app.git

Enter the remote server through shell and begin by creating a bare git repository directory to hold the git metadata. The name of the folder should end with .git and preferably contain your project name, the location of the folder isn't important. Assuming your project is called my-app and you want to store it at the /home/user directory, you would use these commands:

mkdir /home/user/my-app.git
cd /home/user/my-app.git
git init --bare

Then create a post-update hook inside the hooks folder and enable its execution permission:

touch ./hooks/post-update
chmod +x ./hooks/post-update

This script will be executed by git after changes have been applied to the repository. We should use it to trigger the deployment process to be executed assyncronously by another process. Let's implement a node script that does the following:

Creates a folder to process the deployment for the new instance
Clones the updated repository contents into that folder
Schedules the deployment processing of the new instance

Open the ./hooks/post-update file and put the following script to perform these steps:

#!/bin/node

// ./hooks/post-update

const fs = require("fs");
const cp = require("child_process");

// The port the app instance manager listens for deployment pipeline requests
const appInstanceManagerPort = 19861;

// Generate an instance id to uniquely identify the pipeline and instance
const newInstanceId = new Date()
  .toISOString()
  .replace("T", "_")
  .replace(".", "_")
  .replace(/[^(\d|_)]/g, "");

// Get the updated branch ref from arguments
const branchRef = process.argv[2] || "refs/head/master";

(async () => {
  // Create a folder to hold the new repository contents
  const deployFolder = `./instances/${newInstanceId}`;
  await fs.promises.mkdir(deployFolder, { recursive: true });

  // Clone the updated repositoy contents into the folder
  const command = ["git", `--work-tree=${deployFolder}`, "checkout", "-f"];
  await execCommandAsync(command, process.cwd(), (data) =>
    process.stdout.write(data)
  );

  // Sends the deployment request to the app instance manager
  await sendDeploymentRequest({ branchRef, deployFolder });

  console.log(`Successfully scheduled deployment pipeline "${newInstanceId}"`);
})().catch((err) => {
  console.log(`Failed to schedule deployment pipeline "${newInstanceId}":`);
  console.error(err);
  process.exit(1);
});

// Method to send the deployment request to the app instance manager
async function sendDeploymentRequest(body) {
  const response = await fetch(`http://localhost:${appInstanceManagerPort}/`, {
    method: "POST",
    body: JSON.stringify(body),
  });
  const text = await response.text();
  if (!response.ok) {
    throw new Error(
      `App instance manager failed: ${text ? text : "No error message"}`
    );
  }
  if (text) {
    console.log(`App instance manager response: ${text}`);
  }
}

// Method to execute a process assyncronously
async function execCommandAsync(command, cwd, onData) {
  await new Promise((resolve, reject) => {
    const child = cp.spawn(command[0], command.slice(1), {
      cwd: cwd,
      stdio: ["ignore", "pipe", "pipe"],
    });
    child.stderr.on("data", onData);
    child.stderr.on("data", onData);
    child.on("error", reject);
    child.on("exit", (code) => {
      if (code === 0) {
        resolve();
      } else {
        reject(
          new Error(
            `Command "${command.join(" ")}" exited with error code ${code}`
          )
        );
      }
    });
  });
}

Then we must create the app manager to process deployments and run our app instance:

touch ./app-instance-manager.js
chmod +x ./app-instance-manager.js

It should do the following on start:

Create an internal server to receive deployment requests from the post-update script
Process deployment requests and restart the app when pipelines finishes successfully
If the instance server is not running, start it on the last place it was deployed the last time the script executed

So we add the following script to it:

#!/bin/node

// ./app-instance-manager.js

const fs = require("fs");
const http = require("http");
const path = require("path");
const cp = require("child_process");

// The port the app instance manager listens for deployment pipeline requests
const appInstanceManagerPort = 19861;

let child;
let deployFolder;

(async () => {
  // Start the server to receive deployment requests
  await startDeploymentRequestServer();

  // Attempt to start the previous instance if it was running previously
  if (fs.existsSync("./current-instance-folder.txt")) {
    try {
      deployFolder = await fs.promises
        .readFile("./current-instance-folder.txt", "utf-8")
        .trim();
      await startInstanceChildProcess(deployFolder);
    } catch (err) {
      console.log(`Could not start previous instance: ${err.message}`);
    }
  }
})().catch((err) => {
  console.log(`Failed to start app instance manager:`);
  console.error(err);
  process.exit(1);
});

// Method to write logs to a file
async function logToFile(folder, logName, message) {
  process.stdout.write(message);
  await fs.promises.appendFile(
    path.resolve(folder, `${logName}.log`),
    message,
    "utf-8"
  );
}

// Method to start the instance child process
async function startInstanceChildProcess() {
  const currentDeployFolder = deployFolder;
  child = cp.spawn("yarn", ["start"], {
    cwd: currentDeployFolder,
    stdio: ["ignore", "pipe", "pipe"],
  });
  await logToFile(
    currentDeployFolder,
    "production",
    `[${new Date().toISOString()}] [Process Manager] Starting app instance process (pid ${JSON.stringify(
      child.pid
    )}) at cwd "${currentDeployFolder}"\n`
  );
  child.stderr.on("data", (data) =>
    logToFile(currentDeployFolder, "production", data.toString("utf-8"))
  );
  child.stdout.on("data", (data) =>
    logToFile(currentDeployFolder, "production", data.toString("utf-8"))
  );
  child.on("error", (err) => {
    logToFile(
      currentDeployFolder,
      "production",
      `[${new Date().toISOString()}] [Process Manager] App process failed to start: ${
        err.message
      }\n`
    );
    child = null;
  });
  child.on("exit", (code) => {
    logToFile(
      currentDeployFolder,
      "production",
      `[${new Date().toISOString()}] [Process Manager] App exited with exit code ${code}\n`
    );
    child = null;
  });
}

// Method to process a deployment request and execute the pipeline
async function onReceiveDeploymentRequest(newDeployTarget) {
  let stepName;
  let command;
  stepName = "Installing dependencies";
  command = ["yarn"];
  try {
    await logToFile(
      newDeployTarget,
      "deployment",
      `[${new Date().toISOString()}] Starting deployment process at "${newDeployTarget}"\n`
    );
    await logToFile(
      newDeployTarget,
      "deployment",
      `[${new Date().toISOString()}] ${stepName} begin\n`
    );
    await execCommandAsync(command, newDeployTarget, (data) =>
      logToFile(newDeployTarget, "deployment", data.toString("utf-8"))
    );
  } catch (err) {
    await logToFile(
      newDeployTarget,
      "deployment",
      `[${new Date().toISOString()}] ${stepName} failed: ${err.message}\n`
    );
    return;
  }
  stepName = "Building app";
  command = ["yarn", "build"];
  try {
    await logToFile(
      newDeployTarget,
      "deployment",
      `[${new Date().toISOString()}] ${stepName} begin\n`
    );
    await execCommandAsync(command, newDeployTarget, (data) =>
      logToFile(newDeployTarget, "deployment", data.toString("utf-8"))
    );
  } catch (err) {
    await logToFile(
      newDeployTarget,
      "deployment",
      `[${new Date().toISOString()}] ${stepName} failed: ${err.message}\n`
    );
    return;
  }
  stepName = "Testing app";
  command = ["yarn", "test"];
  try {
    await logToFile(
      newDeployTarget,
      "deployment",
      `[${new Date().toISOString()}] ${stepName} begin\n`
    );
    await execCommandAsync(command, newDeployTarget, (data) =>
      logToFile(newDeployTarget, "deployment", data.toString("utf-8"))
    );
  } catch (err) {
    await logToFile(
      newDeployTarget,
      "deployment",
      `[${new Date().toISOString()}] ${stepName} failed: ${err.message}\n`
    );
    return;
  }
  stepName = child ? "Restarting app" : "Starting app";
  try {
    await logToFile(
      newDeployTarget,
      "deployment",
      `[${new Date().toISOString()}] ${stepName} begin\n`
    );
    // Stop previous instance
    if (child) {
      child.kill();
      for (let i = 0; i < 50 && child !== null; i++) {
        await new Promise((resolve) => setTimeout(resolve, 100));
      }
      if (child !== null) {
        throw new Error(
          `Could not kill previously deployed app${
            deployFolder ? ` running at ${deployFolder}` : ""
          }`
        );
      }
    }
    // Saves new instance path
    fs.writeFileSync("./current-instance-folder.txt", newDeployTarget, "utf-8");
    deployFolder = newDeployTarget;
    await startInstanceChildProcess();
  } catch (err) {
    await logToFile(
      newDeployTarget,
      "deployment",
      `[${new Date().toISOString()}] ${stepName} failed: ${err.message}\n`
    );
    return;
  }
}

// Method to start the http server to receive deployment requests
async function startDeploymentRequestServer() {
  await new Promise((resolve, reject) => {
    // Method called by the server when a request is received
    function onServerRequest(req, res) {
      if (req.method !== "POST") {
        return res.end(
          `App instance ${
            child ? "is running" : "is not running"
          }. Last deployment target was ${deployFolder || "null"}`
        );
      }
      const chunks = [];
      req.on("data", (d) => chunks.push(d));
      req.on("end", () => {
        try {
          const job = JSON.parse(Buffer.concat(chunks).toString("utf-8"));
          if (
            !job.deployFolder ||
            !fs.existsSync(job.deployFolder) ||
            !fs.statSync(job.deployFolder).isDirectory()
          ) {
            throw new Error(
              'Missing or invalid "deployFolder" property on request'
            );
          }
          // No need to wait on deployment response
          onReceiveDeploymentRequest(job.deployFolder);
          res.end();
        } catch (err) {
          res.writeHead(500);
          res.end(err.stack);
        }
      });
    }

    const server = http.createServer(onServerRequest);

    server.on("error", (err) => {
      reject(
        new Error(
          `App instance manager could not start HTTP server at http://localhost:${appInstanceManagerPort}/: ${err.message}`
        )
      );
    });

    server.listen(appInstanceManagerPort, () => {
      console.log(
        `App instance manager started HTTP server at http://localhost:${appInstanceManagerPort}/`
      );
      resolve();
    });
  });
}

// Method to execute a process assyncronously
async function execCommandAsync(command, cwd, onData) {
  await new Promise((resolve, reject) => {
    const child = cp.spawn(command[0], command.slice(1), {
      cwd: cwd,
      stdio: ["ignore", "pipe", "pipe"],
    });
    child.stderr.on("data", onData);
    child.stderr.on("data", onData);
    child.on("error", reject);
    child.on("exit", (code) => {
      if (code === 0) {
        resolve();
      } else {
        reject(
          new Error(
            `Command "${command.join(" ")}" exited with error code ${code}`
          )
        );
      }
    });
  });
}

This script will be the process manager for our app (it will run npm run start when deployment succeeds) so it must be kept executing at all times. I recommend adding it to the cronjob as a reboot script.

For now let's just run it on the background by using the nohup utility:

nohup node ./app-instance-manager.js &

A nohup.out file will be created with logs for all the deployment processes, the deployment logs for each instance will also be written to ./instances/yyyymmdd_hhiiss_zzz/deployment.log and the output of the running instance will be written to a ./instances/yyyymmdd_hhiiss_zzz/production.log.

To verify if this is working correctly let's test it by cloning into a temp-repo folder and submit a minimally testable project:

git clone . ./temp-repo # copies the git bare to a "temp-repo" folder
cd temp-repo # enters the folder
npm init -y # creates a package.json file

Now change the new package.json file to add the scripts executed by the deployment process:

{
  "name": "temp-repo",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Test script\"",
    "build": "echo \"Build script\"",
    "start": "echo \"Start script\""
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}

Then submit this change to the git bare repository from the terminal:

git add .
git commit -m "First commit"
git push

The output should contain a line like this:

remote: Successfully scheduled deployment pipeline "20231211_083615_311"

Verify if you can read the deployment logs for the new instance by running this command:

cat ../instances/*/deployment.log

You should see something like this:

[2023-12-11T08:36:15.356Z] Starting deployment process at "./instances/20231211_083615_311"
[2023-12-11T08:36:15.360Z] Installing dependencies begin
[2023-12-11T08:36:15.620Z] Building app begin
[2023-12-11T08:36:15.865Z] Testing app begin
[2023-12-11T08:36:16.107Z] Starting app begin

And the production logs should be accessible by running this:

cat ../instances/*/production.log

Which should have output something like this:

[2023-12-11T08:36:16.109Z] [Process Manager] Starting app process (pid 6839) at cwd "./instances/20231211_083615_311"
$ echo "Start script"
Start script
Done in 0.05s.
[2023-12-11T08:36:16.348Z] [Process Manager] App exited with code 0

This means our app passed through the build, test, and run steps, and then exited as it had nothing to do.

You can now go to your local machine and attempt to clone the repository to replace its contents with a real app: From your local machine use the aforementioned clone command through ssh:

git clone ssh://[server-username]@[ip-or-host-of-server]:[ssh-port]/home/user/my-app.git

Change the contents and push, then enter the server to follow the deployment process logs, if everything went right a new version will be processed, deployed, and will be left running.

If anything went wrong you must check the log files inside the remote server, you can even implement scripts to route them in easier ways since the deployment is handled assyncronously. When a step like build or the test fail the logs will tell you why.

Speed up deployment

Installing dependencies on real apps and performing checkout are time-consuming steps, so copying the contents of the previous instance is a simple and good optimization.

To do that just alter the post-update script to copy it before the checkout is executed, like this:

#!/bin/node

// ./hooks/post-update

const fs = require("fs");
const cp = require("child_process");

// The port the app instance manager listens for deployment pipeline requests
const appInstanceManagerPort = 19861;

// Generate an instance id to uniquely identify the pipeline and instance
const newInstanceId = new Date()
  .toISOString()
  .replace("T", "_")
  .replace(".", "_")
  .replace(/[^(\d|_)]/g, "");

// Get the updated branch ref from arguments
const branchRef = process.argv[2] || "refs/head/master";

(async () => {
  // Assign a folder to hold the new repository contents
  const deployFolder = `./instances/${newInstanceId}`;

  const previous = (await fs.promises.readdir("./instances")).sort().pop();
  if (previous) {
    // Copies the previous instance repository contents to speed up the process
    await execCommandAsync(
      ["cp", "-rf", `./instances/${previous}`, deployFolder],
      process.cwd(),
      (data) => process.stdout.write(data)
    );
  } else {
    // Create a folder to hold the new repository contents
    await fs.promises.mkdir(deployFolder, { recursive: true });
  }

  // Clone the updated repository contents into the folder
  const command = ["git", `--work-tree=${deployFolder}`, "checkout", "-f"];
  await execCommandAsync(command, process.cwd(), (data) =>
    process.stdout.write(data)
  );

  // Sends the deployment request to the app instance manager
  await sendDeploymentRequest({ branchRef, deployFolder });

  console.log(`Successfully scheduled deployment pipeline "${newInstanceId}"`);
})().catch((err) => {
  console.log(`Failed to schedule deployment pipeline "${newInstanceId}":`);
  console.error(err);
  process.exit(1);
});

// Method to send the deployment request to the app instance manager
async function sendDeploymentRequest(body) {
  const response = await fetch(`http://localhost:${appInstanceManagerPort}/`, {
    method: "POST",
    body: JSON.stringify(body),
  });
  const text = await response.text();
  if (!response.ok) {
    throw new Error(
      `App instance manager failed: ${text ? text : "No error message"}`
    );
  }
  if (text) {
    console.log(`App instance manager response: ${text}`);
  }
}

// Method to execute a process assyncronously
async function execCommandAsync(command, cwd, onData) {
  await new Promise((resolve, reject) => {
    const child = cp.spawn(command[0], command.slice(1), {
      cwd: cwd,
      stdio: ["ignore", "pipe", "pipe"],
    });
    child.stderr.on("data", onData);
    child.stderr.on("data", onData);
    child.on("error", reject);
    child.on("exit", (code) => {
      if (code === 0) {
        resolve();
      } else {
        reject(
          new Error(
            `Command "${command.join(" ")}" exited with error code ${code}`
          )
        );
      }
    });
  });
}

Conclusion

That's enough to have a personal CI/CD on your own remote machine that automatically deploys new versions when pushes are submitted. It is very crude and has a lot of room for improvement, but it should have given you the overall strategy people have been using for years before cloud providers came along to automate that process.

It should be noted that the observability of this setup is terrible: developers are not informed when pipelines fail and the only way is to check the logs inside each instance. This can be improved in many ways, like implementing an externally accessible dashboard that renders the pipeline information, but this can become a very time-consuming project so I didn't work on it.

The fact that the app-instance-manager has two responsabilities (processes deployment and restarting of instances) is not a good practice and this can easily be fixed by separating it into two scripts. I choose not to do that as I was focused on making a simpler setup.

Implementing a notification to be sent when the deployment fails, maybe even to the developer that triggered the deployment, is a great addition, and helps me personally, so if you are using this setup I.

I have also implemented a way to get a notification when a deployment fails. It is a great addition and helps me greatly as I can just assume it worked otherwise.

Thanks for reading.