Learn DevOps - Implementing CI/CD (using Node.js)

Guilherme Rossato, July 2024

TLDR; I implement a simple CI/CD process for web apps that processes deployment and restarts the app instance automatically when changes are sent to the privately stored project repository. Learn why it is useful, what it abstracts, and how it works.
Source code is publicly available at this Github repository.

A CI/CD process helps developers by automating the testing, building and deployment process for web apps. The standard industry practice is more complicated than what we will be implementing but this will give you an idea of how it works, and perhaps you can use and customize this solution for your needs.

Modern web apps start with a simple deployment pipeline. Node.js apps for example:

Install dependencies:

npm install / npm ci / yarn

Build the project:

npm run build / yarn build

Test the project:

npm run build / yarn test

Server restart:

npm run start / yarn start

Most cloud providers provide the tools for you to setup your own pipeline process in their infrastructure, here are some of them:

Github

Amazon

Google

Microsoft

They are reliable, provide observability in dashboards, states, actions, notifications, integrations, etc, and the downsides are that their learning curve might be steep, and each of them invented their configuration process, but the most improtant is that they are more expensive the faster you want your deployment to run.

Objectives

Creating the CI/CD process from the ground up gives a deeper understanding of how they work and provides you with more flexibility and efficiency. Learning what goes on internally on these tools will also help you debug them, configure them, and value what they are abstracting away.

We can also use it in our own projects so that no external services are needed and our source code (and project credentials) are not stored in a third-party server. I am using this deployment script to handle the CI/CD of this personal blog (Next.js app).

Operation

A developer clones the repository from the production server to his machine
The developer sends his changes to the remote server through git push.
The push triggers a git hook and the remote server creates a new folder with the updated contents of the repository for the new deployment to start.
The pipeline for the deployment process is executed and its steps are processed.
The previous instance of the app is stopped and a new one is started on the new version where the deployment process was successful.

In an effort to simplify this I will assume the repository and the intance process are in the same machine.

Setup

If you are acessing the remote server through SSH, enter it with your ssh command (ssh [user]@[host]) to start:

Assuming your project will be stored at /home/user/my-app, create and init the project folder as a bare repository:

git init --bare /home/user/my-app.git
cd /home/user/my-app.git

Scheduling

Initialize the post-update hook inside the hooks folder at the new repository directory and enable its execution permission:

touch ./hooks/post-update
chmod +x ./hooks/post-update

This script is executed by git when changes have been pushed to the repository. We will use it to initialize the deployment but not process it as it would hold the git push command (sent by the developer).

The script is simple and should do these steps:

Create a new folder for the new deployment
Clone the new repository contents into that folder
Schedules the deployment process of the new instance

Lets update the ./hooks/post-update to execute with node (assuming your node is installed at /bin/node). Open the file with nano ./hooks/post-update and add this:

#!/bin/node
// ./hooks/post-update

// The server that processes deployment requests for this project
const deployProcessorPort = "19861";
const deployProcessorUrl = `http://127.0.0.1:${deployProcessorPort}/`;

const fs = require("fs");
const path = require("path");
const cp = require("child_process");

// The updated branch ref (sent by git)
const branchRef = process.argv[2] || "refs/heads/master";

// Create an instance id with the current date
let instanceId = new Date()
  .toISOString()
  .replace("T", "_")
  .replace(".", "_")
  .replace(
    /[^(\d|_)]/g,
    ""
  )(async () => {
    const cwd = process.cwd();
    console.log(
      `Initializing post-update (ref ${JSON.stringify(
        branchRef
      )}) at ${JSON.stringify(cwd)}`
    );
    // Get the last commit message
    let commitMessage = "";
    try {
      await execCommandAsync(
        ["git", "log", '--format="%s"', "-1"],
        cwd,
        (data) => {
          commitMessage = commitMessage + data.toString("utf-8");
        }
      );
      if (commitMessage) {
        console.log("Current commit:", commitMessage);
      }
    } catch (err) {
      console.log("Could not get commit message:", err.message);
    }
    const words = commitMessage
      .toLowerCase()
      .replace(/\W/g, " ")
      .split(" ")
      .filter((a) => a.length);
    const prefix = words
      .map((w) => w.substring(0, 4))
      .join("")
      .substring(0, 10);
    if (prefix) {
      instanceId = prefix + instanceId;
    }
    // Create a folder to hold the new repository contents
    const relativePath = `./instances/${instanceId}`;
    console.log(
      `Creating new instance ${JSON.stringify(instanceId)} at ${JSON.stringify(
        relativePath
      )}`
    );
    await fs.promises.mkdir(relativePath, { recursive: true });
    // Copy files from the current instance to speed up dependency installation
    const currFilePath = "./deployment/current-instance-folder.txt";
    try {
      const appPath = await fs.promises.readFile(currFilePath, "utf-8");
      console.log(`Copying current instance from ${JSON.stringify(appPath)}`);
      await execCommandAsync(
        ["cp", "-rf", path.resolve(appPath), relativePath],
        path.resolve(appPath),
        (data) => process.stdout.write(data)
      );
    } catch (err) {
      console.log(
        `Could not copy from current instance: ${
          err.code === "ENOENT" ? "(path file not found)" : err.message
        }`
      );
    }

    // Checkout the new version of the project
    const command = ["git", `--work-tree=${relativePath}`, "checkout", "-f"];
    console.log(
      `Executing ${JSON.stringify(command)} at ${JSON.stringify(cwd)}`
    );
    await execCommandAsync(command, cwd, (data) => process.stdout.write(data));

    // Sending deployment process data
    const targetPath = path.resolve(cwd, relativePath);
    console.log(
      `Requesting deployment process to ${JSON.stringify(
        deployProcessorUrl
      )} for ${JSON.stringify(targetPath)}`
    );
    const response = await sendPostRequest(deployProcessorUrl, {
      cwd,
      branchRef,
      deployFolder: targetPath,
    });
    if (response) {
      console.log(`Deployment processor response: ${response}`);
    }
    console.log(
      `Successfully scheduled deployment pipeline for "${instanceId}"`
    );
  })()
  .catch((err) => {
    console.log(`Failed to schedule deployment pipeline "${instanceId}":`);
    console.log(err);
    process.exit(1);
  });

// Method to send a POST with JSON payload
async function sendPostRequest(url, obj) {
  const response = await fetch(url, {
    method: "POST",
    body: JSON.stringify(obj),
  });
  const text = await response.text();
  if (!response.ok) {
    throw new Error(
      `Request failed with status ${response.status}: ${text.substring(0, 160)}`
    );
  }
  return text || "";
}

// Method to execute a child process asyncronously
async function execCommandAsync(command, cwd, onData) {
  await new Promise((resolve, reject) => {
    const child = cp.spawn(command[0], command.slice(1), {
      cwd,
      stdio: ["ignore", "pipe", "pipe"],
    });
    child.stderr.on("data", onData);
    child.stderr.on("data", onData);
    child.on("error", reject);
    child.on("exit", (code) => {
      if (code === 0) {
        return resolve();
      }
      reject(
        new Error(
          `Process "${command.join(" ")}" exited with error code ${code}`
        )
      );
    });
  });
}

When executed it will create a new folder inside ./instances/ and initialize it with the new version of the project, copying the files from the previous deployment if it exists, and send a JSON request to the server specified at deployProcessorUrl (http://127.0.0.1:19861/) to start processing the instance.

Deployment Handler

Let's create a folder for deployment and add a script to handle deployment:

mkdir /home/user/my-app.git/deployment
cd /home/user/my-app.git/deployment
touch ./node-deploy.cjs
chmod +x ./node-deploy.cjs

Create a script that will receive deployment requests from other processes inside the new deployment folder with nano node-deploy.cjs:

#!/bin/node
// ./deployment/node-deploy.cjs

// The server that processes deployment requests for this project
const deployProcessorPort = "19861";
const deployProcessorUrl = `http://127.0.0.1:${deployProcessorPort}/`;

// Deployment pipeline steps executed before starting the app instance process
const scriptSteps = [
  { name: "Installing dependencies", command: ["npm", "install"] },
  { name: "Testing app", command: ["npm", "run", "test"] },
  { name: "Building app", command: ["npm", "run", "build"] },
];

// The target server used to verify if it is still running
const instanceServer = "http://localhost:8089/";

const fs = require("fs");
const http = require("http");
const path = require("path");
const cp = require("child_process");

let processing = false;
let child;
let deployFolder;

(async () => {
  logPersist("Starting deploy processor");
  // Start the server to receive deployment requests
  await startDeploymentRequestServer();

  // Start the previous instance if it was running
  if (fs.existsSync("./current-instance-folder.txt")) {
    try {
      deployFolder = (
        await fs.promises.readFile("./current-instance-folder.txt", "utf-8")
      ).trim();
      await startInstanceChildProcess();
    } catch (err) {
      console.log(`Could not start previous instance: ${err.message}`);
    }
  }
})().catch((err) => {
  console.log(`Failed to execute deployment handler:`);
  console.log(err);
  process.exit(1);
});

async function isServerResponding(url) {
  try {
    await fetch(url);
    return true;
  } catch (err) {
    return false;
  }
}
// Method to persist logs in a file
function logPersist(message, folder = ".", fileName = "deploy") {
  message = `[${new Date().toISOString()}] ${message}${
    message.endsWith("\n") ? "" : "\n"
  }`;
  process.stdout.write(message);
  fs.appendFileSync(path.resolve(folder, `${fileName}.log`), message, "utf-8");
  return message.length;
}

// Method to start the instance child process
async function startInstanceChildProcess() {
  const appPath = deployFolder;
  if (!appPath || !fs.existsSync(appPath)) {
    logPersist(`App process cannot be started on: ${appPath}`);
    return;
  }
  await new Promise((resolve) =>
    setTimeout(resolve, 250 + Math.random() * 250)
  );
  logPersist(`Starting app instance process at "${appPath}"\n`);
  child = cp.spawn("npm", ["run", "start"], {
    cwd: appPath,
    stdio: ["ignore", "pipe", "pipe"],
  });
  let outc = 0;
  let errc = 0;
  child.stdout.on("data", (data) => {
    outc += logPersist(data.toString("utf-8"), appPath, "stdout");
  });
  child.stderr.on("data", (data) => {
    errc += logPersist(data.toString("utf-8"), appPath, "stderr");
  });
  let start = new Date();
  child.on("spawn", () => {
    start = new Date();
    logPersist(
      `App process spawned with pid ${
        child.pid
      } at "${appPath}" on ${start.toISOString()}`
    );
  });
  child.on("error", (err) => {
    logPersist(`App process failed to start: ${err.message}`);
    child = null;
  });
  child.on("exit", (code) => {
    const elapsed = new Date().getTime() - start.getTime();
    logPersist(
      `App instance process exited with code ${code} after running for ${(
        elapsed / 1000
      ).toFixed(1)} seconds at "${appPath}"`
    );
    if (outc + errc > 0) {
      logPersist(
        `The ${
          code === 0 || code === null ? "finishing" : "failing"
        } process wrote ${outc} on stdout and ${errc} on stderr`
      );
      if (code !== null && outc > 0) {
        const text = fs.readFileSync(
          path.resolve(appPath, "stdout.log"),
          "utf-8"
        );
        logPersist(
          `stdout ${text.length > 200 ? "tail:" : "data:"} ${JSON.stringify(
            text.substring(Math.max(0, text.length - 200))
          )}`
        );
      }
      if (code !== null && errc > 0) {
        const text = fs.readFileSync(
          path.resolve(appPath, "stderr.log"),
          "utf-8"
        );
        logPersist(
          `stderr ${text.length > 200 ? "tail:" : "data:"} ${JSON.stringify(
            text.substring(Math.max(0, text.length - 200))
          )}`
        );
      }
    } else {
      logPersist(
        `The ${
          code === 0 || code === null ? "finishing" : "failing"
        } process did not write to stdout nor stderr.`
      );
    }
    child = null;
  });
}

// Method to process a deployment request and execute the pipeline
async function onReceiveDeploymentRequest(newDeployTarget) {
  logPersist(
    `Starting deployment process at "${newDeployTarget}" with ${
      scriptSteps.length + 2
    } steps`
  );
  let stepName = "";
  for (const { name, command } of scriptSteps) {
    stepName = name;
    try {
      logPersist(`${stepName} starting`);
      logPersist(` > ${JSON.stringify(command)}`);
      await execCommandAsync(command, newDeployTarget, (data) =>
        logPersist(data.toString("utf-8"))
      );
      logPersist(`${stepName} ended`);
    } catch (err) {
      return `${stepName} failed: ${err.stack}`;
    }
  }
  try {
    stepName = child ? "Restarting app" : "Starting app";
    logPersist(`${stepName} starting`);
    // Stop previous instance
    if (child && child.kill) {
      logPersist(`Killing current app process forcefully at pid ${child.pid}`);
      const command = ["kill", "-9", child.pid];
      logPersist(` > ${JSON.stringify(command)}`);
      await execCommandAsync(command, newDeployTarget, (data) =>
        logPersist(data.toString("utf-8"))
      );
      for (let i = 0; i < 75 && child !== null; i++) {
        await new Promise((resolve) => setTimeout(resolve, 100));
      }
      await new Promise((resolve) => setTimeout(resolve, 2000));
      if (child) {
        throw new Error(
          `Failed to stop previous app process (pid ${child.pid}) at "${deployFolder}"`
        );
      }
    }
    if (instanceServer) {
      let active = await isServerResponding(instanceServer);
      if (active) {
        logPersist(
          `An instance server process is still listening to "${instanceServer}"...`
        );
      } else {
        logPersist(
          `There is no instance server process listening to "${instanceServer}"`
        );
      }
      for (let i = 0; i < 100 && active; i++) {
        await new Promise((resolve) => setTimeout(resolve, 100));
        active = await isServerResponding(instanceServer);
      }
      if (active) {
        throw new Error(
          `Failed to stop instance app (server at "${instanceServer}" is still responding)`
        );
      }
    }
    // Save the new instance path
    fs.writeFileSync("./current-instance-folder.txt", newDeployTarget, "utf-8");
    deployFolder = newDeployTarget;
    await startInstanceChildProcess();
    logPersist(`${stepName} ended`);
  } catch (err) {
    return `${stepName} failed: ${err.stack}`;
  }
  // Create a folder link to the instance
  stepName = "Creating link";
  try {
    logPersist(`${stepName} starting`);
    const fromPath = path.resolve(process.cwd(), "instance");
    if (fs.existsSync(fromPath)) {
      const command = ["rm", "-rf", fromPath];
      logPersist(` > ${JSON.stringify(command)}`);
      await execCommandAsync(command, process.cwd(), (data) =>
        logPersist(data.toString("utf-8"))
      );
      await new Promise((resolve) => setTimeout(resolve, 100));
    }
    const command = ["ln", "-s", fromPath, newDeployTarget];
    logPersist(` > ${JSON.stringify(command)}`);
    await execCommandAsync(command, process.cwd(), (data) =>
      logPersist(data.toString("utf-8"))
    );
    logPersist(`${stepName} ended`);
  } catch (err) {
    return `${stepName} failed: ${err.stack}`;
  }
}

// Method to start the http server to receive deployment requests
async function startDeploymentRequestServer() {
  await new Promise((resolve, reject) => {
    // Method called by the server when a request is received
    function onServerRequest(req, res) {
      if (req.method !== "POST") {
        return res.end(
          [
            "App Deployment Handler Server",
            `Instance ${
              child ? "is running (pid " + child.pid + ")" : "is not running"
            }`,
            `${child ? "Instance" : "Last"} path: ${deployFolder || ""}`,
          ].join(" - ")
        );
      }
      const chunks = [];
      req.on("data", (d) => chunks.push(d));
      req.on("end", async () => {
        try {
          const job = JSON.parse(Buffer.concat(chunks).toString("utf-8"));
          if (
            !job.deployFolder ||
            !fs.existsSync(job.deployFolder) ||
            !fs.statSync(job.deployFolder).isDirectory()
          ) {
            throw new Error(
              'Missing or invalid "deployFolder" property on request: ' +
                JSON.stringify(deployFolder)
            );
          }
          let out = "";
          if (processing) {
            await new Promise((resolve) => setTimeout(resolve, 1000));
            if (processing) {
              throw new Error("Deployment is already in progress");
            }
          }
          processing = true;
          const promise = onReceiveDeploymentRequest(job.deployFolder);
          const prefix = `Deployment "${path.basename(job.deployFolder)}`;
          promise.then((res) => {
            processing = false;
            if (!res) {
              return;
            }
            out = res ? JSON.stringify(res) : "";
            logPersist(`${prefix} result: ${out}`);
          });
          promise.catch((err) => {
            processing = false;
            out = err.message;
            logPersist(`${prefix} failed: ${out}`);
          });
          await new Promise((resolve) => setTimeout(resolve, 500));
          res.end(out);
        } catch (err) {
          res.writeHead(500);
          res.end(err.stack);
        }
      });
    }

    const server = http.createServer(onServerRequest);
    server.on("error", (err) => {
      reject(
        new Error(
          `Deployment processor failed to listen at ${deployProcessorUrl}: ${err.message}`
        )
      );
    });
    server.listen(deployProcessorPort, "127.0.0.1", () => {
      logPersist(
        `Deployment processor started server at ${deployProcessorUrl}`
      );
      resolve(server);
    });
  });
}

// Execute a child process asyncronously
async function execCommandAsync(command, cwd, onData) {
  await new Promise((resolve, reject) => {
    const child = cp.spawn(command[0], command.slice(1), {
      cwd: cwd,
      stdio: ["ignore", "pipe", "pipe"],
    });
    child.stderr.on("data", onData);
    child.stdout.on("data", onData);
    child.on("error", reject);
    child.on("exit", (code) => {
      if (code === 0) {
        return resolve();
      }
      reject(
        new Error(
          `Process "${command.join(" ")}" exited with error code ${code}`
        )
      );
    });
  });
}

The script runs our app with npm run start on startup and restarts it when a deployment pipeline finishes successfully. Because it handles our app process it must be kept in execution so I recommend adding it to the cronjob as a reboot script, which you can open with crontab -e and append this line:

@reboot cd /home/user/my-app.git/deployment && node /home/user/my-app.git/deployment/node-deploy.cjs

Let's run it on the background by using the nohup utility for now:

cd /home/user/my-app.git/deployment
nohup node node-deploy.cjs &

The deployment logs will be written at deploy.log and the output of the instance process will be written to either stdout.log or stderr.log.

Verify

Lets validate the process by manually cloning the project to a new instance folder and pushing to it:

cd /home/user/my-app.git
mkdir instances
git clone . ./instances/first
cd ./instances/first
npm init -y
touch index.js

You may also want to update package.json to configure the build, test, and start scripts that will be executed at the deployment process:

{
  "name": "my-app",
  "scripts": {
    "test": "echo \"Test script\"",
    "build": "echo \"Build script\"",
    "start": "echo \"Start script\""
  }
}

Submit the change to the repository to trigger the post-update hook:

git add .
git commit -m "Deploy test"
git push

The output should end with a line like this if everything went right:

remote: Successfully scheduled deployment pipeline for "..."

The new version will be processed, deployed, and executed. You will have to check the log files a when step fails. The end of the deploy.log file inside the deployment folder can be read with:

tail -n 100 -f /home/user/my-app.git/deployment/deploy.log

That's it. You can now configure the repository contents with your actual project and push updates to it to execute deployment.

Best practices

This post should have given you the overall strategy people have been using for years before cloud providers came along to automate that process and provided it as a service.

Although that's enough to have a personal CI/CD on your own remote machine to automatically deploy new versions when pushes are submitted, it is very crude and has a lot of room for improvement.

A good optimization is to separate the repository storage from where it executes (the production server) to offload the heavy work of deployment elsewhere to avoid making your production server slow.

It should be noted that the observability of this setup is not ideal: developers are not informed when pipelines fail and the only way is to check the logs inside each instance. This can be improved in many ways, like implementing an externally accessible dashboard that renders the pipeline information, but this can become a very time-consuming project so I didn't work on it.

The fact that the instance script has multiple responsabilities (processes deployment and restarting of instances) is not a good practice and this can easily be fixed by separating it into two scripts. I choose not to do that as I was focused on making a simpler setup.

I have also implemented a way to get a notification when a deployment fails in my own variation of the tool. It is a great addition.

Node Deployment

I decided to create a project to perform these configuration steps as I frequently need to create new projects to test them in production. The source is publicly available at this Github repository and it can be executed by downloading its main script and executing or with this fetch-eval command:

node -e "fetch('https://raw.githubusercontent.com/GuilhermeRossato/node-deployment/master/node-deploy.cjs').then(r=>r.text()).then(t=>new Function(t)()).catch(console.log))"

It guides the setup process and also allows for streaming of logs (--logs), retrieving status (--status) and starting / stopping the project process (--start / --stop) manually.

Thanks for reading.