TypeScript Tutorial: Building a Simple Web-Based Code Syntax Highlighter

Written by

learnmodernjavascript

in

In the world of web development, code syntax highlighting is a must-have feature for any application where code is displayed. It makes code easier to read, understand, and debug. Imagine trying to decipher a block of code without any color-coding or formatting – a nightmare, right? This tutorial will guide you through building a simple, yet effective, web-based code syntax highlighter using TypeScript. We’ll explore the core concepts, step-by-step implementation, common pitfalls, and best practices, empowering you to integrate syntax highlighting into your projects.

Why Syntax Highlighting Matters

Syntax highlighting isn’t just about aesthetics; it significantly enhances the developer experience. Here’s why it’s crucial:

  • Readability: Color-coding and formatting make it easier to distinguish between different code elements (keywords, variables, comments, etc.).
  • Error Detection: Highlighting helps you quickly spot syntax errors or logical inconsistencies.
  • Code Understanding: It clarifies the structure and meaning of the code, making it easier to grasp complex logic.
  • Collaboration: When sharing code, highlighting ensures everyone can understand it, regardless of their preferred IDE or editor.

Core Concepts: Tokenization and Styling

At its heart, syntax highlighting involves two primary steps:

  1. Tokenization: This is the process of breaking down the code into meaningful units called tokens. These tokens represent keywords, identifiers, operators, strings, numbers, and comments. The tokenizer analyzes the code and identifies these tokens based on the language’s grammar.
  2. Styling: Once the code is tokenized, each token is assigned a specific style (color, font weight, etc.) based on its type. For example, keywords might be blue, strings might be green, and comments might be gray.

Setting Up the Project

Let’s set up a basic TypeScript project. We’ll use a simple HTML file, a TypeScript file for our logic, and a CSS file for styling.

1. Project Structure:

my-syntax-highlighter/
├── index.html
├── src/
│   └── highlighter.ts
├── style.css
├── tsconfig.json
└── package.json

2. Create `index.html`:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Simple Syntax Highlighter</title>
    <link rel="stylesheet" href="style.css">
</head>
<body>
    <textarea id="code" rows="10" cols="80">
// Sample JavaScript code
function greet(name) {
  console.log("Hello, " + name + "!");
}

greet("World");
    </textarea>
    <div id="highlighted-code"></div>
    <script src="./dist/highlighter.js"></script>
</body>
</html>

3. Create `style.css`:

.keyword {
    color: blue;
    font-weight: bold;
}

.string {
    color: green;
}

.comment {
    color: gray;
}

.number {
    color: purple;
}

4. Create `tsconfig.json`:

{
  "compilerOptions": {
    "target": "es5",
    "module": "commonjs",
    "outDir": "./dist",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true
  },
  "include": ["src/**/*"]
}

5. Initialize `package.json` (if you don’t have one):

npm init -y

6. Install TypeScript (if you haven’t already):

npm install typescript --save-dev

Implementing the Tokenizer in TypeScript

Now, let’s create the core logic in `src/highlighter.ts`. We’ll start with a basic tokenizer for JavaScript.

// Define token types
const enum TokenType {
    Keyword,
    String,
    Comment,
    Number,
    Identifier,
    Operator,
    Punctuation
}

// Define keywords
const keywords = [
    "function", "var", "let", "const", "if", "else", "for", "while", "return", "true", "false", "null", "console", "log"
];

interface Token {
    type: TokenType;
    value: string;
}

function tokenize(code: string): Token[] {
    const tokens: Token[] = [];
    let i = 0;

    while (i < code.length) {
        const char = code[i];

        if (/s/.test(char)) {
            i++; // Skip whitespace
            continue;
        }

        // Comments
        if (char === '/' && code[i + 1] === '/') {
            let j = i + 2;
            while (j < code.length && code[j] !== 'n') {
                j++;
            }
            tokens.push({ type: TokenType.Comment, value: code.substring(i, j) });
            i = j;
            continue;
        }

        // Strings
        if (char === '"' || char === "'") {
            let j = i + 1;
            while (j < code.length && code[j] !== char) {
                j++;
            }
            tokens.push({ type: TokenType.String, value: code.substring(i, j + 1) });
            i = j + 1;
            continue;
        }

        // Numbers
        if (/[0-9]/.test(char)) {
            let j = i;
            while (j < code.length && /[0-9]/.test(code[j])) {
                j++;
            }
            tokens.push({ type: TokenType.Number, value: code.substring(i, j) });
            i = j;
            continue;
        }

        // Keywords
        if (/[a-zA-Z]/.test(char)) {
            let j = i;
            while (j < code.length && /[a-zA-Z0-9_]/.test(code[j])) {
                j++;
            }
            const word = code.substring(i, j);
            if (keywords.includes(word)) {
                tokens.push({ type: TokenType.Keyword, value: word });
            } else {
                tokens.push({ type: TokenType.Identifier, value: word });
            }
            i = j;
            continue;
        }

        // Operators and Punctuation (simplified)
        if (/[+-*/=!(){}[];:,.]/.test(char)) {
            tokens.push({ type: TokenType.Operator, value: char }); // Or Punctuation, depending on the character
            i++;
            continue;
        }

        // If we reach here, it's an unrecognized character
        i++;
    }

    return tokens;
}

Explanation of the Code:

  • `TokenType` enum: Defines the different types of tokens we’ll identify.
  • `keywords` array: Contains a list of JavaScript keywords.
  • `Token` interface: Defines the structure of a token (type and value).
  • `tokenize` function: This is the core function. It iterates through the code character by character and identifies tokens based on predefined rules.
  • Whitespace Handling: Skips whitespace characters.
  • Comment Handling: Identifies single-line comments (`//`).
  • String Handling: Identifies strings enclosed in quotes (`”` or `’`).
  • Number Handling: Identifies numeric values.
  • Keyword and Identifier Handling: Checks if a word is a keyword or an identifier.
  • Operator and Punctuation Handling: Handles common operators and punctuation marks.
  • Unrecognized Characters: Skips unrecognized characters to prevent errors.

Implementing the Highlighter in TypeScript

Now, let’s create the highlighter function that takes the tokens and applies the styles.

function highlight(code: string, tokens: Token[]): string {
    let highlightedHtml = '';
    let currentIndex = 0;

    for (const token of tokens) {
        highlightedHtml += code.substring(currentIndex, code.indexOf(token.value, currentIndex));
        switch (token.type) {
            case TokenType.Keyword:
                highlightedHtml += `<span class="keyword">${token.value}</span>`;
                break;
            case TokenType.String:
                highlightedHtml += `<span class="string">${token.value}</span>`;
                break;
            case TokenType.Comment:
                highlightedHtml += `<span class="comment">${token.value}</span>`;
                break;
            case TokenType.Number:
                highlightedHtml += `<span class="number">${token.value}</span>`;
                break;
            case TokenType.Identifier:
                highlightedHtml += token.value;
                break;
            case TokenType.Operator:
                highlightedHtml += token.value;
                break;
            case TokenType.Punctuation:
                highlightedHtml += token.value;
                break;
        }
        currentIndex = code.indexOf(token.value, currentIndex) + token.value.length;
    }
    highlightedHtml += code.substring(currentIndex);
    return highlightedHtml;
}

Explanation of the Code:

  • `highlight` function: Takes the original code and the array of tokens as input.
  • Iterating Through Tokens: It iterates through each token.
  • Building HTML: For each token, it adds the appropriate HTML span with the corresponding CSS class to apply the styling.
  • Handling Identifiers and Operators: For identifiers and operators, it adds the value directly.
  • Returning Highlighted HTML: It returns the complete HTML string with the highlighted code.

Integrating the Highlighter with the HTML

Let’s add the code that uses the tokenizer and highlighter in `src/highlighter.ts`.

// Get the textarea and the output div
const codeTextArea = document.getElementById('code') as HTMLTextAreaElement;
const highlightedCodeDiv = document.getElementById('highlighted-code') as HTMLDivElement;

function updateHighlighting() {
    if (!codeTextArea || !highlightedCodeDiv) {
        return;
    }

    const code = codeTextArea.value;
    const tokens = tokenize(code);
    const highlightedHtml = highlight(code, tokens);
    highlightedCodeDiv.innerHTML = highlightedHtml;
}

// Initial highlighting
updateHighlighting();

// Add an event listener to the textarea for real-time updates
if (codeTextArea) {
    codeTextArea.addEventListener('input', updateHighlighting);
}

Explanation of the Code:

  • Get Elements: Retrieves the textarea and the `div` element from the HTML.
  • `updateHighlighting` function: This function gets the code from the textarea, tokenizes it, highlights it, and updates the `div` with the highlighted HTML.
  • Initial Highlighting: Calls `updateHighlighting` initially to display the highlighted code when the page loads.
  • Event Listener: Adds an `input` event listener to the textarea. This ensures that the code is highlighted dynamically as the user types.

Compiling and Running the Code

Now, let’s compile the TypeScript code and run the application.

1. Compile:

tsc

This command will compile the TypeScript code and generate `highlighter.js` in the `dist` directory.

2. Run:

Open `index.html` in your web browser. You should see the code in the textarea highlighted based on the CSS styles we defined.

Common Mistakes and How to Fix Them

Here are some common mistakes and how to avoid them:

  • Incorrect Tokenization: The tokenizer is the most critical part. Incorrect tokenization can lead to incorrect highlighting. Make sure to thoroughly test the tokenizer with various code snippets to catch edge cases.
  • Missing CSS Styles: If the code isn’t highlighted, check if the CSS styles are correctly linked and if the class names in the HTML and TypeScript match.
  • Performance Issues: For large code files, the highlighting process can become slow. Consider optimizing the tokenizer and highlighter functions or using techniques like lazy highlighting (highlighting only the visible part of the code).
  • Incorrect Character Handling: Ensure that your tokenizer correctly handles special characters, escape sequences, and different types of quotes.
  • Event Listener Issues: Ensure the event listener is correctly attached to the textarea. Double-check the element’s ID and the event type.

Enhancements and Advanced Features

Here are some ways to enhance the syntax highlighter:

  • Support for More Languages: Extend the tokenizer to support other languages like Python, Java, C++, etc. This involves defining new keywords, operators, and token types for each language.
  • Themes: Allow users to switch between different color themes.
  • Line Numbers: Add line numbers to the code display for easier navigation.
  • Code Folding: Implement code folding to collapse and expand code blocks.
  • Error Highlighting: Highlight syntax errors and warnings.
  • Code Autocompletion: Integrate code autocompletion to improve the developer experience.
  • Real-time Highlighting: Optimize the highlighting process for real-time updates, especially for large codebases.

Key Takeaways

  • Syntax highlighting significantly improves code readability and understanding.
  • Tokenization and styling are the core components of syntax highlighting.
  • TypeScript provides a robust environment for building complex applications.
  • Testing is crucial to ensure the accuracy of the tokenizer.
  • Consider performance optimizations for large code files.

FAQ

1. What is the difference between tokenization and parsing?

Tokenization is the process of breaking down code into tokens (e.g., keywords, identifiers, operators). Parsing is the process of analyzing the tokens to determine the code’s structure and meaning, often building a syntax tree. Tokenization is a prerequisite for parsing.

2. How can I add support for a new language?

You need to modify the tokenizer to recognize the keywords, operators, and syntax of the new language. This will involve updating the token types, keywords list, and the tokenizing logic.

3. How can I improve the performance of the highlighter?

Optimize the tokenizer and highlighter functions. Consider techniques like lazy highlighting (only highlighting the visible part of the code) and caching the highlighted output.

4. What are some popular syntax highlighting libraries?

Some popular libraries include Prism.js, highlight.js, and CodeMirror. These libraries provide pre-built syntax highlighting functionality and are often easier to integrate than building your own from scratch.

5. How can I handle nested structures like parentheses and brackets correctly?

The tokenizer needs to keep track of the nesting levels. For example, when it encounters an opening parenthesis, it increments a counter. When it encounters a closing parenthesis, it decrements the counter. This helps to ensure that the highlighting is correct even with nested structures.

The journey of building a syntax highlighter is a rewarding one, providing a deeper understanding of how code is structured and interpreted. While this tutorial provides a basic foundation, the possibilities for customization and enhancement are vast. As you delve deeper into the intricacies of tokenization, styling, and language-specific rules, you’ll gain valuable insights into the art of code analysis and presentation. The creation of tools that enhance the developer experience is a continuous process of learning and refinement, and with each line of code, you’re contributing to a more readable and efficient coding world.

More posts