Spending last month at ITP Summer Camp 2024, I was exposed to p5.js for the first time in quite a while, with sessions on hand tracking, audio visualization, and a whole lot more.

Artists especially tend to flock to p5.js because it offers such a welcoming onramp to coding interactive and multimedia experiences—it’s the only framework I’ve ever seen where hello world is drawing a circle on the screen.

In my more art-centric days in Osaka between 2010 and 2015, Processing—p5’s Java-based ancestor—was of particular fascination to me. Around the time my interests starting moving to other areas in software development, p5.js was shipping its first versions. Since at the time I was focusing elsewhere, I never did much more than a hello world with p5.js.

Last month during ITP Summer Camp, I led a session called “AI for the mere mortal web developer”. In the session, I showed the first steps of building LLM-powered features in web apps, with Vibebox as an example.

A fellow ITP camper recently reached out about the session, asking how to connect LLMs to p5.js sketches. This is something I wanted to explore during camp but didn’t have time, so I took the opportunity to investigate.

What I built

Since the goal was to show how to connect LLMs and p5.js, the use case is contrived and simple:

Create random colors for a p5.js canvas background.

In other words:

• Ask an LLM for 3 numbers between 0-255 (8-bit unsigned integers)
• Parse the answer
• Feed the parsed answer into p5.js’s background color setter

That’s it!

Project structure

The project is comprised of two files hosted on two different services:

• Client: hosted on p5js.org
• Server: hosted on Val Town

Let’s have a look at both.

Server

Since communicating with an LLM service like OpenAI requires handling secret credentials, the project has a proxy server to avoid leaking secret API keys in the browser.

A lot of the server code is boilerplate HTTP request/response/error handling that isn’t worth getting into here.

The core functionality is in this getLlmResponse() helper function:

/**
 * Gets the response from the OpenAI language model.
 * @param {string} prompt - The prompt for the language model.
 * @returns {Promise<string>} - The response from the language model.
 */
async function getLlmResponse(prompt: string) {
  const completion = await openai.chat.completions.create({
    "messages": [
      { "role": "user", "content": prompt },
    ],
    model: "gpt-3.5-turbo",
    max_tokens: 50,
  });

  return completion.choices[0].message.content;
}

The function simply takes a prompt and passes it to GPT 3.5-turbo. There are plenty of knobs and levers you could play with here, but this is enough for a proof-of-concept.

You might notice that, while I mentioned use of a proxy server to avoid leaking secret credentials, the code doesn’t seem to be using credentials at all. This is because, for the sake of simplicity, I’m using the free tier of Val Town’s OpenAI standard library module.

That means I’m effectively using Val Town’s API key, no settings required. Should I ever need to graduate from that free tier, I can bring my own OpenAI key, which I would store as an environment variable.

Client

For this proof-of-concept, I’ve stored my front end as a sketch on the p5.js editor.

The sketch does the following:

1. In preload(), requests random RGB values from the LLM and set them as background colors
2. In setup(), creates the canvas and uses the backgroundRgb values as the background color
3. In draw(), uses the latest backgroundRgb values as the background color

The core functionality for this demo is found here:

/**
 * Fetches RGB values from a language model via a POST request.
 * @returns {Promise<number[]>} - An array of three numbers representing RGB values.
 */
async function getRgbValues() {
  const headers = new Headers();
  headers.append("Content-Type", "application/json");

  const body = JSON.stringify({
    "prompt": "Give me three numbers between 0 and 255, comma separated"
  });

  const requestOptions = {
    method: "POST",
    headers,
    body
  };

  try {
    const llmProxyBaseUrl = "https://ashryanio-openaiproxy.web.val.run";
    const response = await fetch(llmProxyBaseUrl, requestOptions);
    
    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }
    
    const { llmResponse } = await response.json();
    console.log(`The LLM responded with: ${llmResponse}`);
    
    const rgbValues = llmResponse.split(", ").map(Number);
    console.log(`The LLM response was parsed into an Array of Numbers as: ${rgbValues}`);
    
    return rgbValues
  } catch (error) {
    console.error("Error making POST request to LLM:", error);
  }
}

This getRgbValues() helper function:

1. Sets up the request to the server, including my LLM prompt
2. Makes the request
3. Handles the server response
4. Parses the response into rgbValues that can be used elsewhere in the p5 code

The way the p5 sketch is currently written, you’ll see this output in console:

[Preload] Calling LLM for color values... 
[Setup] Setting these background color values: 0,0,0 
The LLM responded with: 34, 156, 209 
The LLM response was parsed into an Array of Numbers as: 34,156,209 
[Preload] New background color values: 34,156,209 

This demonstrates that anytime backgroundRgb is updated the canvas background color will be updated accordingly:

1. During setup(), the background gets the default 0,0,0 (black)
2. During preload(), after the server returns the response from the LLM, the background is updated to 34,156,209 (some shade of sky blue)

As a side note from me as a p5 novice, I remain surprised that preload() doesn’t block setup(). I’m sure there’s another way to handle it, but it didn’t feel worth pursuing for this little demo.

Either way, this does the trick—we’re getting values from an LLM and using them in our p5.js code!

Things to consider

Here’s a grab bag of things on my mind as I share this:

1. This project is naively assuming that the LLM will follow my prompt to the letter, giving me back 3 8-bit unsigned integers and nothing else. This is a bad assumption. Production code would need to cope with potentially anything coming back from the LLM.
2. As I mentioned in the Client section, I’d prefer to have a better conceptual handle on asynchrony in p5’s preload() and setup() lifecycle functions. Since this is a demo, there’s no right answer here, but any project does need to handle the loading state for external API calls. By default, my loading state here is a black background. But you could imagine showing a loading animation instead.

Jump in!

You can try the project and get the code here:

• Client: hosted on p5js.org
• Server: hosted on Val Town

To learn more about the underlying technologies, checkout:

• https://p5js.org
• https://www.val.town

You can also check out my recent post on what Val Town is.

If you have any feedback or questions, I’d love to hear from you.