Project Mantis Shrimp — A (Guide to) Automated Course Registration System

Get the courses you want. Automatically.

Background

Advanced Registration is Penn's batched course preference matching system. Students are given a chance to put in their classes of choice while the system evaluates the priority of course requests based on major, school year, and other requirements to give students a subset of what they have put down. The fortunate few get all their requested classes when the computed results are reported, leaving the course registration arena early and victoriously. The rest, however, are all ready to wake up the following day at 7 am to compete for the remaining spots that are still up for grabs.

Sad reacts only.

While the most fierce contest will take place in the hour after the regular registration system is open. The race does not ultimately stop until the end of add/drop period next semester. The contenders use technologies like Penn Course Alert(by Penn Labs) and Pennmate Notify to learn when a course is open for registration — some are even willing to pay real dollars for premium varieties (e.g., Coursicle) to get better notifications.

Penn Couse Alert UI

With course alert services becoming such a commodity, any student can get some form of notification within a few minutes of the course becoming available. However, the real constraint is not knowing but instead acting on the information. Most people can echo the immense frustration after they missed the opportunity of an open course due to sleep, class, attending an event, or just sheer carelessness. How great would it be if you could just hire someone to sit at a desk and register the courses for you when the revelation from Penn Course Alert arrives.

Stock photo

Oh wait, that sounds like a job for, let's say… a computer?

Designing Your Computer Course Secretary(ok, Bot)

Let's first answer the question of "What." What exactly do we need our "Secretary" to do?

Think about how we humans would normally perform course registration in steps:

1. decide what courses we want to register
2. if the course is open-> we could just stop here; however, if the course is closed currently…
3. wait for the course to become open (by checking PIT often or using a service like PCA)
4. once the course is open, sign onto PIT and register for the course.

Sounds simple enough — Number 1 is trivial; Number 2 can simply be done by hand; Number 3 is a painstaking process that requires a long wait and quick action once the course is open; Number 4 seems to be more difficult with the many steps of browsing and clicking required.

To summarize our functional requirements:

1. know when a class opens
2. logically filter and only act on classes we care about
3. Simulate a human user login, 2FA, and navigate to course registration

Naturally, there are also some non-functional requirements that we need to consider that are common sense.

1. the course status monitoring needs to be 24/7 as a course can open at any time
2. all actions performed by the Bot must adhere to reasonable resource consumption for its own server, the Penn course database, and PIT itself.
3. No human intervention is required for the registration flow, including 2FA (a challenge).
4. The user should get some kind of feedback on the status of the Bot.

With all this in mind, it's time for implementation. Since the three functional requirements naturally fall into three components/stages of the system, they will be discussed in their own chapters.

Part I: Know When Courses Are Open

Several clear paths are available to determine when courses are open:

1. Use Penn Course Alert and have a server that parses email/SMS message
2. Poll PIT course search page / Penn Open Data Course Status API

Solution 1 is the most intuitive and direct replication of the human process. However, it relies on PCA's performance and email/SMS's reliability and performance. The latency is through the roof on this one, not to mention the multiple conversion of data and the inherent inefficiency in such a process (Course status -> PCA -> email/SMS in string-> actionable data).

On the other hand, solution 2 is a bit closer to the data source — the Bot is accessing the raw data directly at a set interval. The only latency is internet speed and the potential time difference between course status updates and the next scheduled polling. Time is of the essence in this application; a 30-second difference may mean a highly sought-after course is falling into someone else's schedule. To minimize the latency, one must decrease the polling interval. The drawback is significant; frequent polling is inefficient and disastrous to the server performance. At the end of the day, we don't want to be DDoSing Penn's already fragile server.

Now the less obvious (and best) option is saved for the last: Penn Open Data Course Status Webhook.

Webhooks are also called reverse APIs. In contrast to how the most prevalent form of what we call "Internet" work — A sends a request to B asking for something, and B sends it back — this special communication protocol, in an abstracted form, expects B (the person who has some resource) to know that they are to send some payload back, as a POST request.

In our case, rather than A (our bot) trying to see whether a course is open or not, B(Penn server that manages courses) proactively broadcast that there has, in fact, been a course status change. While it isn't possible to subscribe to exactly the courses required, the webhook provides a direct yet resource-efficient way to detect course openings.

One note on implementation: A long-running server of some sort is still required to listen for the webhook. However, the long-running server only needs to do very little work, and the more demanding (only slightly) work can be handled by short-lived workers.

(Azure, and other cloud vendors as well, gives us a very attractive SaaS…)

Part II: Filter And Action

Now that we have figured out the notification part. It's time to decide how we want to handle incoming requests.

In essence, we will have a long-running server to listen for incoming POST requests (webhook). The schema of the request body is the following:

{ "course_section" : "REAL891751", "previous_status" : "X", "status" : "O", "status_code_normalized" : "Open", "term" : "2019A" }

Whenever a request has "course_section" being one of the courses of interest, and the "status" is "O," it is time to do work. However, in other cases, the server would yawn and go back to sleep.

We could have written an Express.js server and made function calls inside of specific paths as they are called. However, there's another convenient way — Azure Functions with HTTP triggers.

Azure abstracts the paradigm by separating the triggers from the (serverless) "functions," this saves me the trouble of setting up the server myself and instead focusing on what "happens" given a webhook request.

Another benefit of serverless functions(of any vendors, really) is that the user is (generally) agnostic to the underlying resource. Whereas I would have chosen different spec servers for different stages of the task, here, I just trust that Azure would give me "enough" compute cores and memory to let me execute the task. For premium Azure Functions, you can specify hardware with simple parameters, similar to how in AWS you can specify the underlying Fargate resource type.

We are at SaaS

Now that we have a valid request (with the schema above), we will dedicate the third chapter to the actual execution via Browser Simulation.

Part III: Browser Simulation

As a self-proclaimed robotics nerd, I love to use code to simulate human behavior. Puppeteer is just such a technology that enables us to simulate action that is done in a browser.

The Google-developed NPM package provides a full Chromium headless browser that is instrumented to simulate operations (read, type, click, etc.) via code. While its primary use case is integrated testing for the web, here we are using it to get us through the PIT login and course selection steps since we can't call an API to do it directly.

An example from the internet of how Puppeteer works in headFUL mode

User authentication

There are two steps to authenticate a given user in PIT:

1. Fill in the username and password. Click Submit.
2. Fill in the 2FA code from an authenticator app and click Submit again.

Step 1 is trivial; Puppeteer simply needs to fill in the 2 form fields via `page.type()`.

Step 2 requires the user to open their phone, either click the green check button or copy and paste the code as the Bot waits… No, no, no, this violates our requirement of "no human intervention." If only we could have a robot that sits in front of our 2FA smartphone and bridge the gap… Oh wait, can we skip all the physical steps and put a simulated 2FA authenticator right in our code?

How TOTP works from Twilio

It turns out that there is a supported (and very common) method of 2FA code generation called TOTP(Time-Based One-Time Password), which takes in a set secret seed and the current time as inputs and outputs a valid 2FA code. This means that we simply need to supply the Bot with the secret seed, and it will be able to run the algorithm and generate the 2FA code on the fly.

Of course, that is the quickest way (less than 50 ms) the Bot can Waltz pass the 2FA step without any hiccup.

Course Registration

Once the user is authenticated, the course registration is only a few clicks away (literally only two). Handling them with a puppeteer is not harder than clicking the buttons yourself. Do note that the Puppeteer needs to wait for some network requests to register certain clicks before proceeding to the next step.

When we finally arrive at the course registration page, suppose that the course is still open (since less than 5 seconds ago, API sent the course status change); Puppeteer just needs to sequentially select the correct value from the drop-down and click submit.

Error Handling

Oh No. Do you say the course requires a permit to register? Or perhaps the course conflicts with an existing course on the schedule? PIT wouldn't let you register, and we would probably want to know the failed registration, so we don't show up to FNCE 207 without realizing we are not in the class.

There are two (based on empirical wisdom) kinds of errors that PIT can throw once the "Add request" button is clicked (given that it is available to click in the first place)

1. A modal pops up, signaling a failed request. The error message is printed in the modal
2. An error message is printed right below the course request component (can we just take a moment and reflect this terrible UX design?)

The Bot needs to check whether either of these errors is present after "Add request" and extract proper error messages to relay back to the puppet master.

Of course, if non of the error states are detected, then it means that the course has been successfully added. As an added layer of confirmation, one could check if the course is listed as part of "Course enrollments."

And we are done!

Summary

This has been an interesting exploration of automation with interesting implementation in HTTP webhook and TOTP 2FA. A similar idea can be deployed in something like sniping streetwear drops or automating grunt work that is predictable and repetitive.

It becomes much more difficult when you factor in reCaptcha and scaling, though.

Upon further investigation into the field, I found a few interesting tools and infra solutions that may make the process of automation even easier:

1. https://parseur.com/ Extract text from emails and PDFs with a real-time pipeline.
2. https://pipedream.com/ SaaS for trigger invoked serverless functions.
3. https://www.browserless.io/ Headless browser SaaS, convenient for serverless service that can't install a full Chromium (browser.connect())
4. https://ably.com/ This is one layer deeper — everything real-time message related(thinking pub/sub + AMQP mixed with Segment)
5. and of course the serverless vendors— https://azure.microsoft.com/en-us/services/functions/, https://aws.amazon.com/lambda/, https://workers.cloudflare.com/, etc.

Why do I call this Project Mantis Shrimp? Because just like Mantis Shrimp, this system waits and waits, then performs a deadly attack in a flash when the opportunity arises. Oh, also, this Chinese meme.

"Mantis Shrimp, let's go"

Lastly, does this Bot exist? On the record, NO.

AIGHT.