<h1>Introduction</h1>
I decided to start working on a public project where I’ll be sharing a step-by-step approach to building it. I’ve already created a project that should be great for this purpose, so in this post, I’ll describe what I’ve already done. In the future, we’ll be working together! 😁
The project’s code is publicly available on GitHub under this repository, so feel free to learn from it and contribute: <a href="https://github.com/carson2222/hl-monitor" target="_blank" rel="noopener nofollow">REPO</a>
<h1>Motivations</h1>
I’m running this project mainly for educational purposes, but I’m also helping my Solana DAO, SaveTheWhal3s, by providing these tools to them. Currently, there’s only a tool for extracting large <a href="https://hyperliquid.gitbook.io/hyperliquid-docs/trading/order-types#twap-details" target="_blank" rel="noopener nofollow">TWAPS</a> placed on HL-related coins. Since I already have all the data scraped, I can use it in many ways.
<h1>Is it really a blockchain?</h1>
As you might find strange, I’m not actually using any blockchain-related stuff. To scrape data, I use API and WebSocket (WS) calls. Why? I don’t want to go into too much detail about how Hyperliquid works in this article, so I’ll keep it short.
Yes, Hyperliquid will be a fully operational ENV chain like Polygon or Arbitrum. Right now, it’s only in testnet mode, meaning you can build and experiment on it, but there’s no real value involved yet. Normally, other chains at this stage just provide a testnet and leave it at that. However, we’ve been able to trade their token, HYPE, for quite some time now and use the blockchain as if it were fully operational.
It’s like the mainnet EVM mode is working but not yet public. A simplified version of the blockchain is accessible via their API, meaning you can only perform specific actions they allow. Imagine they’ve built just one app so far, and that’s all you can use. Once they fully open up, we’ll be able to create new apps, use the blockchain more efficiently, and truly hold our tokens in our own EVM wallets. Right now, everything is API-based, and we don’t really know when that will change.
Please don’t punish me for oversimplifying it! lol 😅
<h1>Finally, the CODE!</h1>
Let’s start discussing my approach to monitoring all new transactions and how I manage the data.
<h2>1. Connecting to the Hyperliquid WebSocket</h2>
Here, I simply use their UI explorer WebSocket to get all new blocks created on the blockchain. Initially, I tried using the WebSocket that returns new transactions, but I realized it only returns the first transaction of each block, even though there are hundreds or thousands per block. So the only way to go is by fetching blocks instead.
Since this isn’t a normal public route, I don’t think they provide an option for fetching all transactions via their main WebSocket.
<a href="https://hyperliquid.gitbook.io/hyperliquid-docs/for-developers/api/websocket" target="_blank" rel="noopener nofollow">WS DOCS</a>
<pre><code class="language-typescript">export function initWebsocket() {
 const webSocket = new WebSocket(&quot;wss://api-ui.hyperliquid.xyz/ws&quot;);
 handleWebsocketClose(webSocket);

 webSocket.onopen = () =&gt; {
 webSocket.send(
 JSON.stringify({
 method: &quot;subscribe&quot;,
 subscription: {
 type: &quot;explorerBlock&quot;,
 },
 })
 );
 console.log(&quot;Successfully sent subscription request to explorerBlock&quot;);
 };

 return webSocket;
}
</code></pre>
<h2>2. Reading basic data</h2>
<pre><code class="language-typescript">const spotMeta = await loadSpotMeta();
const perpMeta = await loadPerpMeta();
const allMids = await loadCoinMids();
</code></pre>
Here, I make three fetch requests to <code>https://api.hyperliquid.xyz/info</code> to load essential data:
<ul>
<li>SpotMeta &amp; PerpMeta contain metadata related to spot/perp tokens. We need this to determine which coin we’re dealing with when analyzing transaction data.</li>
<li>CoinMids is a list of all coin prices, which I use to calculate TWAP values and filter only the “big” ones.</li>
</ul>
<h2>3. Handling new messages</h2>
Now, we use the WebSocket connection we established earlier. Each message contains an array of numbers representing block IDs, which we use to fetch transaction details. Usually, each message contains just one block, but not always—so we loop through them with <code>forEach</code>.
<pre><code class="language-typescript">webSocket.onmessage = (event) =&gt; {
 //@ts-ignore
 const res = JSON.parse(event.data);

 if (res.channel === &quot;subscriptionResponse&quot; &amp;&amp; res.data.subscription.type === &quot;explorerBlock&quot;) {
 console.log(&quot;Successfully connected to explorerBlock WebSocket&quot;);
 return;
 }

 res.forEach(async (block: any) =&gt; {
 try {
 if (currentProxyIndex &gt;= agents.length) currentProxyIndex = 0;
 else currentProxyIndex++;
 const info = await fetchBlockInfo(block.height, currentProxyIndex);

 info?.blockDetails?.txs.forEach((tx: any) =&gt; {
 const type = tx.action.type;
 if (type === &quot;twapOrder&quot;) {
 handleTwapOrderTx(tx, { spotMeta, perpMeta, allMids } as CoinsData);
 }
 });
 } catch (error) {
 console.error(error);
 }
 });
};
</code></pre>
<h2>4. Fetching all block transactions</h2>
Now, let’s talk about the <code>fetchBlockInfo</code> function from earlier. It’s simply an API call to fetch all transactions in a given block. Again, this uses an undocumented UI API route.
I added a max 5 retry loop to minimize skipping blocks while avoiding an endless loop due to Hyperliquid errors.
<h2>5. Rate limit problem</h2>
Hyperliquid states that block creation time is 200ms, meaning a new block appears every 0.2 seconds—that’s 300 requests per minute. At this point, I hit a wall, and I didn’t like the only solution available. They have a pretty strict rate limit.
The only thing I could do to resolve this was to use multiple proxies. I found that ~25 proxies were enough to avoid <code>429</code> errors.
A single residential proxy would have been enough, but the problem is the huge amount of data we’re scraping—around 100-200GB per day. Using a residential proxy would cost me thousands of dollars per month.
So instead, I opted for 25 of the cheapest database proxies I could find. The speed doesn’t really matter; we just need the data.
<h1>END</h1>
That’s the end of this article, but just the beginning of our journey! I haven’t fully covered everything I’ve built yet, and there’s still a lot of cool stuff to do.
In the next article, I’ll show you how I’m filtering the data and formatting it to generate neat webhooks like this:
<img src="https://media.daily.dev/image/upload/s--DRorgJBL--/f_auto/v1739361704/ugc/content_d1d4c0a4-3c9a-4bd2-8d86-765d9f315d8f" alt="image">

Unofficial Hyperliquid Squad, for content, questions, discussions about Hyperliquid Blockchain 

Hyperliquid

An overview of a public project that demonstrates step-by-step data scraping techniques using Hyperliquid's WebSocket API. The project aims to help a Solana DAO and is currently in the testnet mode with a simplified EVM chain accessible via API. It details approaches to connect, fetch, and process transaction data using multiple proxies to deal with rate limits.

Hyperliquid data scraping