SocialCrawl

Use SocialCrawl with Pipedream

Pull social media data into Pipedream without writing a server.

You can call SocialCrawl from a Pipedream workflow with a Node.js step. One GET request returns structured data for the next step. Setup takes two steps: store the key, add the code step.

Before you start

Create an API key in the dashboard under API Keys. Store it as an environment variable in Pipedream, for example SOCIALCRAWL_API_KEY, and read it with process.env. Never paste it into the code. Prove the key with a call that costs nothing:

Check the key (0 credits)
curl "https://www.socialcrawl.dev/v1/credits/balance" \
  -H "x-api-key: sc_your_key_here"

A reply with "success": true and your balance in data.balance means the key works.

Connect

Add a Node.js code step, name it socialcrawl, and call the API with axios:

Node.js code step
import axios from "axios";

export default defineComponent({
  async run({ steps, $ }) {
    const res = await axios({
      url: "https://www.socialcrawl.dev/v1/tiktok/profile?handle=charlidamelio",
      headers: { "x-api-key": process.env.SOCIALCRAWL_API_KEY },
      validateStatus: () => true,
    });
    return res.data;
  },
});

validateStatus: () => true matters. By default axios throws on any 400 or 500 answer, and the thrown error hides the body. With it, the step returns the SocialCrawl answer and the next step can read it.

Read the answer

Every response has the same shape. Branch on success. A list puts its rows in data.items, and pagination.next_cursor is the token for the next page.

Response (trimmed)
{
  "success": true,
  "data": { "author": { "username": "charlidamelio" } },
  "credits_used": 1,
  "credits_remaining": 99
}

To read the next page, call again with next_cursor as the cursor param, and stop when pagination.has_more is false. See Pagination.

When it stops

With validateStatus set as above, a refused call returns success: false instead of throwing. Test it in the next step and throw your own error so the run shows as failed:

Fail the run on a refused call
if (steps.socialcrawl.$return_value.success === false) {
  const e = steps.socialcrawl.$return_value.error;
  throw new Error(`${e.type}: ${e.message}`);
}

When credits run out, error.type is INSUFFICIENT_CREDITS. Read error.message for the text and error.details.checkout_url for the link that opens checkout. When the link is absent, error.details.top_up_url opens the billing page.

Never retry a 402, because it fails until credits are added. Retry a 429 after the seconds in the Retry-After header, and a 502, 503 or 504 after a short backoff. See Errors.

What a run costs

It depends on the endpoint, and some endpoints are priced by what they return. Endpoint pricing lists each one, and Credits explains cache hits and refunds. Before you schedule a workflow, GET /v1/utility/estimate quotes a call for 0 credits.

On this page