SEC Enforcement Feed: Zero-Cost Unchanged Records

This guide explains how to build an SEC enforcement action feed where unchanged records cost nothing to store or process. You’ll learn a practical pattern that uses content hashing and append-only storage to avoid re-ingesting or re-paying for data that hasn’t changed. The approach works for any source that publishes periodic updates, such as SEC EDGAR filings, enforcement actions, or litigation releases.

What You’ll Need

• Access to the SEC’s public data sources (EDGAR full-text search API, RSS feeds, or bulk data files).
• A server or cloud function that can run a scheduled job (e.g., every hour or daily).
• A database or object storage for storing records and their hashes (e.g., PostgreSQL, DynamoDB, S3).
• Basic familiarity with command line, Python or JavaScript, and HTTP requests.

Step-by-Step

1. Identify the source and its update pattern. SEC EDGAR provides a full-text search API (https://efts.sec.gov/LATEST/search-index?q=...) and RSS feeds for recent filings. Determine the endpoint that gives you a list of new or modified items. For enforcement actions, the SEC’s litigation releases are available via RSS at https://www.sec.gov/news/pressreleases.rss (filter for enforcement).
2. Define your record schema. Each record should have at least: a unique ID (e.g., accession number for EDGAR), a title, a publication date, a URL, and a content hash. Store the raw content (or a reference to it) only if you need it later.
3. Fetch the latest list of items. Write a script that calls the API or RSS feed and parses the response. For EDGAR, you’ll need to set a User-Agent header with your contact info (SEC requires this). For RSS, parse the XML.
4. For each item, compute a content hash. Use a cryptographic hash like SHA-256 on the item’s unique fields (title, URL, publication date, and possibly the body text if available). This hash uniquely identifies the content. If the item has a body, include a normalized version of it (strip whitespace, lower-case) to avoid false changes from formatting.
5. Store the hash in a lookup table. In your database, create a table seen_hashes with columns hash (primary key) and first_seen_at. When you process an item, compute its hash and check if it exists in this table.
6. If the hash is new, treat it as a new record. Insert the full record into your main table (e.g., enforcement_actions). Also insert the hash into seen_hashes. This is the only time you pay for storage and processing.
7. If the hash already exists, skip it. Do not re-insert or re-process the record. This is the core of the “unchanged records cost nothing” approach. You can optionally log the fact that it was seen again, but that log entry can be minimal (e.g., just the hash and timestamp) and still costs very little.
8. Handle updates to existing records. Some SEC items may be amended (e.g., a press release updated with new information). To catch that, compute the hash on the content that would change. If the hash differs from what you stored, treat it as an update: update the record’s fields and store the new hash. You may want to keep a history of hashes per record ID.
9. Schedule the job. Run the fetch-and-compare script on a cron schedule (e.g., every 15 minutes for RSS, hourly for EDGAR). Use a simple scheduler like cron or a cloud scheduler (AWS CloudWatch, Google Cloud Scheduler).
10. Store only what you need. To minimize costs, store the raw content only for new records. For unchanged records, you don’t store anything new—just the hash lookup. If you need to serve the feed, you can query the main table for all records, but you won’t be re-fetching or re-storing unchanged data.
11. Monitor and clean up. Periodically review the seen_hashes table size. It will grow linearly with the number of unique items, which is fine. If you want to purge old hashes, you can delete hashes older than a certain date, but that may cause re-processing of old items if they reappear.

Common Mistakes

• Including volatile fields in the hash. If you hash the entire JSON response, which may contain timestamps or random order, you’ll get false new records every time. Only hash the stable, content-defining fields.
• Not normalizing content. Extra spaces or line breaks can change the hash. Normalize the text (trim, collapse whitespace) before hashing.
• Ignoring pagination. SEC APIs often paginate results. If you only fetch the first page, you’ll miss older items that may be new to you. Implement pagination to get all items.
• Not setting a User-Agent. SEC blocks requests without a proper User-Agent. Always include one with your email or app name.
• Storing full content for every check. If you store the full content for each item every time you see it, you’ll pay for storage unnecessarily. Only store content when it’s new or updated.
• Using a weak hash. MD5 or SHA-1 are fast but have collision risks. Use SHA-256 or better.

Tips

• Use a dedicated hash table. Keep the hash lookup separate from the main data table to speed up checks.
• Batch inserts. When processing a large batch of new items, use bulk insert operations to reduce database load.
• Log only anomalies. Instead of logging every “seen again” event, log only when you see a hash that you’ve never seen before, or when an update occurs. This keeps logs small.
• Consider using a serverless function. A cloud function that runs on a schedule can scale to zero, so you pay nothing when it’s not running.
• Test with a small sample. Before going live, run the script against a few days of data to verify the hash logic works.
• Store the hash as a binary or hex string. Use a fixed-length column (e.g., CHAR(64) for SHA-256 hex) for faster lookups.

Troubleshooting

• You’re getting duplicate records. Check if your hash is including a field that changes (like a timestamp). Also verify that you’re not processing the same item twice due to overlapping pagination.
• You’re missing new records. Ensure you’re fetching all pages. For RSS, check if the feed only shows the latest 10 items—you might need to poll more frequently or use the full-text search API.
• SEC returns 403 Forbidden. This usually means your User-Agent is missing or not accepted. Add a proper User-Agent with contact info. Also, avoid making too many requests per second; add a delay.
• Hash mismatch on updates. If you expect a record to update but the hash doesn’t change, you might be hashing the wrong fields. For example, if you hash only the title and URL, an updated body won’t be detected. Include the body or a summary in the hash.
• Database grows too large. If you’re storing every hash check, your log table will explode. Move to a model where you only insert new hashes, not every check.
• Script crashes mid-run. Make the script idempotent: if it crashes, the next run should be able to pick up where it left off. Use a “last processed timestamp” stored in a state table.

Frequently Asked Questions

Q: What exactly does “unchanged records cost nothing” mean?

A: It means you don’t pay for storage or processing for records you’ve already seen. You only pay when a new or modified record appears, because you only store new data and only compute the hash for comparison.

Q: How often should I poll the SEC feed?

A: It depends on how quickly you need updates. For enforcement actions, polling every 15 minutes is usually sufficient. SEC EDGAR updates are not real-time but are typically available within minutes of filing. Polling more frequently than every 5 minutes is unnecessary and may get you rate-limited.

Q: Can I use this pattern for other data sources?

A: Yes, any source that publishes periodic updates with a stable identifier or content can use the same hash-and-compare pattern. It’s common for news feeds, job boards, and price lists.

Q: Do I need to store the full text of each enforcement action?

A: Only if you need to serve the content later. If you only need metadata (title, date, URL), you can store just that. The hash is computed from the fields you care about.

Q: What if a record is deleted from the SEC site?

A: Your feed will still have the record. If you want to reflect deletions, you’d need to periodically compare the full list of IDs from the SEC with your stored IDs and mark missing ones as deleted. That’s a separate process and may incur costs, but it’s optional.

Q: How do I handle amended filings?

A: When an existing item changes (e.g., a press release is updated), the hash will differ. You can treat that as an update: update the record’s fields and store the new hash. Keep a history of hashes per record ID if you need an audit trail.

Related

• AI Startup Defensibility as Models Become Commodities
• Make Grok Imagine Videos and Upscale Them for Free
• How Long Can an Astronaut Work Outside on the Moon?