LOADING PYDANTIC MODELS FROM JSON WITHOUT RUNNING OUT OF MEMORY
Memory usage can become a significant bottleneck when working with large JSON files and Pydantic. The default Pydantic JSON loading method can result in a 20x memory multiplier, making it impractical for large files. To reduce memory consumption, it's better to instead use ijson, an incremental JSON parser, to stream the JSON data instead of loading it all at once.