I wrote this post, summarizing years of working with scientific data archives across multiple fields of science.
Everyone wants to make their scientific data archive work with Agents and AI, but this requires a complete rethink of the storage layer. Luckily, the recommendations end up being very similar to what data providers should have been doing anyway: forget about files!
This post takes you from "what is a chunk" all the way to understanding why there is really only one sensible way to manage large scientific datasets in the age of AI.
It's also a choose-your-own-adventure post - I didn't want to have to write the same thing 5 times for 5 different fields of science, so I just made one post in which you can toggle examples between different fields!
I wrote this post, summarizing years of working with scientific data archives across multiple fields of science.
Everyone wants to make their scientific data archive work with Agents and AI, but this requires a complete rethink of the storage layer. Luckily, the recommendations end up being very similar to what data providers should have been doing anyway: forget about files!
This post takes you from "what is a chunk" all the way to understanding why there is really only one sensible way to manage large scientific datasets in the age of AI.
It's also a choose-your-own-adventure post - I didn't want to have to write the same thing 5 times for 5 different fields of science, so I just made one post in which you can toggle examples between different fields!