Freshness score

The freshness score is derived using a modified sigmoid function to evaluate the dataset's recency. This score hinges on the most recent date listed on the datacard, reflecting its relevance to current AI research and application needs. The sigmoid function, characterized by parameters adjusting its steepness, inflection point, and maximum score, ensures a balanced assessment.

# Sigmoid function parameters
a = 1 # Adjust steepness of the sigmoid curve
b = -(current_date + scaling_factor) # Adjust the inflection point
c = 100 # Adjust the maximum score
 
score = c / (1 + math.exp(-(a * time_fraction + b)))

Freshness score visualised

This approach underscores the critical role of current data in developing effective AI models, especially for Retrieval-Augmented Generation (RAG) systems. By incorporating the latest information, RAG pipelines enhance the relevance and accuracy of model outputs, aligning with contemporary developments. In the fast-paced AI domain, the timeliness of data significantly influences model performance, ensuring they reflect current realities and generate more meaningful insights.