How to insert only new data in MongoDB or Mongoose

Recently, I started tracking the RBI Repo rate for my finance app and save the historical data in my database.

Every week, the custom scraper will run and scrape the data from reliable sources and then send it to the personal finance app API.

The payload will be something like this:

{
  "results":[
    {"date":"2022-12-07", "repoRate":6.25},
    {"date":"2022-09-30", "repoRate":5.90}
  ]
}

For the above case, the date will be the single source of truth. So we can check that before inserting new items into the database.

And here is a snippet on how to do it.

// ideally, the result will be from the API payload. For convenience, we're hard-coding it in the snippet
const results = [
    {"date":"2022-12-07", "repoRate":6.25},
    {"date":"2022-09-30", "repoRate":5.90}
]

const convertDate = dt => {
  const year = dt.getFullYear();
  const month = dt.getMonth() + 1;
  const date = dt.getDate()
  return `${year}-${month}-${date}`
}

// query the existing things
const dbRecords = await RepoRate.find().select({ date: true }).lean()

const existingDates = new Set(dbRecords.map(item => convertDate(item.date)));
const uniqueRecords = results.filter(item => !existingDates.has(item?.date))
if (uniqueRecords) {
  await RepoRate.insertMany(uniqueRecords)
}

Happy new data!