Sign in / up

back to article Get in the lake, Sparky: Databricks touts new ingestion file sources

Databricks, the company behind the popular open-source big data tool Apache Spark, has released an ingest technology aimed at getting data into data lakes more quickly and easily. Auto Loader is a file source that helps load data from cloud storage continuously and "efficiently" as new data arrives, which the company claims …

COMMENTS

Post your comment
House rules Send corrections
Add to 'My topics' unstar *
  1. Androgynous Cow Herd

    Technical Wordsmith Wizards!

    "Databricks said Auto Loader avoids file state management by incrementally processing new files as they land in cloud storage."

    sounds so much better than

    "Databricks ignores metadata and relies on FIFO".

    I do like "Lakehouse" though.

    1 0 Reply
  2. AbeSapian

    They Missed a Bet

    Instead of data lakes, they should have called it Pensieve.

    0 0 Reply

POST COMMENT House rules

Not a member of The Register? Create a new account here.

Forgotten password ?

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like