product @cloudera | prev product @aws @dremio @verticaunified • #data #analytics #design #tech for 🌍
Somerville, MA
Joined October 2012
- Tweets1.4K
- Following4.9K
- Followers882
- Likes3.2K
Mark Lyons retweeted
Microsecond-accurate time is now available in EC2 US East. So many cool things this makes possible: aws.amazon.com/about-aws/wha…
Mark Lyons retweeted
TPCH-SF30 ; 180 million rows
#AZURE D16DS_V5; 16 Cores, 64 GB RAM
#Databricks Photon 41 S
#DuckDB : 43 second
Query Parquet files from the VM SSD, no Azure storage involved
Databricks Software cost (not hardware) 4.4 $/Hour
github.com/djouallah/Testing…
Mark Lyons retweeted
Join @dremio’s Tech advocacy & Eng team for the very first installment of the @ApacheIceberg Office Hours 📆 🚀
We will kick-off with a brief presentation on Copy-on-Write Vs Merge-on-Read strategies, followed up by Q&A on anything Iceberg related.
When: December 7th, 12 PM – at Toronto, Ontario
Mark Lyons retweeted
Reminder, if you want to learn more about Apache Iceberg I have loads of resources plus a video series all curated in this article. -> dremio.com/subsurface/apache… #BigData #DataLake #DataLakehouse
Mark Lyons retweeted
Query planning in @ApacheIceberg
Being able to efficiently plan queries is super critical for faster execution of the queries run by analysts 🧑🏻💻
This is specifically critical when dealing with large-scale data such as data in data lakes. Read @IcebergDevs 👇
#dataengineering – at Toronto, Ontario
Mark Lyons retweeted
The @ApacheArrow project has grown in all axes 🚀
In fact, more & more tools/libraries in the #dataanalytics space have started using Arrow.
In this blog post, we go through the evolution of Apache Arrow from usage, capability & community angles.
dremio.com/blog/apache-arrow… – at Toronto, Ontario
Mark Lyons retweeted
My latest article on Apache Iceberg compaction strategies -> dremio.com/subsurface/compac… #BigData #dataengineering #datalake #DataLakehouse
Mark Lyons retweeted
Manage data as code?
Just like Git but for Data?
That's right!
@projectnessie is an open source work that brings the capabilities of Git-like branching to the world of data & specifically to data lake table formats like #ApacheIceberg
#dataengineering – at Toronto, Ontario
Mark Lyons retweeted
We're thrilled to announce that we've been named to @CNBC’s ‘Top Startups for the Enterprise’ Inaugural List 🎉
Read more about our open data lakehouse and this inaugural list here:
bwnews.pr/3UehuIN
#CNBC #TopStartup #Tech
Mark Lyons retweeted
Are you heading to AWS re:Invent later this month? Check out this link for all the details on how you can:
➡️ Schedule a meeting with us
➡️ Enter our Dremio Cloud data challenge (for a chance to win a PS5!)
➡️ RSVP to our cocktail reception
awsreinventdremio2022.splash…
#AWSreInvent
Mark Lyons retweeted
If you find what you see interesting here is a tutorial I wrote giving you a step by step guide getting setup and doing an example exercise -> dremio.com/blog/managing-dat…
Mark Lyons retweeted
How do we migrate from one catalog to another for @ApacheIceberg tables?
if you are already using a catalog (say HDFS) & want to change it to something else (say AWS Glue), how is that possible?
A 🧵 for @IcebergDevs
#dataengineering – at Toronto, Ontario
Mark Lyons retweeted
With all the recent news about #ApacheIceberg we thought we'd share this video from last year's Subsurface Conference. We're looking for speakers for our event happening in spring 2023 🎤 submit your talk today!
sessionize.com/subsurface-li…
#CallForSpeakers
Mark Lyons retweeted
Merge-On-Read (MOR) Vs Copy-On-Write (COW) in @ApacheIceberg.
Both these approaches are used to deal with deletes & updates of data files in the Data lake.
Let’s break down @IcebergDevs👇
#DataEngineering #data
Mark Lyons retweeted
Don't miss your chance to take the stage at Subsurface LIVE, coming in the Spring of 2023 🎉
We’re accepting proposals now for key topics. See details + submit your proposal now 🎤
lnkd.in/gMXtSTSJ
#CallForSpeakers #Data #ApacheIceberg #DataLakehouse
Mark Lyons retweeted
Subsurface LIVE is back! Coming in the Spring of 2023 🎉
We’re accepting proposals now for key topics. See details + submit your proposal now 🎤
lnkd.in/gMXtSTSJ
#CallForSpeakers #Data #ApacheIceberg #DataLakehouse
Mark Lyons retweeted
Easiest way to get started with a Lakehouse 🏠 platform?
- Adopt a table format like @ApacheIceberg
- Load the @ApacheParquet files in S3 bucket
- Connect @dremio cloud to S3 source
- Create an Ice table using CTAS
- DML
#dataengineering
Mark Lyons retweeted
Big Data London is the place to be this week 🇬🇧 #BigDataLDN
Mark Lyons retweeted
Watch this video to hear about how Voltron's Matt Topol @zeroshade got a book deal by being a speaker at the Subsurface Conference. Subsurface will be held March 1-2 live online and you can become a speaker by submitting your application at bit.ly/3Dy0FTO, apply today!