@dask_dev

Dask natively scales Python Dask provides advanced parallelism for analytics, enabling performance at scale for the tools you love

dask.org
Joined July 2017
  • Tweets926
  • Following0
  • Followers8.5K
106Media
Read about the latest improvement to GroupBy.map with Dask: xarray.dev/blog/dask-detrend… Thanks to Patrick Hoefler of @CoiledHQ for the great work here!
1
6
1
31
3,678
New blog post: Scale AI-based Data Processing EASY The FineWeb-Edu dataset comes from processing 45TB (🤯) of FineWeb And it uses a Language Model to classify the educational level of the text 😭😭 Still, we reproduced it in a few lines of code ! The key ? HF + Dask 😎
1
25
1
97
15,996
Introducing a new Python package called dask-ee! It can turn an #EarthEngine Feature Collection into a #dask DataFrame with one line of code. Accelerate your geospatial workflow with parallel computing github.com/alxmrs/dask-ee

GitHub - alxmrs/dask-ee: Google Earth Engine Feature Collections via Dask Dataframes

Google Earth Engine Feature Collections via Dask Dataframes - alxmrs/dask-ee

github.com
8
48
1
220
10,848
I'm updating the Dask.org homepage and could use quotes from happy Dask users that don't mind having their name and face on the site. Any kind words you can share about why you like Dask, find it easy, or how helped you solve a problem? Short answers preferred!

Dask | Scale the Python tools you love

Dask is a flexible open-source Python library for parallel computing maintained by OSS contributors across dozens of companies including Anaconda, Coiled, SaturnCloud, and nvidia.

dask.org
2
3
1
11
2,522
📢Really happy to announce the "Scaling with Dask guide" for @Panel_org. 🤝In many ways @Panel_org and @dask_dev is a perfect couple as they share both philosophies and key technologies. 👇Link to guide below #python #dataviz @pandas_dev @CoiledHQ #datascience #Analytics
2
9
25
2,676
Great demonstration of really large-scale data processing with the @pangeo_data stack. @CoiledHQ makes managing @dask_dev clusters so easy!
This Post is from an account that no longer exists. Learn more
1
6
38
4,129
💪@argoproj Workflows and @dask_dev Bag—the perfect mix for scalability and flexibility on a recent project. See how we helped a banking client scale their #Python DataFrame calculations in our latest post. “Scaling Python: The Banking Edition.” 👀 quansight.com/post/scaling-p…

Scaling Python: Banking Edition

See how Quansight helped a large banking client scale their Python DataFrame calculations.

quansight.com
3
12
1,529
Dask Kubernetes 2023.6.0 is out and has some features I'm really excited about. • New CLI tools • Improved kubectl output • Idle cluster shutdown • A ton of Day 2 quality-of-life improvements • and more... 🧵/ pypi.org/project/dask-kubern…
1
4
16
4,068
Dask Office Hours!!! First session: June 14th - 11am EDT We are launching Dask Office Hours, bring your questions and get help from Dask maintainers. Checkout the calendar with the zoom info at content.coiled.io/dask-offic…
3
2
832
"With [Dask and Coiled], we've found a good way to enable the vast majority of Python users to run their analyses at scale." Join @ESIPfed Cloud Computing Cluster to hear more from @mrocklin June 26! discourse.pangeo.io/t/esip-c… cc @dask_dev @zarr_dev

ESIP Cloud Computing Cluster Session June 26: Dask and Coiled with Matt Rocklin of Coiled.io

While we look forward to next month’s ESIP summer meeting, we have this month’s knowledge sharing session to tide us over. This month we’re pleased to have Matt Rocklin, CEO of Coiled.io and early...

discourse.pangeo.io
10
1
24
5,438
Performance testing at Coiled In order to avoid surprises when we publish a new release, Dask needs to be covered by a comprehensive battery of tests — both for functionality and performance. Learn how we do it in this blog by Guido Imperiale medium.com/coiled-hq/perform…
5
5
1,189
Replying to @knotoast @dask_dev @DirhousssiAmine @__jrbourbeau__
You can buy Dask gear from the @NumFOCUS store numfocus.myspreadshop.com/da…
1
4
616
I'm particularly proud of our implementation of a @dask_dev-delayed @PyTorch predictor for stackstac-composed @xarray_dev-backed STAC collections. This allowed us to scale our inference to 40 years of imagery with arbitrarily-large parallelism. (github.com/oxfordeo/oxeo-wat…)

GitHub - oxfordeo/oxeo-water: A research repository for water inference

A research repository for water inference. Contribute to oxfordeo/oxeo-water development by creating an account on GitHub.

github.com
2
5
13
1,496
This Thursday 04/13 I'll be giving a virtual talk on a local meetup about Scaling Python with @dask_dev and @CoiledHQ ! You can sign up here: meetup.com/bethesda-data-sci…
2
5
1,079
While speed improves, the real benefit here I think is the memory savings and the GIL release, which allows you to more data in less hardware. We were able to do the same operations in about a third the hardware on standard benchmark workloads.
1
9
1,248
The @pandas_dev 2.0 release improved support for @ApacheArrow strings. This has pretty dramatic effects for people using @dask_dev at large scale. Short blogpost exploring the benefits here: medium.com/coiled-hq/pyarrow…
1
6
33
4,639
I'll be talking about "Scaling Python with @dask_dev and @Coiled_HQ" in a couple of weeks, this is a free online event hosted by Bethesda Data Science Meetup. You can register at the following link meetup.com/bethesda-data-sci…
3
5
1,475
(n/) Thanks to all the open source projects that were used to create this package! 😁 @xarray_dev @dask_dev #stackstac #xmltodict @STACspec #rasterio #rioxarray #pystac and many others! 🚀
3
2
5
1,327
📢#xcube v1.0 released 🎉! Create, serve, view, analyse, and process geospatial data cubes. Builds on @xarray_dev, @dask_dev, compatible with @pangeo-data. Data stores (@CMEMS_EU, @esaclimate, @sentinel_hub, etc.), cube server (WMTS, STAC, S3), viewer. ➡️github.com/dcs4cop/xcube
2
21
70
7,465
Debugging a dask-geopandas performance issue, which turned out to be a data partitioning issue: tomaugspurger.github.io/post…
1
3
11
3,944
Load more