AWS Big Data Blog

Today we are announcing that Amazon has signed a definitive agreement to acquire DuckLabs, the Amsterdam-based company behind the open-source analytical database DuckDB. We expect the transaction to close shortly, subject to customary closing conditions. Hannes Mühleisen and Mark Raasveldt, who created DuckDB and co-founded DuckLabs, will continue leading the team and the open-source project’s technical direction as part of AWS. The DuckDB open-source project will also continue to be driven by the DuckLabs team, remain open source under the independent Foundation (the non-profit that oversees DuckDB), and available under the MIT license as it does today.

Agentic Data Ingestion is an open-source solution that uses AI agents on Amazon Bedrock to automate data pipeline creation, from file upload to queryable tables. Point it at a file or database table, and foundation models propose the field mapping, generate and test the AWS Glue transform, and run the job behind human approval gates.

Workday Data Cloud now integrates with AWS, giving developers bi-directional, zero-copy access to Workday’s governed people and money data from the AWS analytics and AI services they already use, without brittle pipelines or data duplication.

Amazon Redshift has no native semantic layer, yet teams still need consistent, governed metric definitions across BI tools. This post shows how to define a metric once with the open-source dbt Semantic Layer (MetricFlow) on Amazon Redshift so it stays correct at any grain, and proves a non-additive weekly average comes out right ($280, not $550).

The CUDOS 5.10 dashboard adds an Amazon Redshift RA3-to-RG section that helps you assess the cost-efficiency opportunity of moving to Graviton-based RG instances, track adoption as your instance-generation mix shifts, and quantify the gains already achieved on RG.

Amazon Athena now offers system tables: SQL-queryable, per-query execution metadata stored as Apache Iceberg in your own account. This post walks through what system tables are, how to turn them on, and how to use them for usage analysis, cost attribution, performance trends, audit, and capacity optimization without custom pipelines.

Storing large semi-structured data such as JSON and arrays in a data warehouse often means splitting or truncating your data. With PartiQL and the SUPER data type, Amazon Redshift is now equipped to handle these issues. In this post, we walk you through three key SUPER data type enhancements that simplify semi-structured data workflows and reduce common migration challenges.

Governed, multi-tenant data access is a key requirement for SaaS and B2C products that embed AI-powered analytics. This post shows how to combine Amazon Cognito and AWS Lake Formation on Amazon Bedrock AgentCore to enforce row- and column-level data access for external users, with a minimal-privilege Lambda acting as a credential broker.

Learn how to migrate Google BigQuery workloads to an open lakehouse on AWS built on Apache Iceberg and Amazon S3. This post explains why an open lakehouse future-proofs your data, presents the target architecture on AWS, and walks through a step-by-step migration: assess, design, convert, migrate, cut over, and optimize.

Amazon SageMaker Unified Studio now offers designations: named permission sets that give administrators granular, least-privilege control over what each project member can do. Learn how designations work and how to create and assign your first one from the console and the AWS CLI.

A repeatable process for keeping Apache HBase on Amazon EMR highly available. Part 1 covers triaging an HBase coordinator that is slow to start, a RegionServer that leaves the cluster, and regions that stay in transition, with the web UI pages, commands, and log signatures that point to each cause.