mbobin's blog

Understanding PostgreSQL table partitioning

Fri, 21 Feb 2025 19:01:19 +0000

Database performance and manageability become challenging as applications grow and data volumes expand. PostgreSQL’s table partitioning feature splits large tables into smaller pieces while maintaining the appearance of a single table. This approach improves query performance and simplifies database maintenance, but it requires careful planning to implement effectively.

The fundamentals of table partitioning

Table partitioning helps manage large datasets. Rather than storing data in a single table, partitioning distributes it across multiple smaller tables. From the application’s perspective, these partitioned tables behave as a single entity. Under the hood, PostgreSQL manages them as separate physical tables.

Dividing data based on logical criteria, such as date ranges or numeric boundaries, makes it easier to query specific records and manage the overall dataset.

The benefits of proper partitioning

When implemented thoughtfully, table partitioning improves database performance. A key benefit is partition pruning. This feature allows PostgreSQL to eliminate irrelevant partitions during query execution. For example, when searching for transactions from February 2025 in a date-partitioned table, PostgreSQL only scans the February 2025 partition. This reduces the amount of data processed.

Database maintenance also becomes easier. Administrators can run VACUUM operations on individual partitions independently, which reduces system resource requirements. For data retention, removing old data requires only dropping the corresponding partition. This operation is much faster than deleting millions of rows from a single table.

Choosing a partitioning strategy

PostgreSQL offers several partitioning strategies for different use cases.

Date range partitioning

Date range partitioning works well for time-series data. Consider an audit events table that tracks security activities. Users typically query recent events or specific time frames. Partitioning this table by month creates a natural organization for the data.

CREATE TABLE audit_events (
  id SERIAL NOT NULL,
  author_id INT NOT NULL,
  details jsonb NOT NULL,
  created_at timestamptz NOT NULL,
  PRIMARY KEY (id, created_at))
PARTITION BY RANGE(created_at);

Partitions might look like this:

audit_events_202502 FOR VALUES FROM ('2025-02-01') TO ('2025-03-01')
audit_events_202503 FOR VALUES FROM ('2025-03-01') TO ('2025-04-01')

A query for February 2025 audit events will only scan the audit_events_202502 partition.

Integer range partitioning

Integer range partitioning uses numeric boundaries. This approach suits tables with sequential IDs. For example, a merge request diff files table:

CREATE TABLE merge_request_diff_files (
  merge_request_diff_id INT NOT NULL,
  relative_order INT NOT NULL,
  PRIMARY KEY (merge_request_diff_id, relative_order))
PARTITION BY RANGE(merge_request_diff_id);

This structure partitions data based on ranges of merge request IDs.

Hash partitioning

Hash partitioning distributes data based on a hash of the partition key. This strategy maintains even data distribution and ensures ID uniqueness across partitions. The number of partitions must be set at creation time. This method is useful when queries look up records by exact key matches, even data distribution is necessary, and range-based queries are infrequent.

List partitioning

List partitioning organizes data based on specific values. This approach fits data that falls into discrete groups, such as partitioning a global customer database by country code or a multi-tenant application by tenant ID.

Implementation considerations

Successfully implementing table partitioning requires aligning the partition key with your most common query patterns. If queries frequently filter by date ranges, date-range partitioning is appropriate. If the application primarily looks up records by customer ID, list or hash partitioning might fit better.

Partitioning strategies are typically set at table creation time and are difficult to change later. Administrators must plan for data growth, the creation of new partitions, data retention policies, and data distribution across partitions.

Partitioning adds management overhead. For smaller tables or those with unpredictable query patterns, this overhead might outweigh the performance benefits.

Making the transition

Migrating an existing table to a partitioned structure involves creating a new partitioned table, setting up the partitions, migrating existing data, and updating application code to handle any behavioral differences. This migration often spans multiple releases to ensure a smooth deployment.

Conclusion

PostgreSQL’s table partitioning improves database performance and manageability. Success depends on understanding data access patterns, selecting the right partitioning strategy, and planning the implementation. Properly executed partitioning helps databases scale efficiently as data volumes grow.

PostgreSQL database scaling: a short overview

Mon, 17 Feb 2025 16:10:06 +0000

As data volumes increase, database scaling maintains system performance and reliability. PostgreSQL provides multiple scaling strategies for different workloads. This guide explores these approaches, from vertical scaling to sharding and partitioning.

Vertical scaling

Scaling often begins with vertical scaling, or “scaling up.” This approach enhances the resources of your existing database server. Instead of distributing the load, you optimize the server by increasing RAM to improve buffer cache performance, adding CPU cores for higher concurrent loads, and upgrading to modern NVMe drives. You then fine-tune PostgreSQL parameters to utilize the improved hardware.

Vertical scaling is straightforward. It requires no application changes and preserves all PostgreSQL features, making it effective for read-heavy workloads that benefit from larger memory caches. However, physical single-server capacity eventually becomes a constraint. Hardware costs increase significantly at higher tiers, and a single server remains a single point of failure.

Horizontal scaling

When vertical scaling reaches its physical limits, horizontal scaling, or “scaling out,” distributes the database load across multiple servers. Read replicas form the foundation of this approach. A primary server handles writes while secondary servers manage read operations. This architecture requires tracking replication lag and implementing query routing logic in your application.

PostgreSQL supports various replication strategies. Synchronous replication ensures strict data consistency, while asynchronous replication favors performance. You can use cascading replication for geographic distribution and logical replication for selective tables.

Connection management is critical in distributed environments. Tools like PgBouncer help manage database connections, reducing overhead and optimizing resource use. You must configure pool modes based on workload patterns and adjust them as usage evolves.

Database decomposition

Database decomposition splits a single monolithic database into multiple specialized databases running on dedicated hardware. This strategy isolates workloads and allows independent scaling.

This approach shifts complexity to the application layer. Cross-database joins are no longer possible directly; applications must orchestrate multi-step queries to retrieve related data. Transaction management also requires new logic to maintain atomicity across separate databases.

Table sharding

Table sharding distributes individual tables across multiple databases using a sharding key. While it reduces the size of individual indexes and tables, it introduces distinct infrastructure challenges. Applications must handle sequence management across shards, and the lack of foreign key constraints requires alternative methods for data integrity.

You can implement sharding through application-level routing or middleware. Application-level sharding gives developers direct control but increases code complexity. Middleware solutions offer transparency to the application but add infrastructure overhead. Success depends on key management, typically managed through UUID primary keys, shard-aware composite keys, or centralized ID generators.

Table partitioning

Table partitioning divides large tables into smaller pieces while maintaining a unified interface. PostgreSQL supports range, list, and hash partitioning. Range partitioning organizes time-series data, list partitioning categorizes data, and hash partitioning ensures even distribution.

Partitioning improves query execution times through partition pruning, where PostgreSQL skips irrelevant partitions during a query. Combined with simplified partition-level maintenance and foreign key support, partitioning is an effective internal scaling tool. You must plan capacity and select partition keys early to prevent performance degradation from excessive partitioning.

Monitoring and observability

Effective scaling relies on active monitoring. Metrics like query execution times, cache hit ratios, and transaction rates indicate database health. Tracking resource utilization across CPU, memory, disk I/O, and network helps identify bottlenecks. For distributed setups, you must monitor replication lag, WAL generation rates, and standby server status.

PostgreSQL provides built-in statistics collectors. You can pair these with tools like Prometheus and Grafana for visualization, or pgBadger for log analysis. Set up alerts for critical thresholds, such as replication lag or declining cache hit ratios.

Making the right choice

Choosing a scaling strategy depends on data volume, query patterns, and budget constraints.

Most implementations combine approaches. For example, using table partitioning for large datasets while adding read replicas handles both data volume and read-heavy query loads. Evaluate your system metrics regularly and adjust your infrastructure as application demands change.

Debugging gateway timeouts in a Docker homelab

Fri, 14 Feb 2025 15:15:30 +0000

I recently resolved a recurring gateway timeout issue in my Docker-based homelab. Here is how I identified and fixed the routing problem.

My homelab relies on Docker Compose and Traefik for reverse proxy duties. I chose this setup for its simplicity, which makes mobile debugging easier. Each application operates in a dedicated Docker Compose file, and Traefik handles the routing and SSL certificates.

The system initially worked well, but I began experiencing occasional gateway timeouts in Synapse. Adding Immich to the server increased the frequency of these timeouts. When I bypassed Traefik and accessed the services directly, they functioned normally. This indicated the routing layer caused the timeouts.

The affected containers were connected to two Docker networks:

A backend network for inter-container communication within each service
A frontend network shared by all HTTP-serving containers for Traefik routing

When containers attach to multiple networks, Traefik randomly selects one for traffic routing. The initial tests succeeded because Traefik temporarily routed through the correct network. Subsequent requests failed when it switched to the internal backend network.

To fix this, you must explicitly define the routing network. Traefik provides a setting to specify the network via a container label:

- "traefik.docker.network=mynetwork"

Traefik will now consistently use the specified network, resolving the timeout errors. Explicitly defining the network ensures reliable routing for multi-network containers.

PostgreSQL's temporary tables

Sat, 12 Sep 2020 06:41:38 +0000

Testing ActiveRecord concerns in isolation without including them in an application model can be challenging.

Consider an ActiveRecord concern that extracts a specific database behavior. For example, a query for tagging:

module TaggableConcern
  extend ActiveSupport::Concern

  included do
    scope :with_tag, ->(tag) {
      where('? = ANY(tags)', tag)
    }

    scope :with_tags, ->(tags) {
      where('tags @> ARRAY[?]::varchar[]', Array(tags))
    }
  end
end

To test this cleanly, you can leverage a specific PostgreSQL feature found in the CREATE TABLE synopsis: temporary tables.

CREATE TEMPORARY TABLE table_name (...)
ON COMMIT { PRESERVE ROWS | DELETE ROWS | DROP }

Using this in Rails is straightforward. The [create_table]((https://api.rubyonrails.org/v6.0/classes/ActiveRecord/ConnectionAdapters/SchemaStatements.html#method-i-create_table) method accepts :temporary as a boolean argument to mark the table as ephemeral. It also accepts an options parameter to define extra table instructions, such as when to drop it:

create_table :my_table, temporary: true, options: 'ON COMMIT DROP' do |t|
  ...
end

Here is how to apply this to test the tagging concern in RSpec:

RSpec.describe TaggableConcern, type: :model do
  before do
    class TempBook < ApplicationRecord
      include TaggableConcern

      connection.create_table :temp_books, temporary: true, options: 'ON COMMIT DROP' do |t|
        t.string :name
        t.string :tags, array: true

        t.timestamps
      end
    end
  end

  let!(:ruby) do
    TempBook.create!(
      name: 'Programming Rails',
      tags: %w[ruby rails web]
    )
  end

  let!(:elixir) do
    TempBook.create!(
      name: 'Programming Phoenix',
      tags: %w[elixir phoenix web]
    )
  end

  let!(:go) do
    TempBook.create!(
      name: 'Programming Go',
      tags: %w[go programming]
    )
  end

  describe '.with_tag' do
    subject { TempBook.with_tag(query) }

    context 'with nil value' do
      let(:query) { nil }

      it { is_expected.to be_empty }
    end

    context 'with one match' do
      let(:query) { 'ruby' }

      it { is_expected.to match_array([ruby]) }
    end

    context 'with multiple matches' do
      let(:query) { 'web' }

      it { is_expected.to match_array([ruby, elixir]) }
    end
  end

  describe '.with_tags' do
    subject { TempBook.with_tags(query) }

    context 'with nil value' do
      let(:query) { nil }

      it { is_expected.to be_empty }
    end

    context 'with empty array' do
      let(:query) { [] }

      it { is_expected.to be_empty }
    end

    context 'with a common tag' do
      let(:query) { 'web' }

      it { is_expected.to match_array([ruby, elixir]) }
    end

    context 'with a common tag as array' do
      let(:query) { ['web'] }

      it { is_expected.to match_array([ruby, elixir]) }
    end

    context 'with mixed tags' do
      let(:query) { ['ruby', 'elixir'] }

      it { is_expected.to be_empty }
    end

    context 'with mixed tags' do
      let(:query) { ['ruby', 'web'] }

      it { is_expected.to match_array([ruby]) }
    end
  end
end

Finished in 0.1783 seconds (files took 0.59146 seconds to load)
9 examples, 0 failures

Git patch cheatsheet

Tue, 10 Sep 2019 09:16:37 +0000

I always forget how to generate git patches and need to search for the right commands. This is a collection of useful commands and their source.

  git format-patch -1 <sha>
  git apply --stat file.patch # show stats.
  git apply --check file.patch # check for error before applying.
  git am < file.patch # apply the patch finally.

source

To generate the patches from the topmost commits from a specific SHA1 hash:

git format-patch -<n> <SHA1>

The last 10 patches from head in a single patch file:

git format-patch -10 HEAD --stdout > 0001-last-10-commits.patch

source

Tmux - autostart sessions

Sat, 30 Mar 2019 21:10:18 +0000

if (command -v tmux>/dev/null) && \
  [[ ! $TERM =~ screen ]] && \
  [ -z $TMUX ] && \
  [ "$TMUX_ENABLED" == "yes" ]; then
    /usr/bin/env tmux new -s 0
    /usr/bin/env tmux attach -t 0
fi

Breakdown:

(command -v tmux>/dev/null) - Tmux is installed
[[ ! $TERM =~ screen ]] && [ -z $TMUX ] - try not to run Tmux inside Tmux
[ "$TMUX_ENABLED" == "yes" ] - TMUX_ENABLED is set from Alacritty config file and it will start Tmux only in this terminal emulator.
tmux new -s 0 - find or duplicate a session with name 0
tmux attach -t 0 - attach to session 0

Pi-hole and OpenVPN setup

Mon, 15 Oct 2018 19:01:09 +0000

Tmux configs

Sat, 06 Oct 2018 21:52:09 +0000

Key bindings from my dotfiles for Tmux:

tmux key	Description
`C-a`	Default prefix, used instead of “C-b”.
`<prefix> a`	Move to the beginning of the line.
`<prefix> r`	Reload tmux configuration from ~/.tmux.conf file.
`<prefix> -`	Split new pane horizontally.
`<prefix> \|`	Split new pane vertically.
`<prefix> h`	Select pane on the left.
`<prefix> j`	Select pane on the bottom.
`<prefix> k`	Select pane on the top.
`<prefix> l`	Select pane on the right.
`<prefix> →`	Select pane on the right.
`<prefix> ←`	Select pane on the left.
`<prefix> ↓`	Select pane on the bottom.
`<prefix> ↑`	Select pane on the top.
`<prefix> C-←`	Resize pane to the left.
`<prefix> C-→`	Resize pane to the right.
`<prefix> C-↑`	Resize pane to the top.
`<prefix> C-↓`	Resize pane to the bottom.
`<prefix> +`	Zoom in/out on the active pane.
`<prefix> I`	Synchronize panes from the current window.
`<prefix> c`	Create a new window.
`<prefix> n`	Switch to the next window.
`<prefix> S-→`	Switch to the next window.
`<prefix> p`	Switch to the previous window.
`<prefix> S-←`	Switch to the previous window.
`<prefix> 0-9`	Switch to a window using it’s index number.
`<prefix> C-a`	Switch to the last used window.
`<prefix> v`	Enter into Copy mode.
`<prefix> y`	Copy selection.
`<prefix> Enter`	Copy selection.

RSpec composable matchers

Tue, 02 Oct 2018 18:04:09 +0000

While working on the Stride integration for GitLab, I found myself with the need of matching some complex and repetitive JSONs like the one from below.

{
  "type": "paragraph",
  "content": [
    {
      "type": "text",
      "text": "John Doe16 pushed new tag "
    },
    {
      "type": "text",
      "text": "test",
      "marks": [
        {
          "type": "link",
          "attrs": {
            "href": "http://localhost/namespace8/project8/commits/test"
          }
        }
      ]
    },
    {
      "type": "text",
      "text": " to "
    },
    {
      "type": "text",
      "text": "JohnDoe15/project8",
      "marks": [
        {
          "type": "link",
          "attrs": {
            "href": "http://localhost/namespace8/project8"
          }
        }
      ]
    }
  ]
}

Writing a matcher for this JSON got pretty hairy and not very readable, but RSpec has composable matchers that can be combined to express the exact details of the expectations. Let’s see how a spec will look with these matchers.

context 'description' do
  subject { push_message.call.dig('content', 0, 'content', 0) }

  it 'generates a remove message' do
    ref = Gitlab::Git.ref_name(sample_data[:ref])

    is_expected.to be_a_stride_paragraph_with(
      a_stride_text("#{user.name} removed branch "),
      a_stride_text(ref),
      a_stride_text(" from "),
      a_stride_link(stride_service.project_name, stride_service.project_url)
    )
  end
end

And all we need now is to define those matchers. Only one small problem with this strategy: there isn’t any obivious documentation on how to do it. After browsing the code, we find our way in some comments from the matchers file.

Include {RSpec::Matchers::Composable} in your custom matcher to make it support

being composed (matchers defined using the DSL have this included automatically).

So it’s just a regular matcher that includes RSpec::Matchers::Composable. And here they are.

RSpec::Matchers.define :be_a_stride_paragraph_with do |*nodes|
  include RSpec::Matchers::Composable

  match do |actual|
    expect(actual).to match(
    	hash_including(
    		'type' => 'paragraph',
    		'content' => a_collection_including(*nodes)
    	)
    )
  end
end

RSpec::Matchers.define :a_stride_text do |text, others = {}|
  include RSpec::Matchers::Composable

  data = { 'type' => 'text', 'text' => text }.merge(others)
  match do |actual|
    expect(actual).to match(hash_including(data))
  end
end

RSpec::Matchers.define :a_stride_link do |name, url|
  include RSpec::Matchers::Composable

  match do |actual|
    expect(actual).to match(
    	a_stride_text(name, 'marks' => a_stride_link_mark(url))
    )
  end
end

RSpec::Matchers.define :a_stride_link_mark do |url|
  include RSpec::Matchers::Composable

  match do |actual|
    expect(actual).to match(
    	a_collection_containing_exactly(
	      hash_including('type' => 'link',
	                     'attrs' => hash_including('href' => url)
	                    )
    ))
  end
end

Ruby on Rails on Android

Sat, 25 Feb 2017 12:20:09 +0000

I have fooled around with Ruboto but I don’t want to build native Android apps. I just want to start an irb session without 3 apps.

Here comes Termux. This is its official description:

Termux is an Android terminal emulator and Linux environment app that works directly with no rooting or setup required. A minimal base system is installed automatically - additional packages are available using the APT package manager.

The definition of a gold mine if you ask me. And yeah, they have a lot of packages, including the latest stable version of Ruby. Go and see for yourself on their github repository.

Get it from Google Play, update the packages list and install Ruby in one line.

$ apt update
$ apt install ruby
$ ruby -v
ruby 2.4.0p0 (2016-12-24 revision 57164) [aarch64-linux-android] # or something like this

Formidable! Now let’s put it to the test.

irb> puts "Hello World!"
Hello World!
=> nil

$ mkdir -p projects/ruby
$ echo 'puts "hello world!"' > projects/ruby/hello.rb
$ ruby projects/ruby/hello.rb
hello world!

Isn’t that wonderful?

I have a OnePlus 2 and 2 GiB of RAM are free most of the time. That’s much more than you can get from Heroku’s free dyno. Let’s see if we can run a Rails app on Android.

First step: bundler

$ gem install bundler

Now for the painful part, installing nokogiri’s dependencies.

We need something to build our native extensions.

$ apt install clang make

And these should do the trick:

$ apt install ruby-dev pkg-config libxml2 libxml2-dev libxslt libxslt-dev
$ gem install pkg-config

Say a prayer and hit return after the following line.

$ gem install nokogiri -- --use-system-libraries

Good. I spent quite some time gathering all of them (more than I care to admit as a matter of fact).

We will need a database for our app. Rails comes by default with a sqlite database and it carries along some dependencies that are not satisfied by default on our android device.

$ apt install libffi libffi-dev
$ apt install libsqlite libsqlite-dev

And something for our asset pipeline:

$ apt-install nodejs

Our reward:

$ gem install rails -v 5.2
$ cd projects/ruby
$ rails new demo
$ cd demo

We need to add a couple of gems for timezones. Open Gemfile with vi and add(be carefull with their order and platform options if any):

gem 'tzinfo'
gem 'tzinfo-data'

The moment of truth:

$ bundle install
$ rails server

You can find a working demo app on my GitHub profile.

update 14.01.2018: Updated Rails version and added demo app and pictures.