Mastodon

Another look at creating Rails users in Postgres

Three years (!) has passed since I first wrote about setting up users for your Postgres development database. This does not come up often for me, but every time I see my own list of instructions I shudder and think there must be a better way. This morning I figured it out.

Here is what I have in my config/database.yml:

development:
adapter: postgresql
encoding: unicode
database: myapp_development
pool: 5

Omitting the username and password from the database.yml means that postgres will try to log using your operating system username using the peer method. Since I am logged in as “mike” that is the username that will be used to authenticate. With this in mind I am just going to create a database user with that name.

mike@sleepycat:~/projects/myapp$ sudo -u postgres createuser --interactive mike
Shall the new role be a superuser? (y/n) y

Notice that used sudo -u to switch to the existing “postgres” user (created by default) and created a superuser that matched my operating system account username.
After that, everything works. Even rake commands.

mike@sleepycat:~/projects/myapp$ rake db:create
mike@sleepycat:~/projects/myapp$ rake db:migrate
==  CreateWidgets: migrating ============================================
...
mike@sleepycat:~/projects/myapp$ rails dbconsole
psql (9.1.9)
Type "help" for help.

myapp_development=# 

Now that is far more civilized.

Creating Rails users in Postgres on Ubuntu

Note: Since writing this I have revisited this issue and found a better way. read about it here.

I am getting started with a new greenfield Rails app and had a bit of an adventure setting up a user for my Rails app in Postgres. My attempts to connect or create a database would fail. I thought I had created a user properly but apparently I was wrong.
Now I know for next time.
Start with the create user command:


mike@sleepycat:~$ sudo -u postgres createuser --createdb --pwprompt desired_username

You can see we are using the createdb option in this call, allowing the user we are creating to create new databases. You can take a look at the databases available and the users that can access them with the command:


sudo -u postgres psql -l

So now lets assign the user we created rights on the database we just created with:


mike@sleepycat:~$ sudo -u postgres psql my_project_development -c "grant all privileges on database my_project_development to desired_username;"
GRANT

I had hoped that might be all that was needed but then my rake task crapped out:


mike@sleepycat:~$ rake db:create

Couldn't create database for {"encoding"=>"unicode", "username"=>"desired_username", "adapter"=>"postgresql", "database"=>"my_project_test", "pool"=>5, "password"=>nil}
FATAL:  Ident authentication failed for user "desired_username"

It turns out that Postgres is set to expect the user accessing the database to actually have a system account. That seems like a little much for a development environment where I would end up with tonnes of those accounts. To change that expectation you need to edit the host-based authentication configuration file pg_hba.conf:


sudo vim /etc/postgresql/8.4/main/pg_hba.conf

Press capital G to jump to the bottom of the file and change “local all postgres ident” and “local all all ident” to md5 as shown below:


# DO NOT DISABLE!
# If you change this first entry you will need to make sure that the
# database super user can access the database using some other method.
# Noninteractive access to all databases is required during automatic maintenance
#(custom daily cronjobs, replication, and similar tasks).
# Database administrative login by UNIX sockets
local   all         postgres         md5

# TYPE  DATABASE    USER        CIDR-ADDRESS      METHOD
# "local" is for Unix domain socket connections only

local   all         all         md5
# IPv4 local connections:
host    all         all         127.0.0.1/32    md5
# IPv6 local connections:
host    all         all         ::1/128       md5

Restart your Postgres server and you should be good to go.


mike@sleepycat:~/projects/key_master$ sudo /etc/init.d/postgresql-8.4 restart
* Restarting PostgreSQL 8.4 database server
...done.

Now your Rails users don’t need to have system accounts and your rake tasks should run just fine. Back to work!

Thanks to Mark Berry for his comments below! They have been incorporated into the post now.

Postgres and Rails – for MySQL people

A little while ago I decided to switch from MySQL to Postgres for my development database. Although I love how command line friendly MySQL is and think group_concat is the bomb, I’ve been burned my MySQL’s loosey-goosey nature and had it corrupt some data on me. I decided that it was time to switch to Postgres. The transition was a little aggravating at times but it doesn’t take long to get used to it. Installing Postgres is easy enough on Ubuntu:

mike@sleepycat:~$ sudo aptitude install postgresql postgresql-server-dev-8.4

If you need to start at new Rails project you can specify Postgres right from square one:

mike@sleepycat:~/test$ rails myapp -d postgresql

After installing the database I started to look for the gems that provide the driver. I found myself sorting through a bewildering number of gems. “Postgres”, “postgres-pr”, a few minutes search turned up several options each of which proved to be the wrong thing. The gem you need is actually called ruby-pg and is written by a group of people that includes the actual creator of the Ruby language, Yukihiro Matsumoto. Although it is called ruby-pg if you ever want to find it on the internet, you install it using the name pg:

sudo gem install pg

The next stumbling block for me was my assumption that the naming of the Rails adapter would be similar to MySQL. Since adapter: mysql is what I have been using in my project already, I assumed adapter: postgres would be fine. It’s not.

development:
adapter: postgresql
database: myapp_development
username: postgres
password: password
host: localhost

Coming from MySQL, it took a fair bit of reading to collect the Postgres equivalent of all the commands I use most often. Just to save you some looking here is my “Cole’s Notes” version:

Setting a password for the default user ‘postgres’:

mike@sleepycat:~$ sudo -u postgres psql postgres
[sudo] password for mike:
psql (8.4.2)
Type “help” for help.
postgres=# \password postgres
Enter new password:
Enter it again:

Create a database as user ‘postgres’:

mike@sleepycat:~$ sudo -u postgres createdb myapp_development

Drop a database as user ‘postgres’:

mike@sleepycat:~$ sudo -u postgres dropdb myapp_development

Get to the command line of the database myapp_development as user ‘postgres’:

mike@sleepycat:~$ sudo -u postgres psql myapp_development

Load the file ‘pgdump.sql’ into the myapp_development database as the user ‘postgres’:

mike@sleepycat:~/$ sudo -u postgres psql -f pgdump.sql myapp_development

Dump a Postgres database to a file:

mike@sleepycat:~/$ sudo -u postgres pg_dump myapp_development > ~/myapp_dev_dump.sql

Show tables:

postgres=# \dt

show a table:

postgres=# \d tablename

quit:

postgres=# \q

Thats what worked for me. In general I feel like using postgres is forcing me to write more portable code. Pretty much anything I write in Rails that runs in postgres will run on MySQL, but the reverse is really not true. Little things like using ‘1’ for true. MySQL is fine with that but for Postgres with a real boolean datatype, true is true. 1 is not. I stumbled arcoss that when I was using validates_acceptance_of. I feel like its also forcing me to write better SQL as well. If you’ve written a bunch of “find_by_sql” in an existing app you might not appreciate its pickyness though. However once the initial pain of the switch passes, Postgres is pretty great.

Lost in translation: the down side of taps

UPDATE: Both Taps and Sequel have been updated and the problems that caused my issues have been resolved. I am now happily db:pushing and db:pulling. It took a while, but it got fixed in the end. :)

I have been using Heroku for a while now and its definitely great. One of the things that really blew me away when I started using it was being able to send my my entire database data up to the server with a single command:

heroku db:push

Could it be any easier? Thanks to the magic of taps, Schema AND data, transferred between my local MySQL database and Heroku’s Postgres database. At the time I might of stopped for a second to marvel at how big a deal that is, but only for a second. It turns out that its really worth stopping and thinking about what goes on when you run that command; especially when you are developing on MySQL. MySQL and Postgres have different datatypes to store data in and making an finding an exact equivalent from one database to the next is much easier said than done.

I think I will let the data do the talking for me on this. Keep you eye on the terminal_id column:

mysql> desc terminals;
+————-+—————+——+—–+————-+—————-+
| Field       | Type          | Null | Key | Default     | Extra          |
+————-+—————+——+—–+————-+—————-+
| id          | int(11)       | NO   | PRI | NULL        | auto_increment |
| location_id | int(11)       | YES  |     | NULL        |                |
| merchant_id | int(11)       | YES  |     | NULL        |                |
| terminal_id | decimal(11,0) | YES  |     | NULL        |                |
| reference   | varchar(255)  | YES  |     | NULL        |                |

8 rows in set (0.00 sec)

The data:

mysql> select terminal_id from terminals;
+————-+
| terminal_id |
+————-+
| 10792146001 |
| 10152407001 |
| 10392407002 |
| 10152617003 |
| 10184619001 |

16 rows in set (0.00 sec)

Pushing it up to the server  this suddenly becomes:

CREATE TABLE terminals (
id integer NOT NULL,
location_id integer,
merchant_id integer,
terminal_id integer,
reference character varying(255),

After a Heroku db:pull command:

mysql> desc terminals;
+————-+————–+——+—–+————-+—————-+
| Field       | Type         | Null | Key | Default     | Extra          |
+————-+————–+——+—–+————-+—————-+
| id          | int(11)      | NO   | PRI | NULL        | auto_increment |
| location_id | int(11)      | YES  |     | NULL        |                |
| merchant_id | int(11)      | YES  |     | NULL        |                |
| terminal_id | int(11)      | YES  |     | NULL        |                |
| reference   | varchar(255) | YES  |     | NULL        |                |

8 rows in set (0.00 sec)

The data:

mysql> select terminal_id from terminals;
+————-+
| terminal_id |
+————-+
|  2147483647 |
|  2147483647 |
|  2147483647 |
|  2147483647 |
|  2147483647 |
|  2147483647 |
|  2147483647 |
|  2147483647 |

How about that. Now I have a column of maxint values and a lot of very confusing bugs. Nothing like a little integer overflow to make the day go quickly. Worse still is pushing back up to the staging server before noticing, duplicating the corrupted data up there. Good thing it didn’t effect production. I am guessing that had I been developing using Postgres there would be no problem, because taps would not have to convert from one datatype to another.  I am going to have to think about doing that even though I am much more familiar with MySQL. Either way the ease of that heroku db:push/pull command definitely belies the magnitude of what that command is doing.

So the takeaway from this? For me:

Backups = good.

Staging server = good.

Taps = use with caution.