Ryanair as well as other airlines follow general aviation rules, which however do not forbid selling more tickets than seats on a flight. If you are bumped off a Ryanair flight due to overbooking you can claim a maximum of 600 euro per passenger, depending on the distance of your flight. EC261/2006 states airlines are liable to pay compensations for passengers who did not get on a flight because there was no seat available.
Click here and start a claim, it's free.Check your compensation
As a common practice airlines like Ryanair would rely on statistics of the flights and no-shows in order to decide the maximum number of tickets they can sell extra. The reason for overbooking is to maximize revenue from a certain flight knowing that an average of 5-10 passengers are no-show. However, no one knows who will be the unlucky passenger to be denied boarding due to overbooking. Airlines usually offer an alternative flight and voucher right at the check-in counter. In some cases when departure is the next day Ryanair would as well offer hotel accommodation. Very often the value of the vouchers is lower than compensation established by EC261/2004. In this case though if you accept the voucher and give up your seat voluntarily, you automatically cancel your right to claim official compensation.
There is also the case when Ryanair can decide to change the aircraft due to commercial reasons. For instance on a flight operated by an aircraft with a capacity of 200 seats, only 150 have been sold. While on another flight demand is higher than capacity of the assigned aircraft. In this case there will be an optimization swap and some passengers will be asked at the airport to take another flight at no extra cost and to accept a voucher.
In order to ensure a smooth claim process those bumped off a flight should always get a confirmation in writing from Ryanair airport staff, otherwise it can get really challenging to prove boarding was denied due to overbooking. Of course it can be proven, however lack of this confirmation can require the involvement of higher authorities and so it will take longer to process the claim and win the case.