It uses the VLOOKUP salesforce formula function which is only available for validation rules (unfortunately). This validation rule has two parts:
1) It looks for a record of the same name
2) It makes sure that the record isn’t the exact same record as opposed to just a duplicate. This will happen when a record is edited, the name already exists (as itself) and when saved will trigger the part 1 result.
Here is the rule. Simply replace the object name and field name (which can both be entered automatically when using the formula editor)
and(
Name = VLOOKUP
($ObjectType.Service__c.Fields.Name , $ObjectType.Service__c.Fields.Name, Name ),
not(
Id=VLOOKUP
($ObjectType.Service__c.Fields.Id , $ObjectType.Service__c.Fields.Name, Name )))